Sujet 3 ème carte installée!

Citation :
Mais genre ça veut dire qu'en cas de multi cpu, un cpu gere par exemple le bus ultra ata et un autre le controleur usb ?

Most (all) Intel-MP compliant SMP boards have the so-called 'IO-APIC',

which is an enhanced interrupt controller, it enables us to route

hardware interrupts to multiple CPUs, or to CPU groups.



Linux supports all variants of compliant SMP boards, including ones with

multiple IO-APICs. (multiple IO-APICs are used in high-end servers to

distribute IRQ load further).



There are (a few) known breakages in certain older boards, which bugs are

usually worked around by the kernel. If your MP-compliant SMP board does

not boot Linux, then consult the linux-smp mailing list archives first.



If your box boots fine with enabled IO-APIC IRQs, then your

/proc/interrupts will look like this one:



   ---------------------------->

  hell:~> cat /proc/interrupts

             CPU0

    0:    1360293    IO-APIC-edge  timer

    1:          4    IO-APIC-edge  keyboard

    2:          0          XT-PIC  cascade

   13:          1          XT-PIC  fpu

   14:       1448    IO-APIC-edge  ide0

   16:      28232   IO-APIC-level  Intel EtherExpress Pro 10/100 Ethernet

   17:      51304   IO-APIC-level  eth0

  NMI:          0

  ERR:          0

  hell:~>

  <----------------------------



some interrupts are still listed as 'XT PIC', but this is not a problem,

none of those IRQ sources is performance-critical.





in the unlikely case that your board does not create a working mp-table,

you can use the pirq= boot parameter to 'hand-construct' IRQ entries. This

is nontrivial though and cannot be automated. One sample /etc/lilo.conf

entry:



        append="pirq=15,11,10"



the actual numbers depend on your system, on your PCI cards and on their

PCI slot position. Usually PCI slots are 'daisy chained' before they are

connected to the PCI chipset IRQ routing facility (the incoming PIRQ1-4

lines):



               ,-.        ,-.        ,-.        ,-.        ,-.

     PIRQ4 ----| |-.    ,-| |-.    ,-| |-.    ,-| |--------| |

               |S|  \  /  |S|  \  /  |S|  \  /  |S|        |S|

     PIRQ3 ----|l|-. `/---|l|-. `/---|l|-. `/---|l|--------|l|

               |o|  \/    |o|  \/    |o|  \/    |o|        |o|

     PIRQ2 ----|t|-./`----|t|-./`----|t|-./`----|t|--------|t|

               |1| /\     |2| /\     |3| /\     |4|        |5|

     PIRQ1 ----| |-  `----| |-  `----| |-  `----| |--------| |

               `-'        `-'        `-'        `-'        `-'



every PCI card emits a PCI IRQ, which can be INTA,INTB,INTC,INTD:



                               ,-.

                         INTD--| |

                               |S|

                         INTC--|l|

                               |o|

                         INTB--|t|

                               |x|

                         INTA--| |

                               `-'



These INTA-D PCI IRQs are always 'local to the card', their real meaning

depends on which slot they are in. If you look at the daisy chaining diagram,

a card in slot4, issuing INTA IRQ, it will end up as a signal on PIRQ2 of

the PCI chipset. Most cards issue INTA, this creates optimal distribution

between the PIRQ lines. (distributing IRQ sources properly is not a

necessity, PCI IRQs can be shared at will, but it's a good for performance

to have non shared interrupts). Slot5 should be used for videocards, they

do not use interrupts normally, thus they are not daisy chained either.



so if you have your SCSI card (IRQ11) in Slot1, Tulip card (IRQ9) in

Slot2, then you'll have to specify this pirq= line:



        append="pirq=11,9"



the following script tries to figure out such a default pirq= line from

your PCI configuration:



        echo -n pirq=; echo `scanpci | grep T_L | cut -c56-` | sed 's/ /,/g'



note that this script wont work if you have skipped a few slots or if your

board does not do default daisy-chaining. (or the IO-APIC has the PIRQ pins

connected in some strange way). E.g. if in the above case you have your SCSI

card (IRQ11) in Slot3, and have Slot1 empty:



        append="pirq=0,9,11"



[value '0' is a generic 'placeholder', reserved for empty (or non-IRQ emitting)

slots.]



generally, it's always possible to find out the correct pirq= settings, just

permute all IRQ numbers properly ... it will take some time though. An

'incorrect' pirq line will cause the booting process to hang, or a device

won't function properly (if it's inserted as eg. a module).



If you have 2 PCI buses, then you can use up to 8 pirq values. Although such

boards tend to have a good configuration.



Be prepared that it might happen that you need some strange pirq line:



        append="pirq=0,0,0,0,0,0,9,11"



use smart try-and-err techniques to find out the correct pirq line ...



good luck and mail to linux-smp@vger.kernel.org or

linux-kernel@vger.kernel.org if you have any problems that are not covered

by this document.



-- mingo

SMP IRQ Affinity



Background:  



Whenever a piece of hardware, such as disk controller or ethernet card, 

needs attention from the CPU, it throws an interrupt.  The interrupt tells

the CPU that something has happened and that the CPU should drop what

it's doing to handle the event.  In order to prevent mutliple devices from

sending the same interrupts, the IRQ system was established where each device

in a computer system is assigned its own special IRQ so that its interrupts

are unique.



Starting with the 2.4 kernel, Linux has gained the ability to assign certain

IRQs to specific processors (or groups of processors).  This is known

as SMP IRQ affinity, and it allows you control how your system will respond

to various hardware events.  It allows you to restrict or repartition

the work load that you server must do so that it can more efficiently do

it's job.



Obviously, in order for this to work, you will need a system that has more

than one processor (SMP).  You will also need to be running a 2.4 or higher

kernel.



Some brief and very bare information on SMP IRQ affinity is provided in

the kernel source tree of the 2.4 kernel in the file:



    /usr/src/linux-2.4/Documentation/IRQ-affinity.txt





How to use it:



SMP affinity is controlled by manipulating files in the /proc/irq/ directory.

In /proc/irq/ are directories that correspond to the IRQs present on your

system (not all IRQs may be available). In each of these directories is

the "smp_affinity" file, and this is where we will work our magic.



The first order of business is to figure out what IRQ a device is using.

This information is available in the /proc/interrupts file.  Here's a sample:



 [root@archimedes /proc]# cat /proc/interrupts 

            CPU0       CPU1       CPU2       CPU3       

   0:    4865302    5084964    4917705    5017077    IO-APIC-edge  timer

   1:        132        108        159        113    IO-APIC-edge  keyboard

   2:          0          0          0          0          XT-PIC  cascade

   8:          0          1          0          0    IO-APIC-edge  rtc

  10:          0          0          0          0   IO-APIC-level  usb-ohci

  14:          0          0          1          1    IO-APIC-edge  ide0

  24:      87298      86066      86012      86626   IO-APIC-level  aic7xxx

  31:      93707     106211     107988      93329   IO-APIC-level  eth0

 NMI:          0          0          0          0 

 LOC:   19883500   19883555   19883441   19883424 

 ERR:          0

 MIS:          0





As you can see, this is a 4 processor machine.  The first column (unlabelled)

lists the IRQs used on the system.  The rows with letters (ie, "NMI", "LOC")

are parts of other drivers used on the system and aren't really accessible

to us, so we'll just ignore them.



The second through fifth columns (labelled CPU0-CPU3) show the number of times

the corresponding process has handled an interrupt from that particular IRQ.

For example, all of the CPUs have handled roughly the same number of interrupts

for IRQ 24 (around 86,000 with CPU0 handling a little over 87,000).



The sixth column lists whether or not the device driver associated with the 

interrupt supports IO-APIC (see /usr/src/linux/Documentation/i386/IO-APIC.txt

for more information).  The only reason to look at this value is that 

SMP affinity will only work for IO-APIC enabled device drivers.  For

example, we will not be able to change the affinity for the "cascade"

driver (IRQ 2) because it doesn't support IO-APIC.



Finally, the seventh and last column lists the driver or device that is 

associated with the interrupt.  In the above example, our ethernet card

(eth0) is using IRQ 31, and our SCSI controller (aic7xxx) is using IRQ 24.



The first and last columns are really the only ones we're interested in here.

For the rest of this example, I'm going to assume that we want to adjust

the SMP affinity for th SCSI controller (IRQ 24).



Now that we've got the IRQ, we can change the processor affinity.  To

do this, we'll go into the /proc/irq/24/ directory, and see what the

affinity is currently set to:



 [root@archimedes Documentation]# cat /proc/irq/24/smp_affinity 

 ffffffff



This is a bitmask that represents which processors any interrupts on IRQ

24 should be routed to.  Each field in the bit mask corresponds to a processor.

The number held in the "smp_affinity" file is presented in hexadecimal format,

so in order to manipulate it properly we will need to convert our bit patterns

from binary to hex before setting them in the proc file.



Each of the "f"s above represents a group of 4 CPUs, with the rightmost

group being the least significant.  For the purposes of our discussion,

we're going to limit ourselves to only the first 4 CPUs (although we can

address up to 32).



In short, this means you only have to worry about the rightmost "f" and you

can assume everything else is a "0" (ie, our bitmask is "0000000f").



"f" is the hexadecimal represenatation for the decimal number 15 (fifteen)

and the binary pattern of "1111".  Each of the places in the binary pattern

corresponds to a CPU in the server, which means we can use the following

chart to represent the CPU bit patterns:



            Binary       Hex 

    CPU 0    0001         1 

    CPU 1    0010         2

    CPU 2    0100         4

    CPU 3    1000         8



By combining these bit patterns (basically, just adding the Hex values), we

can address more than one processor at a time.   For example, if I wanted

to talk to both CPU0 and CPU2 at the same time, the result is:



            Binary       Hex 

    CPU 0    0001         1 

  + CPU 2    0100         4

    -----------------------

    both     0101         5



If I want to address all four of the processors at once, then the result is:



            Binary       Hex 

    CPU 0    0001         1 

    CPU 1    0010         2

    CPU 2    0100         4

  + CPU 3    1000         8

    -----------------------

    both     1111         f



(Remember that we use the letters "a" through "f" to represent the numbers

 "10" to "15" in hex notation).



Given that, we now know that if we have a four processor system, we can 

assign any of 15 different CPU combinations to an IRQ (it would be 16, but

it isn't legal to assign an IRQ affinity of "0" to any IRQ... if you try,

Linux will just ignore your attempt).



So.  Now we get to the fun part.  Remember in our /proc/interrupts listing

above that all four of our CPUs had handled the close to the same amount of

interrupts for our SCSI card?  We now have the tools needed to limit managing

the SCSI card to just one processor and leave the other three free to

concentrate on doing other tasks.   Let's assume that we want to dedicate

our first CPU (CPU0) to handling the SCSI controller interrupts.  To do this,

we would simply run the following command:



 [root@archimedes /proc]# echo 1 > /proc/irq/24/smp_affinity 

 [root@archimedes /proc]# cat /proc/irq/24/smp_affinity 

 00000001



Now, let's test it out and see what happens:



 [root@archimedes /proc]# cd /tmp/

 [root@archimedes /tmp]# tar -zcf test.tgz /usr/src/linux-2.4.2 

 tar: Removing leading `/' from member names

 [root@archimedes /tmp]# tar -zxf test.tgz && rm -rf usr/

 [root@archimedes /tmp]# tar -zxf test.tgz && rm -rf usr/

 [root@archimedes /tmp]# tar -zxf test.tgz && rm -rf usr/

 [root@archimedes /tmp]# tar -zxf test.tgz && rm -rf usr/

 [root@archimedes /tmp]# tar -zxf test.tgz && rm -rf usr/

 [root@archimedes /tmp]# cat /proc/interrupts | grep 24:

  24:      99719      86067      86012      86627   IO-APIC-level  aic7xxx



Compare that to the previous run without having the IRQ bound to CPU0:



  24:      87298      86066      86012      86626   IO-APIC-level  aic7xxx



All of the interrupts from the disk controller are now handled exclusively

by the first CPU (CPU0), which means that our other 3 proccessors are free

to do other stuff now.



Finally, it should be pointed out that if you decide you no longer want

SMP affinity and would rather have the system revert back to the old set up,

then you can simply do:



 [root@archimedes /tmp]# cat /proc/irq/prof_cpu_mask >/proc/irq/24/smp_affinity



This will reset the "smp_affinity" file to use all "f"s, and will return to

the load sharing arrangement that we saw earlier.





What can I use it for?



- "balance" out multiple NICs in a multi-processor machine.  By tying a single

  NIC to a single CPU, you should be able to scale the amount of traffic

  your server can handle nicely.



- database servers (or servers with lots of disk storage) that also have

  heavy network loads can dedicate a CPU to their disk controller and assign

  another to deal with the NIC to help improve response times.





Can I do this with processes?



At this time, no.

Citation : Usually PCI slots are 'daisy chained'

Citation :

Manually assigning IRQs to PCI slots in the system BIOS as a troubleshooting method may work on some non-ACPI systems that use a standard PC hardware abstraction layer (HAL), but these settings are ignored by Plug and Play in Windows if ACPI support is enabled. If you must manually assign IRQ addresses through the BIOS to a device on an ACPI motherboard, you must reinstall Windows to force the installation to use a Standard PC HAL.