PCIe can not rescan for new PCIe device ( FPGA board )
Abdelghani Ouchabane
We are developing a FPGA board connected to a Fedora 15 PC host over PCIe. Right now, in the implementation and debug phase, I often need to power off and
power on the device or try different boards. This causes a problem with the Fedora 15 running on the AMD PC.
Typically the PC is booted when I need to insert the device under test. As expected, the Linux doesn't find the device and the software app cannot talk to it.
* If I do "lspci -v" then it does not list our device.
* Then I execute "echo 1 > /sys/bus/pci/rescan"
* Now "lspci -v" lists our device.
* But our software returns : 0xFFFFFFFF
************************************************************************************************************************************************************
[root@localhost ~]# show_regs
resource file = /sys/bus/pci/devices/0000:02:00.0/resource
base address = 0x40241000
0x40241000 0x00000000 0xFFFFFFFF 0xFFFFFFFF
0x40241008 0x00000008 0xFFFFFFFF 0xFFFFFFFF
0x40241010 0x00000010 0xFFFFFFFF 0xFFFFFFFF
0x40241018 0x00000018 0xFFFFFFFF 0xFFFFFFFF
0x40241020 0x00000020 0xFFFFFFFF 0xFFFFFFFF
0x40241028 0x00000028 0xFFFFFFFF 0xFFFFFFFF
0x40241030 0x00000030 0xFFFFFFFF 0xFFFFFFFF
0x40241038 0x00000038 0xFFFFFFFF 0xFFFFFFFF
0x40241040 0x00000040 0xFFFFFFFF 0xFFFFFFFF
0x40241048 0x00000048 0xFFFFFFFF 0xFFFFFFFF
0x40241050 0x00000050 0xFFFFFFFF 0xFFFFFFFF
0x40241058 0x00000058 0xFFFFFFFF 0xFFFFFFFF
************************************************************************************************************************************************************
lspci -vvv
02:00.0 Signal processing controller: eZono AG eZono Malta - 32 channels ultrasound front end (rev 01)
Subsystem: eZono AG Device 0001
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 7
Region 0: Memory at 40240000 (64-bit, prefetchable) [size=4K]
Region 2: Memory at 40200000 (64-bit, prefetchable) [size=256K]
Region 4: Memory at 40241000 (64-bit, prefetchable) [size=4K]
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [80] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #1, Speed 2.5GT/s, Width x1, ASPM L0s, Latency L0 <1us, L1 <1us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM L0s Enabled; RCB 64 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
************************************************************************************************************************************************************
lspci -xxx
02:00.0 Signal processing controller: eZono AG eZono Malta - 32 channels ultrasound front end (rev 01)
00: 34 12 02 00 02 00 10 00 01 00 80 11 00 00 00 00
10: 0c 00 24 40 00 00 00 00 0c 00 20 40 00 00 00 00
20: 0c 10 24 40 00 00 00 00 00 00 00 00 34 12 01 00
30: 00 00 00 00 50 00 00 00 00 00 00 00 00 01 00 00
40: 00 00 00 00 70 61 62 01 00 00 00 00 00 00 00 00
50: 05 78 80 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 01 80 03 00 00 00 00 00
80: 10 00 01 00 04 80 2c 01 10 28 00 00 11 44 00 01
90: 01 00 11 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
************************************************************************************************************************************************************
It looks that 'rescan' is not enough to handle cases like this? Or, is there a
way to "insert" the FPGAs later after the kernel boots up?
Many thanks in advance,
Ghani
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Bjorn Helgaas
On Fri, Oct 7, 2011 at 1:16 AM, Abdelghani Ouchabane
<abdel...@ezono.com> wrote:
> Hallo,
>
> We are developing a FPGA board connected to a Fedora 15 PC host over PCIe.
> Right now, in the implementation and debug phase, I often need to power off
> and
> power on the device or try different boards. This causes a problem with the
> Fedora 15 running on the AMD PC.
>
> Typically the PC is booted when I need to insert the device under test. As
> expected, the Linux doesn't find the device and the software app cannot talk
> to it.
>
> * If I do "lspci -v" then it does not list our device.
should automatically rescan the bus. Does dmesg say anything when you
do the hot-add?
> * Then I execute "echo 1 > /sys/bus/pci/rescan"
>
> * Now "lspci -v" lists our device.
> * But our software returns : 0xFFFFFFFF
and P2P bridge apertures containing the BARs, e.g,. the [mem
0x40241000-0x40241fff] region? The dmesg log should show all this
information.
Bjorn
Abdelghani Ouchabane
> [added cc: linu...@vger.kernel.org]
>
> On Fri, Oct 7, 2011 at 1:16 AM, Abdelghani Ouchabane
> <abdel...@ezono.com> wrote:
>
>> Hallo,
>>
>> We are developing a FPGA board connected to a Fedora 15 PC host over PCIe.
>> Right now, in the implementation and debug phase, I often need to power off
>> and
>> power on the device or try different boards. This causes a problem with the
>> Fedora 15 running on the AMD PC.
>>
>> Typically the PC is booted when I need to insert the device under test. As
>> expected, the Linux doesn't find the device and the software app cannot talk
>> to it.
>>
>> * If I do "lspci -v" then it does not list our device.
>>
>
> Do you have pciehp enabled and loaded? I would think a PCIe hot-add
> should automatically rescan the bus. Does dmesg say anything when you
> do the hot-add?
>
Yes, I have them :
****************************************************************************************************
lsmod :
pciehp 20282 0
cgosdrv 17632 0
****************************************************************************************************
/etc/modprobe.d/pciehp.conf
alias pci:v00001234d00000002sv00001234sd00000001bc11sc80i00 pciehp
options pciehp pciehp_force=1 pciehp_debug=1
****************************************************************************************************
After I plug my board in, I executed echo 1 > /sys/bus/pci/rescan
dmesg:
[ 73.203895] pci 0000:02:00.0: [1234:0002] type 0 class 0x001180
[ 73.203895] pci 0000:02:00.0: reg 10: [mem 0x00000000-0x00000fff
64bit pref]
[ 73.204083] pci 0000:02:00.0: reg 18: [mem 0x00000000-0x0003ffff
64bit pref]
[ 73.204114] pci 0000:02:00.0: reg 20: [mem 0x00000000-0x00000fff
64bit pref]
[ 73.206190] pci 0000:00:01.0: BAR 6: [??? 0x00000000 flags 0x2] has
bogus alignment
[ 73.206215] pci 0000:02:00.0: BAR 2: assigned [mem
0x40200000-0x4023ffff 64bit pref]
[ 73.206236] pci 0000:02:00.0: BAR 2: set to [mem
0x40200000-0x4023ffff 64bit pref] (PCI address [0x40200000-0x4023ffff])
[ 73.206247] pci 0000:02:00.0: BAR 0: assigned [mem
0x40240000-0x40240fff 64bit pref]
[ 73.206266] pci 0000:02:00.0: BAR 0: set to [mem
0x40240000-0x40240fff 64bit pref] (PCI address [0x40240000-0x40240fff])
[ 73.206277] pci 0000:02:00.0: BAR 4: assigned [mem
0x40241000-0x40241fff 64bit pref]
[ 73.206295] pci 0000:02:00.0: BAR 4: set to [mem
0x40241000-0x40241fff 64bit pref] (PCI address [0x40241000-0x40241fff])
****************************************************************************************************
>
>> * Then I execute "echo 1 > /sys/bus/pci/rescan"
>>
>> * Now "lspci -v" lists our device.
>>
>
> That means config space accesses to your device seem to work fine.
>
>
>> * But our software returns : 0xFFFFFFFF
>>
>
> Your device is on bus 02. Did you confirm that there are host bridge
> and P2P bridge apertures containing the BARs, e.g,. the [mem
> 0x40241000-0x40241fff] region? The dmesg log should show all this
> information.
>
Yes, it is on the bus 02.
Here is the log of dmesg :
[ 69.806322] pci 0000:02:00.0: [1234:0002] type 0 class 0x001180
[ 69.806426] pci 0000:02:00.0: reg 10: [mem 0x00000000-0x00000fff
64bit pref]
[ 69.806506] pci 0000:02:00.0: reg 18: [mem 0x00000000-0x0003ffff
64bit pref]
[ 69.806585] pci 0000:02:00.0: reg 20: [mem 0x00000000-0x00000fff
64bit pref]
[ 69.808194] pci 0000:00:01.0: BAR 6: [??? 0x00000000 flags 0x2] has
bogus alignment
[ 69.808271] pci 0000:02:00.0: BAR 2: assigned [mem
0x40200000-0x4023ffff 64bit pref]
[ 69.808343] pci 0000:02:00.0: BAR 2: set to [mem
0x40200000-0x4023ffff 64bit pref] (PCI address [0x40200000-0x4023ffff])
[ 69.808412] pci 0000:02:00.0: BAR 0: assigned [mem
0x40240000-0x40240fff 64bit pref]
[ 69.808650] pci 0000:02:00.0: BAR 0: set to [mem
0x40240000-0x40240fff 64bit pref] (PCI address [0x40240000-0x40240fff])
[ 69.808888] pci 0000:02:00.0: BAR 4: assigned [mem
0x40241000-0x40241fff 64bit pref]
[ 69.810805] pci 0000:02:00.0: BAR 4: set to [mem
0x40241000-0x40241fff 64bit pref] (PCI address [0x40241000-0x40241fff])
[ 84.309870] pci 0000:02:00.0: enabling device (0000 -> 0002)
[ 84.310082] pci 0000:02:00.0: PCI INT A -> Link[LNKB] -> GSI 7
(level, low) -> IRQ 7
[ 85.581946] pci 0000:02:00.0: [1234:0002] type 0 class 0x001180
[ 85.582300] pci 0000:02:00.0: reg 10: [mem 0x40240000-0x40240fff
64bit pref]
[ 85.582467] pci 0000:02:00.0: reg 18: [mem 0x40200000-0x4023ffff
64bit pref]
[ 85.582631] pci 0000:02:00.0: reg 20: [mem 0x40241000-0x40241fff
64bit pref]
[ 85.584195] pci 0000:00:01.0: BAR 6: [??? 0x00000000 flags 0x2] has
bogus alignment
[ 85.584442] pci 0000:02:00.0: BAR 2: assigned [mem
0x40200000-0x4023ffff 64bit pref]
[ 85.584682] pci 0000:02:00.0: BAR 2: set to [mem
0x40200000-0x4023ffff 64bit pref] (PCI address [0x40200000-0x4023ffff])
[ 85.584921] pci 0000:02:00.0: BAR 0: assigned [mem
0x40240000-0x40240fff 64bit pref]
[ 85.585346] pci 0000:02:00.0: BAR 0: set to [mem
0x40240000-0x40240fff 64bit pref] (PCI address [0x40240000-0x40240fff])
[ 85.585587] pci 0000:02:00.0: BAR 4: assigned [mem
0x40241000-0x40241fff 64bit pref]
[ 85.585826] pci 0000:02:00.0: BAR 4: set to [mem
0x40241000-0x40241fff 64bit pref] (PCI address [0x40241000-0x40241fff])
[ 86.915177] pci 0000:02:00.0: PCI INT A -> Link[LNKB] -> GSI 7
(level, low) -> IRQ 7
****************************************************************************************************
Many thanks in advance.
Cheers,
Bjorn Helgaas
It seems like a pciehp bug that you have to rescan explicitly. But
I'm not a pciehp expert and I haven't looked at the code.
>>> * But our software returns : 0xFFFFFFFF
This is reading from your device's memory region. It looks like the
device is configured correctly (BAR values are reasonable and memory
decoding is enabled) and I assume the bridges leading to it are
configured correctly (if you posted the complete dmesg log we could
tell for sure). After that point, Linux is out of the way and it's
just a question of whether your device responds correctly to the
memory access.
Are you sure the device is supposed to return something other than
0xFFFFFFFF? If it's memory, can you write to it and read the new
value back? Can you use a PCIe analyzer and see if the device is
responding correctly on the bus?
Other than the possible pciehp rescan problem, this just looks like a
problem with your FPGA.
Bjorn
Kenji Kaneshige
Can you try echo 1 > /sys/bus/pci/slots/XXX/power?
(XXX: slot number)
> It seems like a pciehp bug that you have to rescan explicitly. But
> I'm not a pciehp expert and I haven't looked at the code.
The pciehp automatically scans the bus on presence changed event
(e.g. board is pluged in) if the hot-plug controller supports
surprise removal. Otherwise, you need to power on slot explicitly
by "echo 1 > /sys/bus/pci/slots/XXX/power". So one possibility is
that your controller doesn't support surprise removal. We can
check it by looking at the pciehp's debug output. Can you send
whole dmesg output?
Regards,
Kenji Kaneshige
>
>>>> * But our software returns : 0xFFFFFFFF
>
> This is reading from your device's memory region. It looks like the
> device is configured correctly (BAR values are reasonable and memory
> decoding is enabled) and I assume the bridges leading to it are
> configured correctly (if you posted the complete dmesg log we could
> tell for sure). After that point, Linux is out of the way and it's
> just a question of whether your device responds correctly to the
> memory access.
>
> Are you sure the device is supposed to return something other than
> 0xFFFFFFFF? If it's memory, can you write to it and read the new
> value back? Can you use a PCIe analyzer and see if the device is
> responding correctly on the bus?
>
> Other than the possible pciehp rescan problem, this just looks like a
> problem with your FPGA.
>
> Bjorn
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
Abdelghani Ouchabane
>>>> * But our software returns : 0xFFFFFFFF
> This is reading from your device's memory region. It looks like the
> device is configured correctly (BAR values are reasonable and memory
> decoding is enabled) and I assume the bridges leading to it are
> configured correctly (if you posted the complete dmesg log we could
> tell for sure). After that point, Linux is out of the way and it's
> just a question of whether your device responds correctly to the
> memory access.
> Are you sure the device is supposed to return something other than
> 0xFFFFFFFF?
Yes, the device should return something similar to :
0xFDFFE000 0x00000000 0x00000000 0x00000000
0xFDFFE008 0x00000008 0x00000000 0x00000000
0xFDFFE010 0x00000010 0x00000001 0x00020000
0xFDFFE018 0x00000018 0x00000000 0x00000000
0xFDFFE020 0x00000020 0x00000000 0x00000000
0xFDFFE028 0x00000028 0x00000000 0x00000000
0xFDFFE030 0x00000030 0x00000000 0x00000000
0xFDFFE038 0x00000038 0x00000000 0x00000000
0xFDFFE040 0x00000040 0x30012009 0x00101449
0xFDFFE048 0x00000048 0x00000000 0x00000781
0xFDFFE050 0x00000050 0x00000000 0x00000300
0xFDFFE058 0x00000058 0xCAFEBABE 0xDEADBEEF
> If it's memory, can you write to it and read the new
> value back? Can you use a PCIe analyzer and see if the device is
> responding correctly on the bus?
0xFFFFFFFF
> Other than the possible pciehp rescan problem, this just looks like a
> problem with your FPGA.
I attached to you the whole dmesg log.
Cheers,
Ghani
Abdelghani Ouchabane
>>> After I plug my board in, I executed echo 1> /sys/bus/pci/rescan
>>
>>
>
> Can you try echo 1 > /sys/bus/pci/slots/XXX/power?
> (XXX: slot number)
I am still having the same problem after executing "echo 1 >
/sys/bus/pci/slots/0000\:02\:00.0/power"
>
>> It seems like a pciehp bug that you have to rescan explicitly. But
>> I'm not a pciehp expert and I haven't looked at the code.
>
> The pciehp automatically scans the bus on presence changed event
> (e.g. board is pluged in) if the hot-plug controller supports
> surprise removal. Otherwise, you need to power on slot explicitly
> by "echo 1 > /sys/bus/pci/slots/XXX/power". So one possibility is
> that your controller doesn't support surprise removal. We can
> check it by looking at the pciehp's debug output. Can you send
> whole dmesg output?
Bjorn Helgaas
Maybe the difference between this working scenario and the broken
scenario will have a clue.
>> If it's memory, can you write to it and read the new
>> value back? Can you use a PCIe analyzer and see if the device is
>> responding correctly on the bus?
>
> I have tried to write to the FPGA registers but I was always getting
> 0xFFFFFFFF
>>
>> Other than the possible pciehp rescan problem, this just looks like a
>> problem with your FPGA.
>
> Can it have a relation with the BIOS?
> I attached to you the whole dmesg log.
relevant from your dmesg log:
ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
pci_root PNP0A08:00: host bridge window [mem 0x40000000-0xffffffff]
pci 0000:00:05.0: PCI bridge to [bus 02-02]
pci 0000:00:05.0: bridge window [mem 0x40000000-0x401fffff]
pci 0000:00:05.0: bridge window [mem 0x40200000-0x403fffff 64bit pref]
pci 0000:02:00.0: BAR 2: assigned [mem 0x40200000-0x4023ffff 64bit pref]
Linux allocated and enabled the windows as above, and we assigned BARs
of device 02:00.0 inside those windows. As far as I can tell,
everything leading to the FPGA is set up correctly.
Can you try another, known-working (not your FPGA), card in that slot?
It still looks to me like a problem with your FPGA.
Kenji Kaneshige
According to your dmesg output, pciehp driver doesn't detect any PCIe
hotplug slot. Can you send the following information?
- ls -lR /sys/bus/pci/slots
- lspci -vvvv (as root user)
Regards,
Kenji Kaneshige
Abdelghani Ouchabane
> How did you get this read to work? Is this in a different system?
> Maybe the difference between this working scenario and the broken
> scenario will have a clue.
yes, it was on a different system. I will give you more details:
Our old systems are based on the Congatec COM Express conga-B945 (
based on the Intel Mobile 945GME chipset ), and we are using a GPO line
to control the power switch on the FPGA board that is connected using
PCIe line 0. On the B945 this line is high by default during the boot (
The FPGA board gets powered while the system boots "this is the desired
behaviour" ).
But now we are working to use the new Congatec COM Express conga-BAF
( based on the AMD Embedded G-Series Processors ), The GPO line is zero
by default during the boot. So, after the system boots, we power the
FPGA board on, but unfortunately the system can not access the FPGA
registers.
We are trying to solve this issue and get the FPGA board works
without changing the hardware.
> The BIOS left the bridge to bus 02 with all windows disabled, but
> Linux allocated and enabled the windows as above, and we assigned BARs
> of device 02:00.0 inside those windows. As far as I can tell,
> everything leading to the FPGA is set up correctly. Can you try
> another, known-working (not your FPGA), card in that slot? It still
> looks to me like a problem with your FPGA. Bjorn
Yes, I did the following test cases with a standard WiFi card ( Network
controller: Ralink corp. RT2860 --- Belkin Device 817c --- Kernel
modules: rt2800pci ):
A - I booted the system without the express card.
1 - Introduce the express card after the system has booted up.
2 - lspci -vt doesn't show the express card, express card
divers did not loaded (lsmod).
3 - Rescan the PCIe bus "echo 1 > /sys/bus/pci/rescan"
4 - The card gets detected (lspci), drivers get loaded
(lsmod), But ifconfig doesn't show the interface (wlan0).
dmesg :
[ 159.849763] pci 0000:01:00.0:
[1814:0781] type 0 class 0x000280
[ 159.849932] pci 0000:01:00.0: reg 10:
[mem 0x00000000-0x0000ffff]
[ 159.850220] pci 0000:01:00.0: PME#
supported from D0 D3hot
[ 159.850350] pci 0000:01:00.0: PME# disabled
[ 159.850601] radeon 0000:00:01.0: BAR 6:
[??? 0x00000000 flags 0x2] has bogus alignment
[ 159.850787] pci 0000:01:00.0: BAR 0:
assigned [mem 0x40400000-0x4040ffff]
[ 159.850941] pci 0000:01:00.0: BAR 0:
set to [mem 0x40400000-0x4040ffff] (PCI address [0x40400000-0x4040ffff])
[ 159.879110] cfg80211: Calling CRDA to
update world regulatory domain
[ 159.920212] rt2800pci 0000:01:00.0:
enabling device (0000 -> 0002)
[ 159.920379] rt2800pci 0000:01:00.0: PCI
INT A -> GSI 16 (level, low) -> IRQ 16
[ 159.920551] rt2800pci 0000:01:00.0:
setting latency timer to 64
[ 159.930927] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 159.941346] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 159.951737] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 159.962130] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 159.972521] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 159.982907] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 159.993299] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 160.003676] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 160.014056] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 160.024447] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 160.034825] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 160.045230] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 160.055622] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 160.066014] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 160.079940] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 160.093825] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 160.107635] phy0 ->
rt2x00pci_regbusy_read: Error - Indirect register access failed:
offset=0x00000580, value=0xffffffff
[ 160.111266] phy0 -> rt2800_init_eeprom:
Error - Invalid RT chipset detected.
[ 160.114860] phy0 ->
rt2x00lib_probe_dev: Error - Failed to allocate device.
[ 160.118496] rt2800pci 0000:01:00.0: PCI
INT A disabled
B - I booted the system with express card.
1 - Insert the express card when the system is off.
2 - Boot the system.
3 - The card gets detected (lspci), drivers get loaded
(lsmod), But ifconfig shows the interface(wlan0).
C - I booted the system with express card then removed it then
inserted it.
1 - Insert the express card when the system is off.
2 - Boot the system
3 - The card gets detected (lspci), drivers get loaded
(lsmod), But ifconfig shows the interface(wlan0).
4 - Remove from pci bus "echo 1 >
/sys/bus/pci/devices/0000\:0?\:00.0/remove" - NOTE = ? depends on the slot.
dmesg :
[ 69.510267] rt2800pci 0000:01:00.0: PCI
INT A disabled
5 - Remove the card.
6 - Plug the card back in.
7 - Rescan the bus "echo 1 > /sys/bus/pci/rescan".
8 - The card gets detected (lspci), drivers get loaded
(lsmod), But ifconfig shows the interface(wlan1).
dmesg :
[ 208.328481] pci 0000:01:00.0:
[1814:0781] type 0 class 0x000280
[ 208.328658] pci 0000:01:00.0: reg 10:
[mem 0x00000000-0x0000ffff]
[ 208.328910] pci 0000:01:00.0: PME#
supported from D0 D3hot
[ 208.329068] pci 0000:01:00.0: PME# disabled
[ 208.329328] radeon 0000:00:01.0: BAR 6:
[??? 0x00000000 flags 0x2] has bogus alignment
[ 208.329516] pci 0000:01:00.0: BAR 0:
assigned [mem 0xfea00000-0xfea0ffff]
[ 208.329671] pci 0000:01:00.0: BAR 0:
set to [mem 0xfea00000-0xfea0ffff] (PCI address [0xfea00000-0xfea0ffff])
[ 208.330511] rt2800pci 0000:01:00.0:
enabling device (0000 -> 0002)
[ 208.330665] rt2800pci 0000:01:00.0: PCI
INT A -> GSI 16 (level, low) -> IRQ 16
[ 208.330834] rt2800pci 0000:01:00.0:
setting latency timer to 64
[ 208.341348] ieee80211 phy1: Selected
rate control algorithm 'minstrel_ht'
[ 208.342396] Registered led device:
rt2800pci-phy1::radio
[ 208.342575] Registered led device:
rt2800pci-phy1::assoc
[ 208.342752] Registered led device:
rt2800pci-phy1::quality
[ 208.428306] udev[1339]: renamed network
interface wlan0 to wlan1
As you can see that my system doesn't detect the express card in test
case "A" and the values of registers are 0xffffffff, the same as in my
FPGA board.
Cheers,
Ghani
Bjorn Helgaas
If I understand correctly:
A - fails (card not present at power-on, added later)
B - works (card always present)
C - works (card present at power-on, later removed and re-added,
requires manual rescan)
I see this note in section 9.4.8 (BIOS setup) of the conga-BAF User's Guide:
Note: Unless the hotplug support for this port is enabled as well,
an unpopulated
port will still be disabled if no PCI Express device is connected.
If you don't have hotplug enabled in the BIOS setup, it sounds like it
might result in the behavior you're seeing. Do you have that enabled?
Bjorn
Kenji Kaneshige
I think as follows based on the info.
- There are two hot-plug capable PCIe slots on your machine.
- But, it seems you are using fakephp driver, not pciehp (is that
correct?). The fakephp cannot handle hot-plug event such as presence
change event on the slot. This is why the bus is not scanned automatically.
- Unfortunately, the bus would not be scanned automatically even if you
use pciehp. As I told you in the previous email, current pciehp don't
scan the bus automatically only if the slot is hot-plug surprise
capable. According to the lspci output, your hotplug controller is not
hot-plug surprise capable.
- I don't think pciehp solve invalid register read problem. According to
the lspci output, power controller capability isn't present on your
hotplug controller. On such environment, pciehp driver does almost the
same thing as fakephp does (just scan the bus/remove the pci device data
structure) except hot-plug event handling.
But it's worth whole trying pciehp.
By the way, have you tried acpiphp? It might help you.
Regards,
Kenji Kaneshige
(2011/10/12 17:35), Abdelghani Ouchabane wrote:
> On 12/10/11 08:36, Kenji Kaneshige wrote:
>> Hello,
>>
>> According to your dmesg output, pciehp driver doesn't detect any PCIe
>> hotplug slot. Can you send the following information?
>>
>> - ls -lR /sys/bus/pci/slots
>> - lspci -vvvv (as root user)
>
> Hallo,
>
> - I tested my system with:
> * A standard WiFi card (Network controller: Ralink corp. RT2860 ---
> Belkin Device 817c --- Kernel modules: rt2800pci).
> * My FPGA board.
>
> - I did the following:
>
> * I booted the system without the express card& without my FPGA board.
> 1 - Introduce the express card& my FPGA board after the system has
> booted up.
> 2 - lspci -vt doesn't show the express card& FPGA board, express card
> divers did not loaded (lsmod).
> 3 - lspci -vt doesn't show my FPGA board.
> 4 - Rescan the PCIe bus "echo 1> /sys/bus/pci/rescan"
> 5 - The express card gets detected (lspci), drivers get loaded (lsmod),
> But ifconfig doesn't show the interface (wlan0).
> 6 - My FPGA board gets detected (lspci), but I could not access its
> registers 0xffffffff.
> dmesg :
> [ 152.045433] pci 0000:01:00.0: [1814:0781] type 0 class 0x000280
> [ 152.045602] pci 0000:01:00.0: reg 10: [mem 0x00000000-0x0000ffff]
> [ 152.045854] pci 0000:01:00.0: PME# supported from D0 D3hot
> [ 152.045980] pci 0000:01:00.0: PME# disabled
> [ 152.046263] radeon 0000:00:01.0: BAR 6: [??? 0x00000000 flags 0x2] has
> bogus alignment
> [ 152.046452] pci 0000:01:00.0: BAR 0: assigned [mem 0x40400000-0x4040ffff]
> [ 152.046606] pci 0000:01:00.0: BAR 0: set to [mem
> 0x40400000-0x4040ffff] (PCI address [0x40400000-0x4040ffff])
> [ 152.075665] cfg80211: Calling CRDA to update world regulatory domain
> [ 152.116429] rt2800pci 0000:01:00.0: enabling device (0000 -> 0002)
> [ 152.116598] rt2800pci 0000:01:00.0: PCI INT A -> GSI 16 (level, low)
> -> IRQ 16
> [ 152.116770] rt2800pci 0000:01:00.0: setting latency timer to 64
> [ 152.127146] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.137537] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.147926] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.158332] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.168723] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.179115] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.189489] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.199864] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.210256] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.220634] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.231088] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.241463] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.251865] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.262270] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.276185] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.290055] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.303865] phy0 -> rt2x00pci_regbusy_read: Error - Indirect register
> access failed: offset=0x00000580, value=0xffffffff
> [ 152.307477] phy0 -> rt2800_init_eeprom: Error - Invalid RT chipset
> detected.
> [ 152.311073] phy0 -> rt2x00lib_probe_dev: Error - Failed to allocate
> device.
> [ 152.314700] rt2800pci 0000:01:00.0: PCI INT A disabled
> [ 172.830318] pci 0000:02:00.0: [1234:0002] type 0 class 0x001180
> [ 172.833754] pci 0000:02:00.0: reg 10: [mem 0x00000000-0x00000fff 64bit
> pref]
> [ 172.837127] pci 0000:02:00.0: reg 18: [mem 0x00000000-0x0003ffff 64bit
> pref]
> [ 172.840456] pci 0000:02:00.0: reg 20: [mem 0x00000000-0x00000fff 64bit
> pref]
> [ 172.843964] radeon 0000:00:01.0: BAR 6: [??? 0x00000000 flags 0x2] has
> bogus alignment
> [ 172.847331] pci 0000:02:00.0: BAR 2: assigned [mem
> 0x40200000-0x4023ffff 64bit pref]
> [ 172.850728] pci 0000:02:00.0: BAR 2: set to [mem 0x40200000-0x4023ffff
> 64bit pref] (PCI address [0x40200000-0x4023ffff])
> [ 172.854231] pci 0000:02:00.0: BAR 0: assigned [mem
> 0x40240000-0x40240fff 64bit pref]
> [ 172.857771] pci 0000:02:00.0: BAR 0: set to [mem 0x40240000-0x40240fff
> 64bit pref] (PCI address [0x40240000-0x40240fff])
> [ 172.861339] pci 0000:02:00.0: BAR 4: assigned [mem
> 0x40241000-0x40241fff 64bit pref]
> [ 172.864997] pci 0000:02:00.0: BAR 4: set to [mem 0x40241000-0x40241fff
> 64bit pref] (PCI address [0x40241000-0x40241fff])
>
>
> 7 - Reset the express card "echo 1>
> /sys/bus/pci/devices/0000\:01\:00.0/reset". But ifconfig still doesn't
> show the interface (wlan0)
> dmesg :
> [ 731.717117] pci 0000:01:00.0: restoring config space at offset 0x1
> (was 0x100400, writing 0x100002)
>
> 8 - Reset my FPGA board "echo 1>
> /sys/bus/pci/devices/0000\:02\:00.0/reset". Still I could not access
> into my FPGA registers.
> logs :
> [root@localhost ~]# echo 1> /sys/bus/pci/devices/0000\:02\:00.0/reset
> -bash: echo: write error: Invalid argument
> dmesg :
> [ 910.928564] pci 0000:02:00.0: restoring config space at offset 0x1
> (was 0x100400, writing 0x100000)
>
>
> * I have attached to you the requested information.
>
> Many thanks,
Abdelghani Ouchabane
> Thanks for the details.
>
> If I understand correctly:
>
> A - fails (card not present at power-on, added later)
> B - works (card always present)
> C - works (card present at power-on, later removed and re-added,
> requires manual rescan)
>
> I see this note in section 9.4.8 (BIOS setup) of the conga-BAF User's Guide:
>
> Note: Unless the hotplug support for this port is enabled as well,
> an unpopulated
> port will still be disabled if no PCI Express device is connected.
>
> If you don't have hotplug enabled in the BIOS setup, it sounds like it
> might result in the behavior you're seeing. Do you have that enabled?
>
> Bjorn
Hallo Bjorn & Kenji,
the hot-plug was enabled, but today I have received a new BIOS from
Congatec which solves the problem.
Among the fix is:
_BBRAR009 to BBRAR110:_
2. Enabled PCIExpress / ExpressCard basic hotplug support. Enabled event
handling support.
Now after the manual rescan my FPGA board gets detect correctly and I
can access its registers in the correct way.
Many thanks for all your supports.
My best regards,
Ghani
Abdelghani Ouchabane
> Thank you for the information. Though I don't have any good news for you,
> I think as follows based on the info.
>
> - There are two hot-plug capable PCIe slots on your machine.
>
> - But, it seems you are using fakephp driver, not pciehp (is that
> correct?). The fakephp cannot handle hot-plug event such as presence
> change event on the slot. This is why the bus is not scanned automatically.
>
> - Unfortunately, the bus would not be scanned automatically even if you
> use pciehp. As I told you in the previous email, current pciehp don't
> scan the bus automatically only if the slot is hot-plug surprise
> capable. According to the lspci output, your hotplug controller is not
> hot-plug surprise capable.
>
> - I don't think pciehp solve invalid register read problem. According to
> the lspci output, power controller capability isn't present on your
> hotplug controller. On such environment, pciehp driver does almost the
> same thing as fakephp does (just scan the bus/remove the pci device data
> structure) except hot-plug event handling.
>
> But it's worth whole trying pciehp.
> By the way, have you tried acpiphp? It might help you.
>
> Regards,
> Kenji Kaneshige
>
Hallo Kenji,
many thanks for your great supports. The new BIOS from Congatec solves
the problem.
I using both fakephp & pciehp drivers, Can I use both drivers at the
same time?
I am using fakephp because I need "/sys/bus/pci/slots/0000\:02\:00.0/power".
Other thing: my Kernel has "CONFIG_ACPI_PROC_EVENT is not set", does
this explain why the scan is not performed automatically?
[root@localhost ~]# modprobe acpiphp
FATAL: Error inserting acpiphp
(/lib/modules/2.6.40.3-0.119.delos.i686/kernel/drivers/pci/hotplug/acpiphp.ko):
No such device
What is the advantage to use acpiphp ?
Cheers,
Kenji Kaneshige
I'm sorry for the long delay.
> same time?
I think, "no".
> I am using fakephp because I need "/sys/bus/pci/slots/0000\:02\:00.0/power".
>
> Other thing: my Kernel has "CONFIG_ACPI_PROC_EVENT is not set", does
> this explain why the scan is not performed automatically?
So I don't think there is no way to detect presence change event on the slot.
>
> [root@localhost ~]# modprobe acpiphp
> FATAL: Error inserting acpiphp
> (/lib/modules/2.6.40.3-0.119.delos.i686/kernel/drivers/pci/hotplug/acpiphp.ko):
> No such device
>
> What is the advantage to use acpiphp ?
(like pciehp). The 'pciehp_force' specified your environment is to load pciehp
forcibly on such platform. So I thought acpiphp might solve your problem.
Regards,
Kenji Kaneshige
>
>
> Cheers,
> Ghani
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
Abdelghani Ouchabane
Hallo Kenji,
many thanks for your great supports. The new BIOS from Congatec solves
the problem.
I using both fakephp& pciehp drivers, Can I use both drivers at the
same time?
> I think, "no".
>
>> I am using fakephp because I need "/sys/bus/pci/slots/0000\:02\:00.0/power".
>>
>> Other thing: my Kernel has "CONFIG_ACPI_PROC_EVENT is not set", does
>> this explain why the scan is not performed automatically?
> Fakephp driver doesn't have any corresponding hotplug controller (hardware).
> So I don't think there is no way to detect presence change event on the slot.
>
>> [root@localhost ~]# modprobe acpiphp
>> FATAL: Error inserting acpiphp
>> (/lib/modules/2.6.40.3-0.119.delos.i686/kernel/drivers/pci/hotplug/acpiphp.ko):
>> No such device
>>
>> What is the advantage to use acpiphp ?
> Some platform only allows acpiphp, but doesn't allow PCIe native hotplug driver
> (like pciehp). The 'pciehp_force' specified your environment is to load pciehp
> forcibly on such platform. So I thought acpiphp might solve your problem.
>
> Regards,
> Kenji Kaneshige
>
many thanks for your support, my system is working perfectly.
Cheers,
Ghani
--