Linux PCI subsystem discussion ()

Pratyush Anand | 4 Jan 2012 12:01

Re: Detected Hardware Unit Hang on Intel Wired Ethernet

Adding PCI mailing list too, as problem is coming only when MSI is enabled.

If I connect an PCIe analyzer, I see that at the time of issue MRd(64) 
for 32 words has been issued with a wrong 64 bit address from ethernet 
card to my RC.
In the normal course it always issues MRd(32) only.

Regards
Pratyush

On 1/4/2012 3:18 PM, Pratyush Anand wrote:
> Hi All,
>
> I am trying to use PCIe based Intel PRO/1000 PT Server Adapter card on
> an ARM Corterx A-9 based platform.
> I am using linux 2.6.37.
>
> I am trying to mount my rootfile system on NFS using interface of this card.
>
> I see follwing issue while nfs mount
> ----------------------------------------------------------------------------------------------------------------------------------------------------------
> IP-Config: Complete:
>       device=eth0, addr=192.168.1.10, mask=255.255.255.0, gw=255.255.255.255,
>       host=192.168.1.10, domain=, nis-domain=(none),
>       bootserver=192.168.1.1, rootserver=192.168.1.1, rootpath=
> NFS:1. attempt to mount rootVFS: Mounted root (nfs filesystem) on device 0:13.
> Freeing init memory: 184K
> nfs: server 192.168.1.1 not responding, still trying
> e1000e 0000:03:00.0: eth0: Detected Hardware Unit Hang:
>    TDH<40>

(Continue reading)

Greg KH | 4 Jan 2012 20:22

Re: Detected Hardware Unit Hang on Intel Wired Ethernet

On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote:
> Adding PCI mailing list too, as problem is coming only when MSI is enabled.
> 
> If I connect an PCIe analyzer, I see that at the time of issue
> MRd(64) for 32 words has been issued with a wrong 64 bit address
> from ethernet card to my RC.
> In the normal course it always issues MRd(32) only.

Bug in your pcie firmware controller?

------------------------------------------------------------------------------
Ridiculously easy VDI. With Citrix VDI-in-a-Box, you don't need a complex
infrastructure or vast IT resources to deliver seamless, secure access to
virtual desktops. With this all-in-one solution, easily deploy virtual 
desktops for less than the cost of PCs and save 60% on VDI infrastructure 
costs. Try it free! http://p.sf.net/sfu/Citrix-VDIinabox
_______________________________________________
E1000-devel mailing list
E1000-devel <at> lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel&#174; Ethernet, visit http://communities.intel.com/community/wired

Pratyush Anand | 5 Jan 2012 05:30

Re: Detected Hardware Unit Hang on Intel Wired Ethernet

On 1/5/2012 12:52 AM, Greg KH wrote:
> On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote:
>> Adding PCI mailing list too, as problem is coming only when MSI is enabled.
>>
>> If I connect an PCIe analyzer, I see that at the time of issue
>> MRd(64) for 32 words has been issued with a wrong 64 bit address
>> from ethernet card to my RC.
>> In the normal course it always issues MRd(32) only.
>
> Bug in your pcie firmware controller?
>
> .
>

when you say "Bug in your pcie firmware controller?", is it RC's 
software or EP's software?

Here I am pasting a part of analyzer log converted into text.
Packet(177940), is an upstream request for MSI. Whenever any device 
writes at address 0x58A8F8, my PCIe RC considers it as MSI and generates 
an interrupt. So I receive MSI interrupt correctly in my software. Also 
MSI controller is correctly able to point me that the interrupt is from 
ethernet card.

Now in Packet(178010), ethernet controller sends another upstream 
request for MRd(64) of 32 dwords with Address(AFECEB87:A9D88B00).Since, 
this address does not exist in my RC's world so, an UR is returned and 
hence the problem occurs.

Now, question is, why ethernet card is generating inbound request with

(Continue reading)

Greg KH | 5 Jan 2012 17:46

Re: Detected Hardware Unit Hang on Intel Wired Ethernet

On Thu, Jan 05, 2012 at 10:00:39AM +0530, Pratyush Anand wrote:
> On 1/5/2012 12:52 AM, Greg KH wrote:
> >On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote:
> >>Adding PCI mailing list too, as problem is coming only when MSI is enabled.
> >>
> >>If I connect an PCIe analyzer, I see that at the time of issue
> >>MRd(64) for 32 words has been issued with a wrong 64 bit address
> >>from ethernet card to my RC.
> >>In the normal course it always issues MRd(32) only.
> >
> >Bug in your pcie firmware controller?
> >
> >.
> >
> 
> when you say "Bug in your pcie firmware controller?", is it RC's
> software or EP's software?

I don't know, but I would place the bet on your motherboard controller,
not the device.

greg k-h

Dave, Tushar N | 6 Jan 2012 00:54

Picon

RE: Detected Hardware Unit Hang on Intel Wired Ethernet

Anand,

Sorry to hear that you have this issue with card. And yeah, thanks for doing the debugging and providing the
bus trace. 
I think we should run the debug driver that prints the HW ring details when hang occurs. I can provide you a
debug driver. You can then install debug driver and also let the bus tracer running. Once the issue occurs,
provide me the full dmesg output (that has HW ring details) and bus trace. 

Tell me which card you have, 1gig or 10gig? Which driver are you running e1000e or igb or ixgbe?
Can you also provide ethtool -i ethx output.

Once I know which driver, I send you debug driver. 

Thanks.

-Tushar

-----Original Message-----
From: netdev-owner <at> vger.kernel.org [mailto:netdev-owner <at> vger.kernel.org] On Behalf Of Pratyush Anand
Sent: Wednesday, January 04, 2012 8:31 PM
To: Greg KH
Cc: Pratyush Anand; e1000-devel <at> lists.sourceforge.net; netdev <at> vger.kernel.org; Shiraz HASHIM;
Deepak SIKRI; Bhavna YADAV; linux-pci <at> vger.kernel.org; Linux NICS
Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet

On 1/5/2012 12:52 AM, Greg KH wrote:
> On Wed, Jan 04, 2012 at 04:31:36PM +0530, Pratyush Anand wrote:
>> Adding PCI mailing list too, as problem is coming only when MSI is enabled.
>>
>> If I connect an PCIe analyzer, I see that at the time of issue

(Continue reading)

Pratyush Anand | 6 Jan 2012 05:25

Re: Detected Hardware Unit Hang on Intel Wired Ethernet

Thanks Tushar,

On 1/6/2012 5:24 AM, Dave, Tushar N wrote:
> Anand,
>
> Sorry to hear that you have this issue with card. And yeah, thanks for doing the debugging and providing the
bus trace.
> I think we should run the debug driver that prints the HW ring details when hang occurs. I can provide you a
debug driver. You can then install debug driver and also let the bus tracer running. Once the issue occurs,
provide me the full dmesg output (that has HW ring details) and bus trace.
>
> Tell me which card you have, 1gig or 10gig? Which driver are you running e1000e or igb or ixgbe?
> Can you also provide ethtool -i ethx output.
>
> Once I know which driver, I send you debug driver.

I am using Intel PRO/1000 PT Server Adapter.
http://www.intel.com/content/www/us/en/network-adapters/gigabit-network-adapters/pro-1000-pt.html

I am using e1000e driver.

I see the problem when I try to mount rootfilesystem using NFS and use 
MSI interrupt. I see this issue even before I can have cell prompt. 
Please see first mail in this thread.

http://www.mail-archive.com/e1000-devel <at> lists.sourceforge.net/msg04894.html

Here, you can also see tx ring details when issue occur.
Please let me know, if you need any more info.

(Continue reading)

Dave, Tushar N | 6 Jan 2012 19:55

Picon

RE: Detected Hardware Unit Hang on Intel Wired Ethernet

Pratyush,

Sorry I got your name reversed.
Are you using in-kernel driver or one from Sourceforge.
Please send me output of ethtool -i ethx.

-Tushar

-----Original Message-----
From: Pratyush Anand [mailto:pratyush.anand <at> st.com] 
Sent: Thursday, January 05, 2012 8:25 PM
To: Dave, Tushar N
Cc: Greg KH; Pratyush Anand; e1000-devel <at> lists.sourceforge.net; netdev <at> vger.kernel.org; Shiraz
HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci <at> vger.kernel.org; Linux NICS
Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet

Thanks Tushar,

On 1/6/2012 5:24 AM, Dave, Tushar N wrote:
> Anand,
>
> Sorry to hear that you have this issue with card. And yeah, thanks for doing the debugging and providing the
bus trace.
> I think we should run the debug driver that prints the HW ring details when hang occurs. I can provide you a
debug driver. You can then install debug driver and also let the bus tracer running. Once the issue occurs,
provide me the full dmesg output (that has HW ring details) and bus trace.
>
> Tell me which card you have, 1gig or 10gig? Which driver are you running e1000e or igb or ixgbe?
> Can you also provide ethtool -i ethx output.
>

(Continue reading)

Pratyush Anand | 10 Jan 2012 05:20

Re: Detected Hardware Unit Hang on Intel Wired Ethernet

On 1/7/2012 12:25 AM, Dave, Tushar N wrote:
> Pratyush,
>
> Sorry I got your name reversed.
> Are you using in-kernel driver or one from Sourceforge.

I am using in-kernel driver from kernel 2.6.37.

> Please send me output of ethtool -i ethx.

root <at> 192.168.1.10:~# ethtool -i eth0
driver: e1000e
version: 1.2.7-k2
firmware-version: 5.11-8
bus-info: 0000:01:00.0

Regards
Pratyush

>
> -Tushar
>
> -----Original Message-----
> From: Pratyush Anand [mailto:pratyush.anand <at> st.com]
> Sent: Thursday, January 05, 2012 8:25 PM
> To: Dave, Tushar N
> Cc: Greg KH; Pratyush Anand; e1000-devel <at> lists.sourceforge.net; netdev <at> vger.kernel.org; Shiraz
HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci <at> vger.kernel.org; Linux NICS
> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet
>

(Continue reading)

Dave, Tushar N | 11 Jan 2012 02:10

Picon

RE: Detected Hardware Unit Hang on Intel Wired Ethernet

Thanks for driver info.
Because you are running in-kernel driver, we can enable the debug message level via ethtool. That will
print HW ring info when issue occurs.

Here is the ethtool command to enable debug messages.
# ethtool -s ethx msglvl 0x3c00
This will enable tx_done, rx_status, pktdata and hw message levels.
You can confirm it by typing ethtool ethx , this will show you 'Current message level'

Next time when issue occurs, please send me the full dmesg log after the issue occurred along with the bus trace.

Thanks.

-Tushar

-----Original Message-----
From: Pratyush Anand [mailto:pratyush.anand <at> st.com] 
Sent: Monday, January 09, 2012 8:21 PM
To: Dave, Tushar N
Cc: Greg KH; Pratyush Anand; e1000-devel <at> lists.sourceforge.net; netdev <at> vger.kernel.org; Shiraz
HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci <at> vger.kernel.org; Linux NICS
Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet

On 1/7/2012 12:25 AM, Dave, Tushar N wrote:
> Pratyush,
>
> Sorry I got your name reversed.
> Are you using in-kernel driver or one from Sourceforge.

I am using in-kernel driver from kernel 2.6.37.

(Continue reading)

Pratyush Anand | 11 Jan 2012 04:33

Re: Detected Hardware Unit Hang on Intel Wired Ethernet

On 1/11/2012 6:40 AM, Dave, Tushar N wrote:
> Thanks for driver info.
> Because you are running in-kernel driver, we can enable the debug message level via ethtool. That will
print HW ring info when issue occurs.
>
> Here is the ethtool command to enable debug messages.
> # ethtool -s ethx msglvl 0x3c00
> This will enable tx_done, rx_status, pktdata and hw message levels.
> You can confirm it by typing ethtool ethx , this will show you 'Current message level'
>
> Next time when issue occurs, please send me the full dmesg log after the issue occurred along with the bus trace.

As I said earlier, issue is reproducible if I try to keep my 
rootfilesystem  over NFS. So, after the booting, kernel tries to mount 
rootfs over NFS and it crashes. So, I see issue even before I can reach 
to # prompt. How can I use "ethtool -s ethx msglvl 0x3c00" to enable any 
debug message. May be I can directly change in kernel code to enable this.

Regards
Pratyush
>
> Thanks.
>
> -Tushar
>
>
> -----Original Message-----
> From: Pratyush Anand [mailto:pratyush.anand <at> st.com]
> Sent: Monday, January 09, 2012 8:21 PM
> To: Dave, Tushar N

(Continue reading)

Dave, Tushar N | 11 Jan 2012 04:48

Picon

Re: Detected Hardware Unit Hang on Intel Wired Ethernet

>As I said earlier, issue is reproducible if I try to keep my 
>rootfilesystem  over NFS. So, after the booting, kernel tries to mount 
>rootfs over NFS and it crashes. So, I see issue even before I can reach 
>to # prompt. How can I use "ethtool -s ethx msglvl 0x3c00" to enable any 
>debug message. May be I can directly change in kernel code to enable this.

Yes either you can modify in-kernel driver code  Or I will provide you the latest e1000e out of tree driver
from http://sourceforge.net/projects/e1000/files/ with debug info enabled.

-Tushar

>
> Thanks.
>
> -Tushar
>
>
> -----Original Message-----
> From: Pratyush Anand [mailto:pratyush.anand <at> st.com]
> Sent: Monday, January 09, 2012 8:21 PM
> To: Dave, Tushar N
> Cc: Greg KH; Pratyush Anand; e1000-devel <at> lists.sourceforge.net; netdev <at> vger.kernel.org; Shiraz
HASHIM; Deepak SIKRI; Bhavna YADAV; linux-pci <at> vger.kernel.org; Linux NICS
> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet
>
> On 1/7/2012 12:25 AM, Dave, Tushar N wrote:
>> Pratyush,
>>
>> Sorry I got your name reversed.
>> Are you using in-kernel driver or one from Sourceforge.

(Continue reading)

Dave, Tushar N | 26 Jan 2012 22:27

Picon

RE: Detected Hardware Unit Hang on Intel Wired Ethernet

>-----Original Message-----
>From: Pratyush Anand [mailto:pratyush.anand <at> st.com]
>Sent: Tuesday, January 10, 2012 7:34 PM
>To: Dave, Tushar N
>Cc: Greg KH; Pratyush Anand; e1000-devel <at> lists.sourceforge.net;
>netdev <at> vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-
>pci <at> vger.kernel.org; Linux NICS
>Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet
>
>As I said earlier, issue is reproducible if I try to keep my
>rootfilesystem  over NFS. So, after the booting, kernel tries to mount
>rootfs over NFS and it crashes. So, I see issue even before I can reach
>to # prompt. How can I use "ethtool -s ethx msglvl 0x3c00" to enable any
>debug message. May be I can directly change in kernel code to enable this.

Any update on this? Did you change in-kernel driver source to print the driver HW ring?
If you did and had reproduced the issue please send me the full dmesg log along with bus trace and I'll take a look.

-Tushar

>> -----Original Message-----
>> From: Pratyush Anand [mailto:pratyush.anand <at> st.com]
>> Sent: Monday, January 09, 2012 8:21 PM
>> To: Dave, Tushar N
>> Cc: Greg KH; Pratyush Anand; e1000-devel <at> lists.sourceforge.net;
>netdev <at> vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-
>pci <at> vger.kernel.org; Linux NICS
>> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet
>>
>> On 1/7/2012 12:25 AM, Dave, Tushar N wrote:

(Continue reading)

Pratyush Anand | 27 Jan 2012 09:03

Re: Detected Hardware Unit Hang on Intel Wired Ethernet

Hello Tushar,

On 1/27/2012 2:57 AM, Dave, Tushar N wrote:
>> -----Original Message-----
>> From: Pratyush Anand [mailto:pratyush.anand <at> st.com]
>> Sent: Tuesday, January 10, 2012 7:34 PM
>> To: Dave, Tushar N
>> Cc: Greg KH; Pratyush Anand; e1000-devel <at> lists.sourceforge.net;
>> netdev <at> vger.kernel.org; Shiraz HASHIM; Deepak SIKRI; Bhavna YADAV; linux-
>> pci <at> vger.kernel.org; Linux NICS
>> Subject: Re: Detected Hardware Unit Hang on Intel Wired Ethernet
>>
>> As I said earlier, issue is reproducible if I try to keep my
>> rootfilesystem  over NFS. So, after the booting, kernel tries to mount
>> rootfs over NFS and it crashes. So, I see issue even before I can reach
>> to # prompt. How can I use "ethtool -s ethx msglvl 0x3c00" to enable any
>> debug message. May be I can directly change in kernel code to enable this.
>
> Any update on this? Did you change in-kernel driver source to print the driver HW ring?
> If you did and had reproduced the issue please send me the full dmesg log along with bus trace and I'll take a look.

I am not able to work with this.
Busy with some other work.
Will get back ,when I again start working with this issue.

Thanks for your support.

Regards
Pratyush

(Continue reading)

Return

Return to gmane.linux.kernel.pci.

Search Archive

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane