headers
Christian Kujau | 1 Jul 2012 09:18
Picon

3.5.0-rc5: BUG: soft lockup - CPU#0 stuck for 22s

Hi,

while trying to upgrade from 3.4.0 to 3.5.0-rc5 on this Powerbook G4 
(powerpc 32 bit), this happens during booting:

--------------
usb 2-1: new full-speed USB device number 4 using ohci_hcd
SCSI subsystem initialized
scsi0 : SBP-2 IEEE-1394
scsi1 : SBP-2 IEEE-1394
firewire_sbp2 fw1.0: logged in to LUN 0000 (0 retries)
scsi 0:0:0:0: Direct-Access     Ext Hard  Disk            PQ: 0 ANSI: 4

[25 seconds later]

BUG: soft lockup - CPU#0 stuck for 22s! [modprobe:695]
Modules linked in: sd_mod arc4 firewire_sbp2 scsi_mod b43 mac80211 cfg80211
irq event stamp: 56975532
hardirqs last  enabled at (56975531): <c053aea8>] _raw_spin_unlock_irqrestore+0x40/0x9c
hardirqs last disabled at (56975532): <c0011688>] reenable_mmu+0x40/0x98
softirqs last  enabled at (56966080): <c000f3f0>] call_do_softirq+0x14/0x24
softirqs last disabled at (56966073): <c000f3f0>] call_do_softirq+0x14/0x24
NIP: c053aeac LR: c053aea8 CTR: c009d510
REGS: eed1fd40 TRAP: 0901   Not tainted  (3.5.0-rc5)
MSR: 00009032 ME IR DR RI > CR: 24042484  XER: 00000000
TASK = eece5c00[695] 'modprobe' THREAD: eed1e000 #012GPR00: 012GPR08: 
NIP [c053aeac] _raw_spin_unlock_irqrestore+0x44/0x9c
LR [c053aea8] _raw_spin_unlock_irqrestore+0x40/0x9c
Call Trace:
[eed1fdf0] [c053aea8] _raw_spin_unlock_irqrestore+0x40/0x9c (unreliable)
(Continue reading)

Permalink | Reply |
headers
Benjamin Herrenschmidt | 2 Jul 2012 06:50

Re: 3.5.0-rc5: BUG: soft lockup - CPU#0 stuck for 22s

On Sun, 2012-07-01 at 00:18 -0700, Christian Kujau wrote:
> Hi,
> 
> while trying to upgrade from 3.4.0 to 3.5.0-rc5 on this Powerbook G4 
> (powerpc 32 bit), this happens during booting:
> 
> --------------
> usb 2-1: new full-speed USB device number 4 using ohci_hcd
> SCSI subsystem initialized
> scsi0 : SBP-2 IEEE-1394
> scsi1 : SBP-2 IEEE-1394
> firewire_sbp2 fw1.0: logged in to LUN 0000 (0 retries)
> scsi 0:0:0:0: Direct-Access     Ext Hard  Disk            PQ: 0 ANSI: 4

Interesting... I observed something roughly similar on a dual G4
the other day associated with a 30s to 1mn pause during boot. RCU
was complaining loudly.

In my case, it did continue booting normally, is that the case for you ?

Also if I compile the kernel without CONFIG_SMP, it did go away as well,
do you observe that too ?

I don't have a spare cycle to investigate this problem this week I'm
afraid, it might help if you could try a bisection though.

Cheers,
Ben.
(Continue reading)

Permalink | Reply |
headers
Christian Kujau | 2 Jul 2012 08:30
Picon

Re: 3.5.0-rc5: BUG: soft lockup - CPU#0 stuck for 22s

On Mon, 2 Jul 2012 at 14:50, Benjamin Herrenschmidt wrote:
> Interesting... I observed something roughly similar on a dual G4
> the other day associated with a 30s to 1mn pause during boot. RCU
> was complaining loudly.
> 
> In my case, it did continue booting normally, is that the case for you ?

No, in my case it stopped booting, though there was no "panic" message.

> Also if I compile the kernel without CONFIG_SMP, it did go away as well,
> do you observe that too ?

This PoweBook G4 is UP anyway and I'm always building w/o CONFIG_SMP.

> I don't have a spare cycle to investigate this problem this week I'm
> afraid, it might help if you could try a bisection though.

Yes, I've started a bisection already and will report back.

FYI, I've put a netconsole-log below from another boot (in the 
middle of a bisection), the instruction dump is slightly more complete.

Thanks!
Christian.

[   40.345973] BUG: soft lockup - CPU#0 stuck for 22s! [modprobe:691]
[   40.347737] Modules linked in: sd_mod arc4 firewire_sbp2 scsi_mod b43 mac80211 cfg80211
[   40.349443] irq event stamp: 65779272
[   40.351156] hardirqs last  enabled at (65779271): [<c052fae8>] _raw_spin_unlock_irqrestore+0x40/0x9c
[   40.352911] hardirqs last disabled at (65779272): [<c0011648>] reenable_mmu+0x40/0x98
(Continue reading)

Permalink | Reply |
headers
Paul E. McKenney | 2 Jul 2012 18:45
Picon

Re: 3.5.0-rc5: BUG: soft lockup - CPU#0 stuck for 22s

On Sun, Jul 01, 2012 at 11:30:40PM -0700, Christian Kujau wrote:
> On Mon, 2 Jul 2012 at 14:50, Benjamin Herrenschmidt wrote:
> > Interesting... I observed something roughly similar on a dual G4
> > the other day associated with a 30s to 1mn pause during boot. RCU
> > was complaining loudly.
> > 
> > In my case, it did continue booting normally, is that the case for you ?
> 
> No, in my case it stopped booting, though there was no "panic" message.
> 
> > Also if I compile the kernel without CONFIG_SMP, it did go away as well,
> > do you observe that too ?
> 
> This PoweBook G4 is UP anyway and I'm always building w/o CONFIG_SMP.
> 
> > I don't have a spare cycle to investigate this problem this week I'm
> > afraid, it might help if you could try a bisection though.
> 
> Yes, I've started a bisection already and will report back.
> 
> FYI, I've put a netconsole-log below from another boot (in the 
> middle of a bisection), the instruction dump is slightly more complete.

Li Zhong posted a patch to fix this (async_synchronize_full() below):

	https://lkml.org/lkml/2012/7/2/32

This gets rid of this problem on my UP testing.  If it works for you,
please give Li Zhong a Tested-by.
(Continue reading)

Permalink | Reply |
headers
Christian Kujau | 2 Jul 2012 20:46
Picon

Re: 3.5.0-rc5: BUG: soft lockup - CPU#0 stuck for 22s

On Mon, 2 Jul 2012 at 09:45, Paul E. McKenney wrote:
> On Sun, Jul 01, 2012 at 11:30:40PM -0700, Christian Kujau wrote:
> > On Mon, 2 Jul 2012 at 14:50, Benjamin Herrenschmidt wrote:
> > > Interesting... I observed something roughly similar on a dual G4
> > > the other day associated with a 30s to 1mn pause during boot. RCU
> > > was complaining loudly.
> > > 
> > > In my case, it did continue booting normally, is that the case for you ?
> > 
> > No, in my case it stopped booting, though there was no "panic" message.
> > 
> > > Also if I compile the kernel without CONFIG_SMP, it did go away as well,
> > > do you observe that too ?
> > 
> > This PoweBook G4 is UP anyway and I'm always building w/o CONFIG_SMP.
> > 
> > > I don't have a spare cycle to investigate this problem this week I'm
> > > afraid, it might help if you could try a bisection though.
> > 
> > Yes, I've started a bisection already and will report back.
> > 
> > FYI, I've put a netconsole-log below from another boot (in the 
> > middle of a bisection), the instruction dump is slightly more complete.
> 
> Li Zhong posted a patch to fix this (async_synchronize_full() below):
> 
> 	https://lkml.org/lkml/2012/7/2/32
> 
> This gets rid of this problem on my UP testing.  If it works for you,
> please give Li Zhong a Tested-by.
(Continue reading)

Permalink | Reply |
headers
Christian Kujau | 2 Jul 2012 20:46
Picon

Re: 3.5.0-rc5: BUG: soft lockup - CPU#0 stuck for 22s

On Mon, 2 Jul 2012 at 09:45, Paul E. McKenney wrote:
> On Sun, Jul 01, 2012 at 11:30:40PM -0700, Christian Kujau wrote:
> > On Mon, 2 Jul 2012 at 14:50, Benjamin Herrenschmidt wrote:
> > > Interesting... I observed something roughly similar on a dual G4
> > > the other day associated with a 30s to 1mn pause during boot. RCU
> > > was complaining loudly.
> > > 
> > > In my case, it did continue booting normally, is that the case for you ?
> > 
> > No, in my case it stopped booting, though there was no "panic" message.
> > 
> > > Also if I compile the kernel without CONFIG_SMP, it did go away as well,
> > > do you observe that too ?
> > 
> > This PoweBook G4 is UP anyway and I'm always building w/o CONFIG_SMP.
> > 
> > > I don't have a spare cycle to investigate this problem this week I'm
> > > afraid, it might help if you could try a bisection though.
> > 
> > Yes, I've started a bisection already and will report back.
> > 
> > FYI, I've put a netconsole-log below from another boot (in the 
> > middle of a bisection), the instruction dump is slightly more complete.
> 
> Li Zhong posted a patch to fix this (async_synchronize_full() below):
> 
> 	https://lkml.org/lkml/2012/7/2/32
> 
> This gets rid of this problem on my UP testing.  If it works for you,
> please give Li Zhong a Tested-by.
(Continue reading)

Permalink | Reply |
headers
Paul E. McKenney | 2 Jul 2012 18:45
Picon

Re: 3.5.0-rc5: BUG: soft lockup - CPU#0 stuck for 22s

On Sun, Jul 01, 2012 at 11:30:40PM -0700, Christian Kujau wrote:
> On Mon, 2 Jul 2012 at 14:50, Benjamin Herrenschmidt wrote:
> > Interesting... I observed something roughly similar on a dual G4
> > the other day associated with a 30s to 1mn pause during boot. RCU
> > was complaining loudly.
> > 
> > In my case, it did continue booting normally, is that the case for you ?
> 
> No, in my case it stopped booting, though there was no "panic" message.
> 
> > Also if I compile the kernel without CONFIG_SMP, it did go away as well,
> > do you observe that too ?
> 
> This PoweBook G4 is UP anyway and I'm always building w/o CONFIG_SMP.
> 
> > I don't have a spare cycle to investigate this problem this week I'm
> > afraid, it might help if you could try a bisection though.
> 
> Yes, I've started a bisection already and will report back.
> 
> FYI, I've put a netconsole-log below from another boot (in the 
> middle of a bisection), the instruction dump is slightly more complete.

Li Zhong posted a patch to fix this (async_synchronize_full() below):

	https://lkml.org/lkml/2012/7/2/32

This gets rid of this problem on my UP testing.  If it works for you,
please give Li Zhong a Tested-by.
(Continue reading)

Permalink | Reply |
headers
Christian Kujau | 2 Jul 2012 08:30
Picon

Re: 3.5.0-rc5: BUG: soft lockup - CPU#0 stuck for 22s

On Mon, 2 Jul 2012 at 14:50, Benjamin Herrenschmidt wrote:
> Interesting... I observed something roughly similar on a dual G4
> the other day associated with a 30s to 1mn pause during boot. RCU
> was complaining loudly.
> 
> In my case, it did continue booting normally, is that the case for you ?

No, in my case it stopped booting, though there was no "panic" message.

> Also if I compile the kernel without CONFIG_SMP, it did go away as well,
> do you observe that too ?

This PoweBook G4 is UP anyway and I'm always building w/o CONFIG_SMP.

> I don't have a spare cycle to investigate this problem this week I'm
> afraid, it might help if you could try a bisection though.

Yes, I've started a bisection already and will report back.

FYI, I've put a netconsole-log below from another boot (in the 
middle of a bisection), the instruction dump is slightly more complete.

Thanks!
Christian.

[   40.345973] BUG: soft lockup - CPU#0 stuck for 22s! [modprobe:691]
[   40.347737] Modules linked in: sd_mod arc4 firewire_sbp2 scsi_mod b43 mac80211 cfg80211
[   40.349443] irq event stamp: 65779272
[   40.351156] hardirqs last  enabled at (65779271): [<c052fae8>] _raw_spin_unlock_irqrestore+0x40/0x9c
[   40.352911] hardirqs last disabled at (65779272): [<c0011648>] reenable_mmu+0x40/0x98
(Continue reading)

Permalink | Reply |
headers
Christian Kujau | 2 Jul 2012 13:58
Picon

Re: 3.5.0-rc5: BUG: soft lockup - CPU#0 stuck for 22s

On Mon, 2 Jul 2012 at 14:50, Benjamin Herrenschmidt wrote:
> > while trying to upgrade from 3.4.0 to 3.5.0-rc5 on this Powerbook G4 
> > (powerpc 32 bit), this happens during booting:
> > 
> > --------------
> > usb 2-1: new full-speed USB device number 4 using ohci_hcd
> > SCSI subsystem initialized
> > scsi0 : SBP-2 IEEE-1394
> > scsi1 : SBP-2 IEEE-1394
> > firewire_sbp2 fw1.0: logged in to LUN 0000 (0 retries)
> > scsi 0:0:0:0: Direct-Access     Ext Hard  Disk            PQ: 0 ANSI: 4
> 
> Interesting... I observed something roughly similar on a dual G4
> the other day associated with a 30s to 1mn pause during boot. RCU
> was complaining loudly.
> 
> In my case, it did continue booting normally, is that the case for you ?
> 
> Also if I compile the kernel without CONFIG_SMP, it did go away as well,
> do you observe that too ?
> 
> I don't have a spare cycle to investigate this problem this week I'm
> afraid, it might help if you could try a bisection though.

OK, after a git-bisect, the following commit has been identified as bad:

---------------------------
a7a20d103994fd760766e6c9d494daa569cbfe06 is the first bad commit
commit a7a20d103994fd760766e6c9d494daa569cbfe06
Author: Dan Williams <dan.j.williams <at> intel.com>
(Continue reading)

Permalink | Reply |
headers
Christian Kujau | 2 Jul 2012 13:58
Picon

Re: 3.5.0-rc5: BUG: soft lockup - CPU#0 stuck for 22s

On Mon, 2 Jul 2012 at 14:50, Benjamin Herrenschmidt wrote:
> > while trying to upgrade from 3.4.0 to 3.5.0-rc5 on this Powerbook G4 
> > (powerpc 32 bit), this happens during booting:
> > 
> > --------------
> > usb 2-1: new full-speed USB device number 4 using ohci_hcd
> > SCSI subsystem initialized
> > scsi0 : SBP-2 IEEE-1394
> > scsi1 : SBP-2 IEEE-1394
> > firewire_sbp2 fw1.0: logged in to LUN 0000 (0 retries)
> > scsi 0:0:0:0: Direct-Access     Ext Hard  Disk            PQ: 0 ANSI: 4
> 
> Interesting... I observed something roughly similar on a dual G4
> the other day associated with a 30s to 1mn pause during boot. RCU
> was complaining loudly.
> 
> In my case, it did continue booting normally, is that the case for you ?
> 
> Also if I compile the kernel without CONFIG_SMP, it did go away as well,
> do you observe that too ?
> 
> I don't have a spare cycle to investigate this problem this week I'm
> afraid, it might help if you could try a bisection though.

OK, after a git-bisect, the following commit has been identified as bad:

---------------------------
a7a20d103994fd760766e6c9d494daa569cbfe06 is the first bad commit
commit a7a20d103994fd760766e6c9d494daa569cbfe06
Author: Dan Williams <dan.j.williams <at> intel.com>
(Continue reading)

Permalink | Reply |
headers
Benjamin Herrenschmidt | 2 Jul 2012 06:50

Re: 3.5.0-rc5: BUG: soft lockup - CPU#0 stuck for 22s

On Sun, 2012-07-01 at 00:18 -0700, Christian Kujau wrote:
> Hi,
> 
> while trying to upgrade from 3.4.0 to 3.5.0-rc5 on this Powerbook G4 
> (powerpc 32 bit), this happens during booting:
> 
> --------------
> usb 2-1: new full-speed USB device number 4 using ohci_hcd
> SCSI subsystem initialized
> scsi0 : SBP-2 IEEE-1394
> scsi1 : SBP-2 IEEE-1394
> firewire_sbp2 fw1.0: logged in to LUN 0000 (0 retries)
> scsi 0:0:0:0: Direct-Access     Ext Hard  Disk            PQ: 0 ANSI: 4

Interesting... I observed something roughly similar on a dual G4
the other day associated with a 30s to 1mn pause during boot. RCU
was complaining loudly.

In my case, it did continue booting normally, is that the case for you ?

Also if I compile the kernel without CONFIG_SMP, it did go away as well,
do you observe that too ?

I don't have a spare cycle to investigate this problem this week I'm
afraid, it might help if you could try a bisection though.

Cheers,
Ben.
Permalink | Reply |

Gmane