Discussion about the port of NetBSD to the virtual machine monitor Xen ()

Roger Pau Monné | 12 Jun 2012 19:37

Qemu-upstream with Xen

Hello,

I'm trying to get qemu-upstream (present in xen-unstable) to work with
NetBSD, but there seems to be some problem with the vga shared memory
mapping. Qemu starts correctly and allocates vram memory, but it seems
like the call to xc_domain_add_to_physmap doesn't work correctly, and
when Qemu tries to write to that memory region after the mapping it
gets a segfault.

When debugging Qemu with gdb I've realized that after calling
xc_domain_add_to_physmap the memory region is no longer "available" to
Qemu, but the same happens with old Qemu and it works fine (gdb tells
me I cannot access the memory region, but Qemu seems to be able to
write to it without problems).

Qemu-upstream uses a different memory region for vram than the old
one, so I don't know if that might be the cause of the error, but I
cannot find any kind of limitation to the memory area/size one can map
using xc_domain_add_to_physmap at least on sys/arch/xen/xen/privcmd.c.

Maybe someone more experienced with this can give me a hand. Any help
on how to track/solve this is appreciated.

Thanks, Roger.

Roger Pau Monné | 13 Jun 2012 14:05

Re: Qemu-upstream with Xen

I've debugged this a little bit more, and I've found that
privpgop_fault gets called, and tries to map the address again after
the call to xc_domain_add_to_physmap. But the call to pmap_enter_ma
returns the error 'EFAULT', and thus the segfault in Qemu. The
previous call to this function in IOCTL_PRIVCMD_MMAPBATCH (which tries
to detect which pages can be mapped) didn't mark this page as invalid,
so something must have changed in between... Suggestions are welcome.

Thanks, Roger.

Jean-Yves Migeon | 13 Jun 2012 15:22

Picon

Re: Qemu-upstream with Xen

Le 13/06/2012 13:05, Roger Pau Monné a écrit :
> I've debugged this a little bit more, and I've found that
> privpgop_fault gets called, and tries to map the address again after
> the call to xc_domain_add_to_physmap. But the call to pmap_enter_ma
> returns the error 'EFAULT', and thus the segfault in Qemu. The
> previous call to this function in IOCTL_PRIVCMD_MMAPBATCH (which 
> tries
> to detect which pages can be mapped) didn't mark this page as 
> invalid,
> so something must have changed in between... Suggestions are welcome.

Nothing appearing on hypervisor console (or xm dmesg?)

EFAULT is usually returned by hypervisor on mmu_update() calls when 
arguments are bogus (or pose a threat to hypervisor VM). Given the role 
of pmap_enter_ma(), I suspect the hypervisor is refusing the mapping.

Can you obtain the arguments passed to the hypercall?

--

-- 
Jean-Yves Migeon

Roger Pau Monné | 13 Jun 2012 16:59

Re: Qemu-upstream with Xen

On Wed, Jun 13, 2012 at 2:22 PM, Jean-Yves Migeon
<jeanyves.migeon <at> free.fr> wrote:
> Le 13/06/2012 13:05, Roger Pau Monné a écrit :
>
>> I've debugged this a little bit more, and I've found that
>> privpgop_fault gets called, and tries to map the address again after
>> the call to xc_domain_add_to_physmap. But the call to pmap_enter_ma
>> returns the error 'EFAULT', and thus the segfault in Qemu. The
>> previous call to this function in IOCTL_PRIVCMD_MMAPBATCH (which tries
>> to detect which pages can be mapped) didn't mark this page as invalid,
>> so something must have changed in between... Suggestions are welcome.
>
>
> Nothing appearing on hypervisor console (or xm dmesg?)

I'm not able to see anything relevant here, but here's the hypervisor output:

(XEN) HVM1: HVM Loader
(XEN) HVM1: Detected Xen v4.2-unstable
(XEN) HVM1: Xenbus rings  <at> 0xfeffc000, event channel 3
(XEN) HVM1: System requested SeaBIOS
(XEN) HVM1: CPU speed is 2661 MHz
(XEN) irq.c:270: Dom1 PCI link 0 changed 0 -> 5
(XEN) HVM1: PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:270: Dom1 PCI link 1 changed 0 -> 10
(XEN) HVM1: PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:270: Dom1 PCI link 2 changed 0 -> 11
(XEN) HVM1: PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:270: Dom1 PCI link 3 changed 0 -> 5
(XEN) HVM1: PCI-ISA link 3 routed to IRQ5

(Continue reading)

Manuel Bouyer | 13 Jun 2012 17:47

Re: Qemu-upstream with Xen

On Wed, Jun 13, 2012 at 03:59:11PM +0100, Roger Pau Monné wrote:
> On Wed, Jun 13, 2012 at 2:22 PM, Jean-Yves Migeon
> <jeanyves.migeon <at> free.fr> wrote:
> > Le 13/06/2012 13:05, Roger Pau Monné a écrit :
> >
> >> I've debugged this a little bit more, and I've found that
> >> privpgop_fault gets called, and tries to map the address again after
> >> the call to xc_domain_add_to_physmap. But the call to pmap_enter_ma
> >> returns the error 'EFAULT', and thus the segfault in Qemu. The
> >> previous call to this function in IOCTL_PRIVCMD_MMAPBATCH (which tries
> >> to detect which pages can be mapped) didn't mark this page as invalid,
> >> so something must have changed in between... Suggestions are welcome.
> >
> >
> > Nothing appearing on hypervisor console (or xm dmesg?)
> 
> I'm not able to see anything relevant here, but here's the hypervisor output:
> [...]

Are you using the xen-debug.gz kernel ?

--

-- 
Manuel Bouyer <bouyer <at> antioche.eu.org>
     NetBSD: 26 ans d'experience feront toujours la difference
--

Roger Pau Monné | 13 Jun 2012 18:25

Re: Qemu-upstream with Xen

On Wed, Jun 13, 2012 at 4:47 PM, Manuel Bouyer <bouyer <at> antioche.eu.org> wrote:
> On Wed, Jun 13, 2012 at 03:59:11PM +0100, Roger Pau Monné wrote:
>> On Wed, Jun 13, 2012 at 2:22 PM, Jean-Yves Migeon
>> <jeanyves.migeon <at> free.fr> wrote:
>> > Le 13/06/2012 13:05, Roger Pau Monné a écrit :
>> >
>> >> I've debugged this a little bit more, and I've found that
>> >> privpgop_fault gets called, and tries to map the address again after
>> >> the call to xc_domain_add_to_physmap. But the call to pmap_enter_ma
>> >> returns the error 'EFAULT', and thus the segfault in Qemu. The
>> >> previous call to this function in IOCTL_PRIVCMD_MMAPBATCH (which tries
>> >> to detect which pages can be mapped) didn't mark this page as invalid,
>> >> so something must have changed in between... Suggestions are welcome.
>> >
>> >
>> > Nothing appearing on hypervisor console (or xm dmesg?)
>>
>> I'm not able to see anything relevant here, but here's the hypervisor output:
>> [...]
>
> Are you using the xen-debug.gz kernel ?

Yes, I'm using a kernel build with debug=y and I use the following
boot options "loglvl=debug guest_loglvl=debug"

Jean-Yves Migeon | 13 Jun 2012 18:46

Picon

Re: Qemu-upstream with Xen

Le 13/06/2012 15:59, Roger Pau Monné a écrit :
> On Wed, Jun 13, 2012 at 2:22 PM, Jean-Yves Migeon
> <jeanyves.migeon <at> free.fr> wrote:
>> Le 13/06/2012 13:05, Roger Pau Monné a écrit :
>>
>>> I've debugged this a little bit more, and I've found that
>>> privpgop_fault gets called, and tries to map the address again 
>>> after
>>> the call to xc_domain_add_to_physmap. But the call to pmap_enter_ma
>>> returns the error 'EFAULT', and thus the segfault in Qemu. The
>>> previous call to this function in IOCTL_PRIVCMD_MMAPBATCH (which 
>>> tries
>>> to detect which pages can be mapped) didn't mark this page as 
>>> invalid,
>>> so something must have changed in between... Suggestions are 
>>> welcome.
>[...]
> vaddr: 7f7feec00000 (which corresponds to the Qemu vaddr that
> generates the segfault)
> paddr: 3f800000
> prot: 3
> flags: 35
> domid: 1 (which belongs to the domu I'm trying to create)

Good, thanks.

>> Can you obtain the arguments passed to the hypercall?
>
> The arguments passed to the hypercall (HYPERVISOR_mmu_update), are:
>

(Continue reading)

Roger Pau Monné | 13 Jun 2012 19:27

Re: Qemu-upstream with Xen

On Wed, Jun 13, 2012 at 5:46 PM, Jean-Yves Migeon
<jeanyves.migeon <at> free.fr> wrote:
> Le 13/06/2012 15:59, Roger Pau Monné a écrit :
>>
>> On Wed, Jun 13, 2012 at 2:22 PM, Jean-Yves Migeon
>> <jeanyves.migeon <at> free.fr> wrote:
>>>
>>> Le 13/06/2012 13:05, Roger Pau Monné a écrit :
>>>
>>>> I've debugged this a little bit more, and I've found that
>>>> privpgop_fault gets called, and tries to map the address again after
>>>> the call to xc_domain_add_to_physmap. But the call to pmap_enter_ma
>>>> returns the error 'EFAULT', and thus the segfault in Qemu. The
>>>> previous call to this function in IOCTL_PRIVCMD_MMAPBATCH (which tries
>>>> to detect which pages can be mapped) didn't mark this page as invalid,
>>>> so something must have changed in between... Suggestions are welcome.
>>
>> [...]
>>
>> vaddr: 7f7feec00000 (which corresponds to the Qemu vaddr that
>> generates the segfault)
>> paddr: 3f800000
>> prot: 3
>> flags: 35
>> domid: 1 (which belongs to the domu I'm trying to create)
>
>
> Good, thanks.
>
>

(Continue reading)

Jean-Yves Migeon | 14 Jun 2012 11:22

Picon

Re: Qemu-upstream with Xen

Le 13/06/2012 18:27, Roger Pau Monné a écrit :
>> Is the P2M table (xpmap_phys_to_machine_mapping) entry for this 
>> address
>> marked as INVALID_MFN?
>
> I'm not sure if this is correct, but I've done a:
>
> pfn_to_mfn((0x3f800000 - XPMAP_OFFSET) >> PAGE_SHIFT) and it is 
> 239540

Should be. Out of interest, what it the value of 
pfn_to_mfn((0x3f800000) >> PAGE_SHIFT

without taking the .text offset into account?

> Also, the syscall HYPERVISOR_mmu_update returns the error -22 
> (EINVAL).

It is definitely the hypervisor refusing the mapping, but it does not 
log why (and according to pfn_to_mfn(), this entry is valid, so there is 
something it does not like about it -- maybe a bit set where it is not 
expected to be for this type of address).

Next possible step: instrument the hypervisor (xen/arch/x86/mm.c) and 
see the path that leads to EINVAL in do_mmu_update().

--

-- 
Jean-Yves Migeon

(Continue reading)

Roger Pau Monné | 14 Jun 2012 15:47

Re: Qemu-upstream with Xen

On Thu, Jun 14, 2012 at 10:22 AM, Jean-Yves Migeon
<jeanyves.migeon <at> free.fr> wrote:
> Le 13/06/2012 18:27, Roger Pau Monné a écrit :
>
>>> Is the P2M table (xpmap_phys_to_machine_mapping) entry for this address
>>> marked as INVALID_MFN?
>>
>>
>> I'm not sure if this is correct, but I've done a:
>>
>> pfn_to_mfn((0x3f800000 - XPMAP_OFFSET) >> PAGE_SHIFT) and it is 239540
>
>
> Should be. Out of interest, what it the value of pfn_to_mfn((0x3f800000) >>
> PAGE_SHIFT
>
> without taking the .text offset into account?

It's the same, 239540.

>> Also, the syscall HYPERVISOR_mmu_update returns the error -22 (EINVAL).
>
>
> It is definitely the hypervisor refusing the mapping, but it does not log
> why (and according to pfn_to_mfn(), this entry is valid, so there is
> something it does not like about it -- maybe a bit set where it is not
> expected to be for this type of address).

My guess is that Xen doesn't like the qemu userspace address we are
passing to it, because in privcmd:433 the mapping of this DomU address

(Continue reading)

Roger Pau Monné | 15 Jun 2012 12:19

Re: Qemu-upstream with Xen

On Thu, Jun 14, 2012 at 10:22 AM, Jean-Yves Migeon
<jeanyves.migeon <at> free.fr> wrote:
> Le 13/06/2012 18:27, Roger Pau Monné a écrit :
>
>>> Is the P2M table (xpmap_phys_to_machine_mapping) entry for this address
>>> marked as INVALID_MFN?
>>
>>
>> I'm not sure if this is correct, but I've done a:
>>
>> pfn_to_mfn((0x3f800000 - XPMAP_OFFSET) >> PAGE_SHIFT) and it is 239540
>
>
> Should be. Out of interest, what it the value of pfn_to_mfn((0x3f800000) >>
> PAGE_SHIFT
>
> without taking the .text offset into account?
>
>
>> Also, the syscall HYPERVISOR_mmu_update returns the error -22 (EINVAL).
>
>
> It is definitely the hypervisor refusing the mapping, but it does not log
> why (and according to pfn_to_mfn(), this entry is valid, so there is
> something it does not like about it -- maybe a bit set where it is not
> expected to be for this type of address).
>
> Next possible step: instrument the hypervisor (xen/arch/x86/mm.c) and see
> the path that leads to EINVAL in do_mmu_update().

(Continue reading)

Roger Pau Monné | 15 Jun 2012 14:39

Re: Qemu-upstream with Xen

Ok, I've found the problem. I'm quite new to all this kernel stuff, so
bear with me if my description is not very accurate.

The main problem was that Qemu allocates a set of pfns in the p2m
table, using xc_map_foreign_bulk, those are allocated correctly,
because the p2m page is marked as p2m_ram_rw, and the correct handlers
for page faults are set in NetBSD kernel.

Then Qemu changes the pfns on our back, with the
xc_domain_add_to_physmap, and moves the region from 0x3f800 to 0xf0000
(yes, really nasty). So, when Qemu tries to write to this page, and
the pagefault handler is called, it is unable to map this region,
because the pfns are no longer correct (it should try to map 0xf0000,
but we have no way of knowing that).

Linux didn't hit this because the Linux kernel maps the page straight
away instead of setting a pagefault handler. I've done a little patch
to map the memory region directly in IOCTL_PRIVCMD_MMAPBATCH instead
of using a pagefault handler, and it seems to work correctly, I will
submit the patch shortly, so it can made it to the 6.0 release and we
can have Qemu upstream working.

Thanks for the help, Roger.

Return

Return to gmane.os.netbsd.ports.xen.

Project Web Page

Discussion about the port of NetBSD to the virtual machine monitor Xen ()

Search Archive

Language

Change language

Options

Current view: Threads only / Showing only 20 lines / Not hiding cited text.
Change to All messages, whole messages, or hide cited text.

Post a message
NNTP Newsgroup
Classic Gmane web interface
XML

XML

RSS Feed
List Information

About Gmane