Discussion of general kernel technical issues

Emmanuel Dreyfus | 9 Aug 2012 18:13

swapcontext() around pthreads

Hi

I encountered a funny portability problem when working on glusterfs.
In its 3.3. branch, it makes heavy use of swapcontext() and pthreads
to get better performance. Unfortunately the code assumes a Linux
specific behavior : a thread calling swapcontext() should not affect 
other threads. Only the calling thread context should be changed.

NetBSD has a much more rich/messy behavior. Typical usage us to call
getcontext() and makecontext() prior calling swapcontext(). 
If they all happen in the same thread, everything works like on 
Linux.

However, if getcontext() was called in thread A and swapcontext()
is called in thread B, then swapcontext causes the context of thread
A to be changed, preempting the code being executed. thread B seems to
terminate, though I suspect that could be changed with makecontext() 
and uc_link.

Attached is a test case that exhibit the behavior, and here is the output:
netbsd# ./tss
before swapcontext self = 0xbb600000
after swapcontext self = 0xbfa00000
linux# ./tss
before swapcontext self = 0x4002
after swapcontext self = 0x4002

This seems to fall into a grey area of unspecified behavior. Standards
cannot help much since that functions were removed from POSIX.1.

(Continue reading)

headers

Joerg Sonnenberger | 9 Aug 2012 18:18

Re: swapcontext() around pthreads

On Thu, Aug 09, 2012 at 04:13:43PM +0000, Emmanuel Dreyfus wrote:
> I encountered a funny portability problem when working on glusterfs.
> In its 3.3. branch, it makes heavy use of swapcontext() and pthreads
> to get better performance. Unfortunately the code assumes a Linux
> specific behavior : a thread calling swapcontext() should not affect 
> other threads. Only the calling thread context should be changed.

It sounds a lot like you should be using setjmp/longjmp in first place.
swapcontext works exactly as advertised and e.g. the thread base is part
of the context.

Joerg

headers

Emmanuel Dreyfus | 9 Aug 2012 20:44

Re: swapcontext() around pthreads

Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:

> It sounds a lot like you should be using setjmp/longjmp in first place.
> swapcontext works exactly as advertised and e.g. the thread base is part
> of the context.

I play back and forth between tech-kern <at> netbsd.org and
gluster-devel <at> nongnu.org: 

setjmp and longjmp are claim to not match the requirement because they
do not allow different stacks for each execution context.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Martin Husemann | 9 Aug 2012 20:47

Re: swapcontext() around pthreads

On Thu, Aug 09, 2012 at 08:44:05PM +0200, Emmanuel Dreyfus wrote:
> setjmp and longjmp are claim to not match the requirement because they
> do not allow different stacks for each execution context.

But there is a strong correlation between thread and stack, you can not
switch one without the other. So what is the intended semantics if you
setcontext() in thread A with a context created by thread B while B is
currently running on another CPU?

Martin

headers

David Holland | 9 Aug 2012 20:54

Re: swapcontext() around pthreads

On Thu, Aug 09, 2012 at 08:47:51PM +0200, Martin Husemann wrote:
 > On Thu, Aug 09, 2012 at 08:44:05PM +0200, Emmanuel Dreyfus wrote:
 > > setjmp and longjmp are claim to not match the requirement because they
 > > do not allow different stacks for each execution context.
 > 
 > But there is a strong correlation between thread and stack, you can not
 > switch one without the other. So what is the intended semantics if you
 > setcontext() in thread A with a context created by thread B while B is
 > currently running on another CPU?

Probably that the entire operating environment of B is cloned and run
in thread A, that is, that it's a magic implementation of threaded
continuations.

--

-- 
David A. Holland
dholland <at> netbsd.org

headers

Emmanuel Dreyfus | 10 Aug 2012 03:00

Re: swapcontext() around pthreads

David Holland <dholland-tech <at> netbsd.org> wrote:

> Probably that the entire operating environment of B is cloned and run
> in thread A, that is, that it's a magic implementation of threaded
> continuations.

Right, but the question is: can we mimick the Linux behavior?

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Thor Lancelot Simon | 10 Aug 2012 03:03

Re: swapcontext() around pthreads

On Fri, Aug 10, 2012 at 03:00:34AM +0200, Emmanuel Dreyfus wrote:
> David Holland <dholland-tech <at> netbsd.org> wrote:
> 
> > Probably that the entire operating environment of B is cloned and run
> > in thread A, that is, that it's a magic implementation of threaded
> > continuations.
> 
> Right, but the question is: can we mimick the Linux behavior?

Would you jump off a bridge if all your friends were doing it?

Thor

headers

Emmanuel Dreyfus | 10 Aug 2012 04:07

Re: swapcontext() around pthreads

Thor Lancelot Simon <tls <at> panix.com> wrote:

> > Right, but the question is: can we mimick the Linux behavior?
> Would you jump off a bridge if all your friends were doing it?

I propose offering optionnally the Linux behavior, which helps porting
Linux applications. glusterfs usage of swapcontext is not a detail, it
is deeply rooted in their code.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Joerg Sonnenberger | 10 Aug 2012 03:20

Re: swapcontext() around pthreads

On Fri, Aug 10, 2012 at 03:00:34AM +0200, Emmanuel Dreyfus wrote:
> David Holland <dholland-tech <at> netbsd.org> wrote:
> 
> > Probably that the entire operating environment of B is cloned and run
> > in thread A, that is, that it's a magic implementation of threaded
> > continuations.
> 
> Right, but the question is: can we mimick the Linux behavior?

Is there a point? Our swapcontext is effectively a system call. 
If it wasn't, it is still quite expensive for a routine that supposedly
only changes %rsp followed by a call, at least for architectures that
don't use a register window algorithm.

Joerg

headers

Emmanuel Dreyfus | 10 Aug 2012 05:11

Re: swapcontext() around pthreads

Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:

> Is there a point? Our swapcontext is effectively a system call. 
> If it wasn't, it is still quite expensive for a routine that supposedly
> only changes %rsp followed by a call, at least for architectures that
> don't use a register window algorithm.

It does more than that: it has to swap CPU register, signal mask.

I observed an interesting thing: in my test case, pthread_self() reports
that we changed thread, but ktrace shows the same lwp id, therefore
swapcontext() would not preempt anything?

And looking at the code, pthread_self is just a CPU register (gs on
i386, fs on amd64). I suspect I can reach my goal by just not changing
that register on swapccontext. Unfortunately that would be a machine
dependant option.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Emmanuel Dreyfus | 10 Aug 2012 05:29

swapcontext() vs pthread bug

Emmanuel Dreyfus <manu <at> netbsd.org> wrote:

> I observed an interesting thing: in my test case, pthread_self() reports
> that we changed thread, but ktrace shows the same lwp id, therefore
> swapcontext() would not preempt anything?

I am now convinced NetBSD swapcontext() behavior is buggy. Below is a test
case that shows how swapcontext() can make two different threads report the
same pthread_self() while both are running:

# ./tss 
thread 1 self = 0xbfa00000
before swapcontext self = 0xbb600000
after swapcontext self = 0xbfa00000
thread 2 self = 0xbfa00000
thread 1 self = 0xbfa00000
thread 2 self = 0xbfa00000
thread 1 self = 0xbfa00000
thread 2 self = 0xbfa00000

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>
#include <err.h>
#include <sysexits.h>
#include <signal.h>
#ifdef __linux__
#include <bits/sigstack.h>
#endif

(Continue reading)

headers

Emmanuel Dreyfus | 10 Aug 2012 07:01

Re: swapcontext() vs pthread bug

Emmanuel Dreyfus <manu <at> netbsd.org> wrote:

> I am now convinced NetBSD swapcontext() behavior is buggy. Below is a test
> case that shows how swapcontext() can make two different threads report the
> same pthread_self() while both are running:

There is a simple, userland-based, machine-dependent workaround that
fixes everything: unset a flag from ucontext_t uc_flags

Here is the target switch in kernel for cpu_mcontext() from
src/sys/arch/i386/i386/machdep.c 

        if ((flags & _UC_TLSBASE) != 0)
                lwp_setprivate(l, (void *)(uintptr_t)mcp->_mc_tlsbase);

Here is how to use it rom userland for our supported CPU:

amd64, arm, hppa, i386, m68k, mips:
        ctx.uc_flags &= ~_UC_TLSBASE;
alpha:
        ctx.uc_flags &= ~_UC_UNIQUE;
sh3, sparc, sparc64, vax:
        The behavior is bad and cannot be changed
powerpc: 
        The behavior is good and cannot be changed (to be verified)

What about creating a machine independant flag for amd64, arm, hppa,
i386, m68k, mips and alpha? sh3, sparc, sparc64, vax, and powerpc should
be changed to make the behavior configurable, but this is beyond what I
can do.

(Continue reading)

headers

Emmanuel Dreyfus | 10 Aug 2012 18:36

[RFC][PATCH] _UC_TLSBASE for all ports

Hi all

Summary of the previous episode: NetBSD's swapcontext restores the
thread_self pointer. When using swapcontext() on a context obtained 
from getcontext() in another thread, this makes two threads with the
same pthread_self, leading to chaos

Some ports have a workaround for this: unset _UC_TLSBASE from
ucontext_t's uc_flags. On alpha it is called _UC_UNIQUE. sh3
and poweropc have no such feature. sh3 always behave like if 
_UC_TLSBASE was set. powerpc always behave like if _UC_TLSBASE was not
set.

Attached is a patch that attempts to bring _UC_TLSBASE to all ports,
so that this useful feature gets machine independant.

I need help for powerpc. By default it does not restore the thread
context, which avoids the bug. Should we change its default to 
match the other ports? And anyone can help on the machine dependant
bit? How does it stores the pthread_self pointer?

--

-- 
Emmanuel Dreyfus
manu <at> netbsd.org

Index: arch/alpha/alpha/machdep.c
===================================================================
RCS file: /cvsroot/src/sys/arch/alpha/alpha/machdep.c,v
retrieving revision 1.340

(Continue reading)

headers

Joerg Sonnenberger | 10 Aug 2012 18:58

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Fri, Aug 10, 2012 at 04:36:17PM +0000, Emmanuel Dreyfus wrote:
> Summary of the previous episode: NetBSD's swapcontext restores the
> thread_self pointer. When using swapcontext() on a context obtained 
> from getcontext() in another thread, this makes two threads with the
> same pthread_self, leading to chaos

I maintain that trying to move contexts between threads is an inherently
bad idea and that it is a very inefficient interface for implementing
coroutines. I object to this change for the sake of misdesigned
software.

Joerg

headers

Matt Thomas | 10 Aug 2012 19:19

Re: [RFC][PATCH] _UC_TLSBASE for all ports


On Aug 10, 2012, at 9:58 AM, Joerg Sonnenberger wrote:

> On Fri, Aug 10, 2012 at 04:36:17PM +0000, Emmanuel Dreyfus wrote:
>> Summary of the previous episode: NetBSD's swapcontext restores the
>> thread_self pointer. When using swapcontext() on a context obtained 
>> from getcontext() in another thread, this makes two threads with the
>> same pthread_self, leading to chaos
> 
> I maintain that trying to move contexts between threads is an inherently
> bad idea and that it is a very inefficient interface for implementing
> coroutines. I object to this change for the sake of misdesigned
> software.

I concur.  A thread's stack is its identity and one can't swap stacks
without swapping its sense of self.  (granted thread local storage
helps but that isn't universally available).

headers

Emmanuel Dreyfus | 10 Aug 2012 19:32

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Matt Thomas <matt <at> 3am-software.com> wrote:

> I concur.  A thread's stack is its identity and one can't swap stacks
> without swapping its sense of self.  (granted thread local storage
> helps but that isn't universally available).

This is not about the stack, this is about pthread_self() which can be
changed by swapcontext(), making duplicate threads with the same
pthread_self() and breaking mapping between userland and kernel threads.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Matt Thomas | 10 Aug 2012 19:42

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Aug 10, 2012, at 10:32 AM, Emmanuel Dreyfus wrote:

> Matt Thomas <matt <at> 3am-software.com> wrote:
> 
>> I concur.  A thread's stack is its identity and one can't swap stacks
>> without swapping its sense of self.  (granted thread local storage
>> helps but that isn't universally available).
> 
> This is not about the stack, this is about pthread_self() which can be
> changed by swapcontext(), making duplicate threads with the same
> pthread_self() and breaking mapping between userland and kernel threads.

Yes it is.  A thread is tied to its stack.  Its local thread
storage is stored adjacent to the stack.  If you tell a thread to
start using another thread's stack (which a swapcontext of a saved
context will do), you are implicitly becoming that other thread.

if you start using using another thread's stack, you are assuming
the identity of that thread.

headers

Emmanuel Dreyfus | 10 Aug 2012 20:00

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Matt Thomas <matt <at> 3am-software.com> wrote:

> if you start using using another thread's stack, you are assuming
> the identity of that thread.

It is some part of its identity, but not all. Another important part of
the thread identity is given by pthread_self(). That is used to ensure
mutexes, locks and conditions work. If you screw pthread_self() as we
allow by default, horrible things happens. For instance, malloc can be
re-entered by two threads, leading to unpleasant crashes.

Having a thread to grab the stack of another one is harmless, and we
even have at least one program (glusterfs) that makes use of it. Messing
with value returned by pthread_self() will lead to chaos. 

I just propose to add an option to avoid that. I have trouble to
understand why it meets such a resistance, since most of our ports
already have that option available. I just add missing bits.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Joerg Sonnenberger | 10 Aug 2012 20:10

Re: [RFC][PATCH] _UC_TLSBASE for all ports]

On Fri, Aug 10, 2012 at 08:00:32PM +0200, Emmanuel Dreyfus wrote:
> I just propose to add an option to avoid that. I have trouble to
> understand why it meets such a resistance, since most of our ports
> already have that option available. I just add missing bits.

The only reason the option exists is to not create surprises for old
libc versions, i.e. to preserve compatibility with old programs.

Joerg

headers

David Holland | 10 Aug 2012 20:18

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Fri, Aug 10, 2012 at 08:00:32PM +0200, Emmanuel Dreyfus wrote:
 > > if you start using using another thread's stack, you are assuming
 > > the identity of that thread.
 > 
 > It is some part of its identity, but not all. [...]
 > 
 > I just propose to add an option to avoid that. I have trouble to
 > understand why it meets such a resistance, since most of our ports
 > already have that option available. I just add missing bits.

This discussion would benefit from some broader context.

(1) What is the C&V of the pertinent standards that describe
setcontext and getcontext?

(2) What is the ostensible point of programming with setcontext and
getcontext?

(3) What (possibly misguided, possibly not) things is glusterfs
actually attempting to do by mucking with setcontext and getcontext in
the way it is?

Only by addressing the complete picture can we reasonably expect to
decide on a semantically sound interface that we won't regret in the
future.

Now, ISTM that the underlying purpose of getcontext and setcontext was
always to allow writing simple userlevel-only thread systems without
needing MD assembler for context handling. If this is in fact the
case, the most reasonable use of these functions in conjunction with

(Continue reading)

headers

Emmanuel Dreyfus | 10 Aug 2012 20:56

Re: [RFC][PATCH] _UC_TLSBASE for all ports

David Holland <dholland-tech <at> netbsd.org> wrote:

> Only by addressing the complete picture can we reasonably expect to
> decide on a semantically sound interface that we won't regret in the
> future.

Short reply, I will address the rest later:

We already have the  _UC_TLSBASE feature for most of our ports. I just
propose to complete the missing ones. This is not about deciding of a
new interface.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

David Holland | 11 Aug 2012 08:19

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Fri, Aug 10, 2012 at 08:56:42PM +0200, Emmanuel Dreyfus wrote:
 > > Only by addressing the complete picture can we reasonably expect to
 > > decide on a semantically sound interface that we won't regret in the
 > > future.
 > 
 > Short reply, I will address the rest later:
 > 
 > We already have the  _UC_TLSBASE feature for most of our ports. I just
 > propose to complete the missing ones. This is not about deciding of a
 > new interface.

Yes yes, you're talking about a tree, I'm asking about this large
forest thing.

--

-- 
David A. Holland
dholland <at> netbsd.org

headers

Emmanuel Dreyfus | 11 Aug 2012 09:12

Re: [RFC][PATCH] _UC_TLSBASE for all ports

David Holland <dholland-tech <at> netbsd.org> wrote:

> (1) What is the C&V of the pertinent standards that describe
> setcontext and getcontext?

As explained in their man page, they are obsolete in the 2004 edition of
POSIX.1, and were removed in the 2008 edition. Standards are of little
help here.

> (2) What is the ostensible point of programming with setcontext and
> getcontext?
> (3) What (possibly misguided, possibly not) things is glusterfs
> actually attempting to do by mucking with setcontext and getcontext in
> the way it is?

They have tasks that map to thread pools. A task may be suspended
because it awaits a FUSE or network reply, and when it resumes, it may
be executed by another thread in the pool, because the original thread
is busy with another task. swapcontext() is used to resume a task from
any thread.

A glusterfs contributor claims he tested a pure pthread implementation
against swapcontext+pthread, and he reports the latter is 9 time faster.
In order to avoid crashes, I tweaked glusterfs parameters so that all
tasks map on the same thread, and indeed the result was crash-free, but
much slower. Barely usable, I would say.

Using _UC_TLSBASE it gets reliable without the need to reduce the
task/thread mapping to N:1, and therefore performances are good.

(Continue reading)

headers

Joerg Sonnenberger | 11 Aug 2012 15:18

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sat, Aug 11, 2012 at 09:12:16AM +0200, Emmanuel Dreyfus wrote:
> A glusterfs contributor claims he tested a pure pthread implementation
> against swapcontext+pthread, and he reports the latter is 9 time faster.

Now you are talking about it being a performance optimisation? You are
aware that this doesn't exactly help your point since *context is
extremely inefficient for the purpose you want to use it for.
Coroutine switches on x86 can be implemented in something like 20
instructions without system calls. makecontext is several orders of
magnitude slower. Which begs the question why they (glusterfs) aren't
using something like libcoro in first place...

Joerg

headers

Emmanuel Dreyfus | 11 Aug 2012 15:42

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:

> Which begs the question why they (glusterfs) aren't
> using something like libcoro in first place...

It would be nice from you to subscribe to gluster-devel <at> nongnu.org and
ask there, that would open an insightful discussion.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

matthew sporleder | 11 Aug 2012 15:56

Re: [RFC][PATCH] _UC_TLSBASE for all ports

As an aside- I've recently start using gluster and it's very very cool.

I'm really pleased with this porting effort.

Thanks manu <at> !

headers

Emmanuel Dreyfus | 11 Aug 2012 17:13

Re: [RFC][PATCH] _UC_TLSBASE for all ports

matthew sporleder <msporleder <at> gmail.com> wrote:

> As an aside- I've recently start using gluster and it's very very cool.

On what netbsd and glusterfs versions are you using it? 

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Mouse | 10 Aug 2012 20:45

Re: [RFC][PATCH] _UC_TLSBASE for all ports

> Having a thread to grab the stack of another one is harmless,

...huh?  I'd expect it to be pretty catastrohpic for two threads to be
trying to use the same piece of RAM as stack: whichever of them has a
deeper stack will find its stack's contents changing out from under it
whenever the other one runs.

Or are we talking a strictly cooperative multitasking model in which a
thread's stack must be empty (or at least at some fixed depth) when it
yields?

/~\ The ASCII				  Mouse
\ / Ribbon Campaign
 X  Against HTML		mouse <at> rodents-montreal.org
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

headers

Emmanuel Dreyfus | 10 Aug 2012 20:56

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Mouse <mouse <at> Rodents-Montreal.ORG> wrote:

> Or are we talking a strictly cooperative multitasking model in which a
> thread's stack must be empty (or at least at some fixed depth) when it
> yields?

Yes, this is used for cooperative multitasking of N user threads over P
kernel threads.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Valeriy E. Ushakov | 10 Aug 2012 19:32

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Fri, Aug 10, 2012 at 10:19:03 -0700, Matt Thomas wrote:

> On Aug 10, 2012, at 9:58 AM, Joerg Sonnenberger wrote:
> 
> > On Fri, Aug 10, 2012 at 04:36:17PM +0000, Emmanuel Dreyfus wrote:
> >> Summary of the previous episode: NetBSD's swapcontext restores the
> >> thread_self pointer. When using swapcontext() on a context obtained 
> >> from getcontext() in another thread, this makes two threads with the
> >> same pthread_self, leading to chaos
> > 
> > I maintain that trying to move contexts between threads is an inherently
> > bad idea and that it is a very inefficient interface for implementing
> > coroutines. I object to this change for the sake of misdesigned
> > software.
> 
> I concur.  A thread's stack is its identity and one can't swap stacks
> without swapping its sense of self.  (granted thread local storage
> helps but that isn't universally available).

+1

I'm not sure it's possible to come up with coherent semantic for
mixing pthreads and *context functions in that way (even if stack is
not used as thread identity).

-uwe

headers

Emmanuel Dreyfus | 10 Aug 2012 19:31

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:

> I maintain that trying to move contexts between threads is an inherently
> bad idea and that it is a very inefficient interface for implementing
> coroutines. I object to this change for the sake of misdesigned
> software.

Did you look at that test case? This is a nasty bug, and I think we need
a way for user to opt out of that behavior.

# ./tss 
thread 1 self = 0xbfa00000
before swapcontext self = 0xbb600000
after swapcontext self = 0xbfa00000
thread 2 self = 0xbfa00000
thread 1 self = 0xbfa00000
thread 2 self = 0xbfa00000
thread 1 self = 0xbfa00000
thread 2 self = 0xbfa00000

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>
#include <err.h>
#include <sysexits.h>
#include <signal.h>
#ifdef __linux__
#include <bits/sigstack.h>
#endif

(Continue reading)

headers

Joerg Sonnenberger | 10 Aug 2012 19:38

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Fri, Aug 10, 2012 at 07:31:59PM +0200, Emmanuel Dreyfus wrote:
> Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:
> 
> > I maintain that trying to move contexts between threads is an inherently
> > bad idea and that it is a very inefficient interface for implementing
> > coroutines. I object to this change for the sake of misdesigned
> > software.
> 
> Did you look at that test case? This is a nasty bug, and I think we need
> a way for user to opt out of that behavior.

I don't agree with it being a nasty bug. Heck, document it as limitation
if you want. But essentially don't mix *context and pthread in this way,
it will create other interesting issues later.

Joerg

headers

Emmanuel Dreyfus | 11 Aug 2012 03:37

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:

> I don't agree with it being a nasty bug. Heck, document it as limitation
> if you want. But essentially don't mix *context and pthread in this way,
> it will create other interesting issues later.

Ok, so here is my point of view:
- we allow mixing swapcontext and pthread in a way that procude a bug
- we have _UC_TLSBASE  on most ports that allow doing it without the bug
- I propose adding _UC_TLSBASE for ports that miss it

Are you really going to oppose making a MD feature available on all
ports with the same interface because you dislike it being used? We are
not going to make the glusterfs guys rewriting their code, they seems to
be happy with it. If my patch is rejected, I will have contribute a
change to glusterfs like below. That looks like a shameful mess.

--- syncop.c.orig
+++ syncop.c
 <at>  <at>  -290,8 +293,17  <at>  <at> 
         THIS = task->xl;

         task->woken = 0;
         task->slept = 0;
+#ifdef __NetBSD__
+#if defined(_UC_TLSBASE)
+        task->ctx.uc_flags &= ~_UC_TLSBASE;
+#elif defined(_UC_UNIQUE) /* For alpha */
+        task->ctx.uc_flags &= ~_UC_UNIQUE;
+#elif !defined(__powerpc__)

(Continue reading)

headers

Joerg Sonnenberger | 11 Aug 2012 15:27

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sat, Aug 11, 2012 at 03:37:05AM +0200, Emmanuel Dreyfus wrote:
> Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:
> 
> > I don't agree with it being a nasty bug. Heck, document it as limitation
> > if you want. But essentially don't mix *context and pthread in this way,
> > it will create other interesting issues later.
> 
> Ok, so here is my point of view:
> - we allow mixing swapcontext and pthread in a way that procude a bug

We always allowed it. In netbsd-5 it is plainly broken. Your point?

> - we have _UC_TLSBASE  on most ports that allow doing it without the bug
> - I propose adding _UC_TLSBASE for ports that miss it
> 
> Are you really going to oppose making a MD feature available on all
> ports with the same interface because you dislike it being used? We are
> not going to make the glusterfs guys rewriting their code, they seems to
> be happy with it. If my patch is rejected, I will have contribute a
> change to glusterfs like below. That looks like a shameful mess.

Yes, I dislike it being used, because it breaks perfectly valid
assumptions like lwpid being constant across the life time of a thread.
There are essentially two reasons why *context existed in the past:
(1) Allow pure userland thread libraries to be created in a
semi-portable fashion.
(2) Allow a basic fallback path for co-routine libraries.

The first part works perfectly fine. The second continues to work as
long as you stay within the same thread and didn't work before. It can

(Continue reading)

headers

Emmanuel Dreyfus | 11 Aug 2012 15:42

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:

> Yes, I dislike it being used, because it breaks perfectly valid
> assumptions like lwpid being constant across the life time of a thread.

Therefore I do not understand why you oppose the proposal of proposing a
MI interface to let the user opt out of that bad behavior.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Joerg Sonnenberger | 11 Aug 2012 21:31

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sat, Aug 11, 2012 at 03:42:59PM +0200, Emmanuel Dreyfus wrote:
> Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:
> 
> > Yes, I dislike it being used, because it breaks perfectly valid
> > assumptions like lwpid being constant across the life time of a thread.
> 
> Therefore I do not understand why you oppose the proposal of proposing a
> MI interface to let the user opt out of that bad behavior.

Your "fix" doesn't change this issue at all.

Joerg

headers

Emmanuel Dreyfus | 12 Aug 2012 03:00

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:

> > Therefore I do not understand why you oppose the proposal of proposing a
> > MI interface to let the user opt out of that bad behavior.
> 
> Your "fix" doesn't change this issue at all.

Of course it does. 

For now if you want to swapcontext() between threads, you can do that
before calling swapcontext() on amd64, arm, hppa, i386, m68k and vax:
        ctx.uc_flags &= _UC_TLSBASE 

The result is that the pthread private pointer will be left untouched,
presserving pthread sanity. You cannot do that on sh3, sparc, sparc64
and powerpc, and this is what I propose to address. 

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Emmanuel Dreyfus | 12 Aug 2012 03:02

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Emmanuel Dreyfus <manu <at> netbsd.org> wrote:

> For now if you want to swapcontext() between threads, you can do that
> before calling swapcontext() on amd64, arm, hppa, i386, m68k and vax:
>         ctx.uc_flags &= _UC_TLSBASE 

I meant this (missing ~):
        ctx.uc_flags &= ~_UC_TLSBASE 
--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Martin Husemann | 12 Aug 2012 10:02

Re: [RFC][PATCH] _UC_TLSBASE for all ports

So, I see various things coming together in this thread, and I would
suggest to fix them all with a change beyound what Emmanuel originaly
proposed:

 - the pthread_self() usage and the binding to a register is a property of
   our libpthread implementation, so it should overwrite setcontext and
   swapcontext with its own implementation, and prevent changing of
   the pthread_self() register (by clearing the _UC_TSLBASE flag before
   passing the context on to the libc version or something)

 - for that to work, we need to do what Emmanuel originaly proposed

 - but also the glusterffs folks should be asked to provide an optional,
   pure pthread based thread pool implementation for portability reasons,
   which will gain the same performance on NetBSD as the swapcontext based
   one.

 - however, we should bring back (under slightly different names, and in libc)
   the old pure userland setcontext (and friends) that we used to have
   in the N:M libpthread. We have the code for all archs, it is tested, and
   it is usefull for situations like this (and the R runtime in pkgsrc, hi
   Ignatios!)

 - manu could then go on and use the new lightweight context functions from
   the previous point to create an improved version of the swapcontext based
   threadpool and speed up glusterffs on NetBSD again

Martin

(Continue reading)

headers

Joerg Sonnenberger | 12 Aug 2012 15:19

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sun, Aug 12, 2012 at 10:02:48AM +0200, Martin Husemann wrote:
>  - however, we should bring back (under slightly different names, and in libc)
>    the old pure userland setcontext (and friends) that we used to have
>    in the N:M libpthread. We have the code for all archs, it is tested, and
>    it is usefull for situations like this (and the R runtime in pkgsrc, hi
>    Ignatios!)

They are still very heavy for coroutine usage.

Joerg

headers

Emmanuel Dreyfus | 12 Aug 2012 15:35

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:

> They are still very heavy for coroutine usage.

I cannot argue on that point. This is where it would be interesting if
you could defend that on gluster-devel <at> nongnu.org.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

David Holland | 13 Aug 2012 22:15

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sun, Aug 12, 2012 at 10:02:48AM +0200, Martin Husemann wrote:
 > So, I see various things coming together in this thread, and I would
 > suggest to fix them all with a change beyound what Emmanuel originaly
 > proposed:
 > 
 >  - the pthread_self() usage and the binding to a register is a property of
 >    our libpthread implementation, so it should overwrite setcontext and
 >    swapcontext with its own implementation, and prevent changing of
 >    the pthread_self() register (by clearing the _UC_TSLBASE flag before
 >    passing the context on to the libc version or something)
 > 
 >  - for that to work, we need to do what Emmanuel originaly proposed
 > 
 >  - but also the glusterffs folks should be asked to provide an optional,
 >    pure pthread based thread pool implementation for portability reasons,
 >    which will gain the same performance on NetBSD as the swapcontext based
 >    one.
 > 
 >  - however, we should bring back (under slightly different names, and in libc)
 >    the old pure userland setcontext (and friends) that we used to have
 >    in the N:M libpthread. We have the code for all archs, it is tested, and
 >    it is usefull for situations like this (and the R runtime in pkgsrc, hi
 >    Ignatios!)
 > 
 >  - manu could then go on and use the new lightweight context functions from
 >    the previous point to create an improved version of the swapcontext based
 >    threadpool and speed up glusterffs on NetBSD again

This seems the right approach to me.

(Continue reading)

headers

Joerg Sonnenberger | 12 Aug 2012 15:20

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sun, Aug 12, 2012 at 03:00:50AM +0200, Emmanuel Dreyfus wrote:
> Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:
> 
> > > Therefore I do not understand why you oppose the proposal of proposing a
> > > MI interface to let the user opt out of that bad behavior.
> > 
> > Your "fix" doesn't change this issue at all.
> 
> Of course it does. 

No, it doesn't. You don't change the lwpid if you move to a different
kernel thread. As such, you are breaking the "lwpid is constant"
property.

Joerg

headers

Emmanuel Dreyfus | 12 Aug 2012 18:19

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:

> No, it doesn't. You don't change the lwpid if you move to a different
> kernel thread. As such, you are breaking the "lwpid is constant"
> property.

This is how NetBSD currently behave, unless the user takes care of
unsetting _UC_TLSBASE before walling swapcontext(). My proposal is just
allow that workaround on all ports, since some of them lack the
possibiliy.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Joerg Sonnenberger | 12 Aug 2012 18:32

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sun, Aug 12, 2012 at 06:19:02PM +0200, Emmanuel Dreyfus wrote:
> Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:
> 
> > No, it doesn't. You don't change the lwpid if you move to a different
> > kernel thread. As such, you are breaking the "lwpid is constant"
> > property.
> 
> This is how NetBSD currently behave, unless the user takes care of
> unsetting _UC_TLSBASE before walling swapcontext(). My proposal is just
> allow that workaround on all ports, since some of them lack the
> possibiliy.

You are mixing up two unrelated items. The thread private area and the
thread ID are not the same thing. Your "fix" only affects the former,
not the latter.

Joerg

headers

Emmanuel Dreyfus | 12 Aug 2012 19:27

Re: [RFC][PATCH] _UC_TLSBASE for all ports

rJoerg Sonnenberger <joerg <at> britannica.bec.de> wrote:

> You are mixing up two unrelated items. The thread private area and the
> thread ID are not the same thing. Your "fix" only affects the former,
> not the latter.

Right, I was not sure what you were talking about. You mean struct lwp's
l_lid, right? If this is the case, then shall I understand you worry
about an application that plays with swapcontext() around pthreads,
retreive its kernel l_lid, and get confused because it changed? Or did I
miss something else?

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Joerg Sonnenberger | 12 Aug 2012 20:28

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sun, Aug 12, 2012 at 07:27:36PM +0200, Emmanuel Dreyfus wrote:
> rJoerg Sonnenberger <joerg <at> britannica.bec.de> wrote:
> 
> > You are mixing up two unrelated items. The thread private area and the
> > thread ID are not the same thing. Your "fix" only affects the former,
> > not the latter.
> 
> Right, I was not sure what you were talking about. You mean struct lwp's
> l_lid, right? If this is the case, then shall I understand you worry
> about an application that plays with swapcontext() around pthreads,
> retreive its kernel l_lid, and get confused because it changed? Or did I
> miss something else?

Or libpthread or rtld for the same kind of issue.

Joerg

headers

Emmanuel Dreyfus | 12 Aug 2012 21:22

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:

> Or libpthread or rtld for the same kind of issue.

I could understand libpthread would get confused if pthread_self() did
not match struct lwp's l_lid, and this is just the problem I want to
address by having _UC_TLSBASE on all ports. If you have the flag, you
can disable it, and that ensures pthread_self() always match struct
lwp's l_lid.

For rtld I cannot see how it is related. Could you enlighten me?

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

matthew green | 12 Aug 2012 00:47

re: [RFC][PATCH] _UC_TLSBASE for all ports


> > Yes, I dislike it being used, because it breaks perfectly valid
> > assumptions like lwpid being constant across the life time of a thread.
> 
> Therefore I do not understand why you oppose the proposal of proposing a
> MI interface to let the user opt out of that bad behavior.

i'm confused how the _UC_* interface can be considered "public".
the whole point of them being _* is that they're private to the
implementation, and depending on them is wrong.

i'm not sure where i stand on this issue, but if this is to be
used in 3rd party code we should publish definitions that do
not start with "_".

.mrg.

headers

Emmanuel Dreyfus | 12 Aug 2012 01:58

Re: [RFC][PATCH] _UC_TLSBASE for all ports

matthew green <mrg <at> eterna.com.au> wrote:

> i'm confused how the _UC_* interface can be considered "public".
> the whole point of them being _* is that they're private to the
> implementation, and depending on them is wrong.

Yes, _UC_* is out of the standard since there is no uc_field:
http://pubs.opengroup.org/onlinepubs/009696799/basedefs/ucontext.h.html

This has to be an extension out of the standard anyway, since the
standard does not tell about what should be done with pthread private
pointer. And while we are dealing with an extension, it is still
desirable to have it MI rather than MD.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Christos Zoulas | 11 Aug 2012 08:45

Re: [RFC][PATCH] _UC_TLSBASE for all ports

In article <20120810173818.GA8131 <at> britannica.bec.de>,
Joerg Sonnenberger  <joerg <at> britannica.bec.de> wrote:
>On Fri, Aug 10, 2012 at 07:31:59PM +0200, Emmanuel Dreyfus wrote:
>> Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:
>> 
>> > I maintain that trying to move contexts between threads is an inherently
>> > bad idea and that it is a very inefficient interface for implementing
>> > coroutines. I object to this change for the sake of misdesigned
>> > software.
>> 
>> Did you look at that test case? This is a nasty bug, and I think we need
>> a way for user to opt out of that behavior.
>
>I don't agree with it being a nasty bug. Heck, document it as limitation
>if you want. But essentially don't mix *context and pthread in this way,
>it will create other interesting issues later.

Like it or not most of the world has turned into linux. We can either
provide compatibility where possible (and not overly disgusting) to
gain compatibility with 3rd party code developed for linux, or simply
say "tough, it will not work on NetBSD because we refuse to compromise".

It is a slippery slope, but I think in this case it is wise to bend.
If we cannot reach agreement here, consult core.

christos

headers

Emmanuel Dreyfus | 11 Aug 2012 11:16

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Christos Zoulas <christos <at> astron.com> wrote:

> Like it or not most of the world has turned into linux. We can either
> provide compatibility where possible (and not overly disgusting) to
> gain compatibility with 3rd party code developed for linux, or simply
> say "tough, it will not work on NetBSD because we refuse to compromise".

IMO here it is even a worse stance, since we already have the desired
fix for amd64, i386, m68k, mips, vax, and hppa. It would be more like:
"it will work on NetBSD, except for sh3, sparc, and sparc64 ports
because we refuse to compromise. And on powerpc and alpha it will work
but with different interfaces". 

We are supposed to focus on portability, it is really weird to argue
that a feature should have a MI interface.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Mouse | 11 Aug 2012 11:46

Re: [RFC][PATCH] _UC_TLSBASE for all ports

>> [...], or simply say "tough, it will not work on NetBSD because we
>> refuse to compromise".

> IMO here it is even a worse stance, since we already have the desired
> fix for amd64, i386, m68k, mips, vax, and hppa.

Ah, but is it really a fix?

IMO this all hinges on just what the interface contract for the
*context calls is.  Depending on that, I can see this as being any of
various possibilities ("you" here is the code mentioned upthread
(glusterfs?) that assumes something about these calls):

(a) "Linux is broken; you're taking advantage of a bug".

(b) "You're using the call outside its interface spec; you're off in
    undefined-behaviour weeds."

(c) "Any of the various behaviours are permitted; none of the
    implementations are broken, and you're depending on unspecified
    behaviour."

(d) "NetBSD ports $LIST are broken; we need to fix them."

Given what manu <at>  has said about standards, we have to either consider
that there exists no de-jure spec, or we have to depend on a past spec.
Personally, I'd be inclined to do the latter; if we were to do the
former, we should remove the functions altogether, it seems to me.

If we do depend on a past spec, then it depends on what it says.  If it

(Continue reading)

headers

Emmanuel Dreyfus | 11 Aug 2012 15:36

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Mouse <mouse <at> Rodents-Montreal.ORG> wrote:

> Assuming we go with the "no spec" stance, I think the truth would be
> more like "trying to use contexts across thread boundaries is
> inherently MD, working the way you expect on ports $LIST1, working this
> other way on ports $LIST2, and this third way on ports $LIST3; you're
> depending on behaviour not portable between CPU architectures".

Yes and no. Indeed there are MD bits here, but pthread and swapcontext()
are MI features. One can resonabily expect our MI features to work the
same way on all ports. At least it should be our goal, as far as I
undertstand NetBSD core values.

Then there is the problem that no spec tells us what swapcontext()
should do with pthread private context, as returned by pthread_self().
Obviously there are two possible behaviors:
1- left pthread private context untouched, as Linux does, and perhaps
our powerpc ports, I am not sure.
2- let user change the pthread private context, as it is the default on
NetBSD (except powerpc ports, not sure).

Behavior 1 seems to have real applications. I am not sure behavior 2 can
lead to anything but SIGSEGV, but perhaps someone reimplementing a new
thread library could enjoy that, I cannot tell. 

A nice way to cope with unspecified behavior is to offer
user-configurable options. Our amd64, arm, hppa, i386, m68k, and vax
based ports offer such an option called _UC_TLSBASE. alpha offer the
same feature with a different name: _UC_UNIQUE.

(Continue reading)

headers

Thor Lancelot Simon | 11 Aug 2012 19:36

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sat, Aug 11, 2012 at 03:36:28PM +0200, Emmanuel Dreyfus wrote:
> 
> Then there is the problem that no spec tells us what swapcontext()
> should do with pthread private context, as returned by pthread_self().

Gee, I wonder why this stuff was removed from the standard.

Thor

headers

Emmanuel Dreyfus | 11 Aug 2012 20:17

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Thor Lancelot Simon <tls <at> panix.com> wrote:

> Gee, I wonder why this stuff was removed from the standard.

It seems the biggest problem is that makecontext does not specify
argument types. If you send a pointer on an LP64 machine and the
implementation assumes arguments are int, you are screwed.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Christos Zoulas | 11 Aug 2012 12:03

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Aug 11, 11:16am, manu <at> netbsd.org (Emmanuel Dreyfus) wrote:
-- Subject: Re: [RFC][PATCH] _UC_TLSBASE for all ports

| Christos Zoulas <christos <at> astron.com> wrote:
| 
| > Like it or not most of the world has turned into linux. We can either
| > provide compatibility where possible (and not overly disgusting) to
| > gain compatibility with 3rd party code developed for linux, or simply
| > say "tough, it will not work on NetBSD because we refuse to compromise".
| 
| IMO here it is even a worse stance, since we already have the desired
| fix for amd64, i386, m68k, mips, vax, and hppa. It would be more like:
| "it will work on NetBSD, except for sh3, sparc, and sparc64 ports
| because we refuse to compromise. And on powerpc and alpha it will work
| but with different interfaces". 
| 
| We are supposed to focus on portability, it is really weird to argue
| that a feature should have a MI interface.

I don't see why this change is met with so much resistance...
I could believe that, if the change suggested to make this the default
behavior (which some would argue it should be...)

christos

headers

Emmanuel Dreyfus | 11 Aug 2012 12:40

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Christos Zoulas <christos <at> zoulas.com> wrote:

> I could believe that, if the change suggested to make this the default
> behavior (which some would argue it should be...)

In an ideal world, we would set _UC_TLSBASE by default (as it is today,
except on powerpc), and we would automatically ignore _UC_TLSBASE when
l->l_proc->p_nlwps > 1, since we cannot think of a sane usage for that.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Martin Husemann | 11 Aug 2012 12:48

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sat, Aug 11, 2012 at 12:40:44PM +0200, Emmanuel Dreyfus wrote:
> In an ideal world, we would set _UC_TLSBASE by default (as it is today,
> except on powerpc), and we would automatically ignore _UC_TLSBASE when
> l->l_proc->p_nlwps > 1, since we cannot think of a sane usage for that.

I would agree with this part.

However, for a dummie like me who hasn't seen the code in question, could
you please explain how they handle the stack pointer in the context?

I know it can be done savely, but I can't think of a MI way to code that,
unless in artifical simple cases of stack usage, and then the stupid
register windows get in the way on some archs.

Martin

headers

Emmanuel Dreyfus | 11 Aug 2012 13:50

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Martin Husemann <martin <at> duskware.de> wrote:

> However, for a dummie like me who hasn't seen the code in question, could
> you please explain how they handle the stack pointer in the context?

- get current ucontext_t wht getcontext();
- allocate a stack with malloc(), add it to ucontext_t (uc_stack field)
- call makecontext() to set the target function and argument
- call swapcontext()

swapcontext(3) uses setcontext(2), which setup the stack pointer
according to what has been specified in ucontext_t. As far as I
understand, setcontext(2) offers an MI interface and takes care of the
MD problems, included those related to the stack.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Martin Husemann | 11 Aug 2012 14:00

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sat, Aug 11, 2012 at 01:50:58PM +0200, Emmanuel Dreyfus wrote:
> Martin Husemann <martin <at> duskware.de> wrote:
> 
> > However, for a dummie like me who hasn't seen the code in question, could
> > you please explain how they handle the stack pointer in the context?
> 
> - get current ucontext_t wht getcontext();
> - allocate a stack with malloc(), add it to ucontext_t (uc_stack field)
> - call makecontext() to set the target function and argument
> - call swapcontext()

Ah, IC, so every task has its own stack and you never use the original pthread
thread stack. Yes, this would work (and swapcontext deals with the register
windows).

Thanks for explanation,

Martin

headers

Emmanuel Dreyfus | 11 Aug 2012 14:33

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Emmanuel Dreyfus <manu <at> netbsd.org> wrote:

> In an ideal world, we would set _UC_TLSBASE by default (as it is today,
> except on powerpc), and we would automatically ignore _UC_TLSBASE when
> l->l_proc->p_nlwps > 1, since we cannot think of a sane usage for that.

Or perhaps do that stuff in swapcontext libc stub? That way someone that
wants to reimplement another thread library on top of our kernel thread
is free to override _UC_TLSBASE being ignored.

But this is another problem beyond the current discussion, which is just
to support _UC_TLSBASE for ports that miss it.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Christos Zoulas | 11 Aug 2012 17:00

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Aug 11, 12:40pm, manu <at> netbsd.org (Emmanuel Dreyfus) wrote:
-- Subject: Re: [RFC][PATCH] _UC_TLSBASE for all ports

| Christos Zoulas <christos <at> zoulas.com> wrote:
| 
| > I could believe that, if the change suggested to make this the default
| > behavior (which some would argue it should be...)
| 
| In an ideal world, we would set _UC_TLSBASE by default (as it is today,
| except on powerpc), and we would automatically ignore _UC_TLSBASE when
| l->l_proc->p_nlwps > 1, since we cannot think of a sane usage for that.

Well, why don't we make it that way then?

christos

headers

Emmanuel Dreyfus | 11 Aug 2012 17:13

Re: [RFC][PATCH] _UC_TLSBASE for all ports

Christos Zoulas <christos <at> zoulas.com> wrote:

> Well, why don't we make it that way then?

We cannot toggle an option that does not exist, so that require adding
_UC_TLSBASE  for ports that miss it. This meets a strong opposition for
now.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Christos Zoulas | 11 Aug 2012 18:08

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Aug 11,  5:13pm, manu <at> netbsd.org (Emmanuel Dreyfus) wrote:
-- Subject: Re: [RFC][PATCH] _UC_TLSBASE for all ports

| > Well, why don't we make it that way then?
| 
| We cannot toggle an option that does not exist, so that require adding
| _UC_TLSBASE  for ports that miss it. This meets a strong opposition for
| now.

Again, if there is no consensus in the lists, we'll have to let core
decide.

christos

headers

Thor Lancelot Simon | 11 Aug 2012 19:35

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sat, Aug 11, 2012 at 06:45:12AM +0000, Christos Zoulas wrote:
> 
> It is a slippery slope, but I think in this case it is wise to bend.
> If we cannot reach agreement here, consult core.

I see no point bending NetBSD into knots in this case if the resulting
performance is as bad as Joerg claims it will be.  Is it actually the
case that our *context() functions are almost as heavy as a full
kernel-level thread switch?

--

-- 
 Thor Lancelot Simon	                                      tls <at> panix.com
   But as he knew no bad language, he had called him all the names of common
 objects that he could think of, and had screamed: "You lamp!  You towel!  You
 plate!" and so on.              --Sigmund Freud

headers

Joerg Sonnenberger | 11 Aug 2012 21:32

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sat, Aug 11, 2012 at 01:35:43PM -0400, Thor Lancelot Simon wrote:
> On Sat, Aug 11, 2012 at 06:45:12AM +0000, Christos Zoulas wrote:
> > 
> > It is a slippery slope, but I think in this case it is wise to bend.
> > If we cannot reach agreement here, consult core.
> 
> I see no point bending NetBSD into knots in this case if the resulting
> performance is as bad as Joerg claims it will be.  Is it actually the
> case that our *context() functions are almost as heavy as a full
> kernel-level thread switch?

They involve two full system calls and saving the FPU state. That's
about as heavy as a context switch...

Joerg

headers

Matt Thomas | 12 Aug 2012 01:03

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Aug 11, 2012, at 10:35 AM, Thor Lancelot Simon wrote:

> On Sat, Aug 11, 2012 at 06:45:12AM +0000, Christos Zoulas wrote:
>> 
>> It is a slippery slope, but I think in this case it is wise to bend.
>> If we cannot reach agreement here, consult core.
> 
> I see no point bending NetBSD into knots in this case if the resulting
> performance is as bad as Joerg claims it will be.  Is it actually the
> case that our *context() functions are almost as heavy as a full
> kernel-level thread switch?

I'm wondering if we need a new makecontext which can allocate a new
private thread-local area.  We can set the stack via uc_stack but
there isn't a way to allocate a new thread-local area.

headers

Valeriy E. Ushakov | 12 Aug 2012 02:16

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sat, Aug 11, 2012 at 16:03:08 -0700, Matt Thomas wrote:

> On Aug 11, 2012, at 10:35 AM, Thor Lancelot Simon wrote:
> 
> > On Sat, Aug 11, 2012 at 06:45:12AM +0000, Christos Zoulas wrote:
> >> 
> >> It is a slippery slope, but I think in this case it is wise to bend.
> >> If we cannot reach agreement here, consult core.
> > 
> > I see no point bending NetBSD into knots in this case if the resulting
> > performance is as bad as Joerg claims it will be.  Is it actually the
> > case that our *context() functions are almost as heavy as a full
> > kernel-level thread switch?
> 
> I'm wondering if we need a new makecontext which can allocate a new
> private thread-local area.  We can set the stack via uc_stack but
> there isn't a way to allocate a new thread-local area.

Like pthread_attr_setcreatesuspend_np(3) + pthread_create(3)?

-uwe

headers

Eduardo Horvath | 13 Aug 2012 17:45

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Sat, 11 Aug 2012, Matt Thomas wrote:

> On Aug 11, 2012, at 10:35 AM, Thor Lancelot Simon wrote:
> 
> > On Sat, Aug 11, 2012 at 06:45:12AM +0000, Christos Zoulas wrote:
> >> 
> >> It is a slippery slope, but I think in this case it is wise to bend.
> >> If we cannot reach agreement here, consult core.
> > 
> > I see no point bending NetBSD into knots in this case if the resulting
> > performance is as bad as Joerg claims it will be.  Is it actually the
> > case that our *context() functions are almost as heavy as a full
> > kernel-level thread switch?
> 
> I'm wondering if we need a new makecontext which can allocate a new
> private thread-local area.  We can set the stack via uc_stack but
> there isn't a way to allocate a new thread-local area.
> 

I think this whole thing needs a careful redesign.  IMHO the reason we 
never got scheduler activations stable across all architectures is that 
the semantics of the *context() routines were never properly specified.  

If I knew what those routines were supposed to do I might have been able 
to fix them.  But as it was implementation defined functionality....

Eduardo

headers

Christos Zoulas | 12 Aug 2012 07:58

Re: [RFC][PATCH] _UC_TLSBASE for all ports

On Aug 11,  1:35pm, tls <at> panix.com (Thor Lancelot Simon) wrote:
-- Subject: Re: [RFC][PATCH] _UC_TLSBASE for all ports

| On Sat, Aug 11, 2012 at 06:45:12AM +0000, Christos Zoulas wrote:
| > 
| > It is a slippery slope, but I think in this case it is wise to bend.
| > If we cannot reach agreement here, consult core.
| 
| I see no point bending NetBSD into knots in this case if the resulting
| performance is as bad as Joerg claims it will be.  Is it actually the
| case that our *context() functions are almost as heavy as a full
| kernel-level thread switch?

The point is that glusterfs works without code modifications (or minimal ones)
and wit acceptable performance.

christos

headers

YAMAMOTO Takashi | 13 Aug 2012 05:14

Re: [RFC][PATCH] _UC_TLSBASE for all ports

hi,

> On Fri, Aug 10, 2012 at 07:31:59PM +0200, Emmanuel Dreyfus wrote:
>> Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:
>> 
>> > I maintain that trying to move contexts between threads is an inherently
>> > bad idea and that it is a very inefficient interface for implementing
>> > coroutines. I object to this change for the sake of misdesigned
>> > software.
>> 
>> Did you look at that test case? This is a nasty bug, and I think we need
>> a way for user to opt out of that behavior.
> 
> I don't agree with it being a nasty bug. Heck, document it as limitation
> if you want. But essentially don't mix *context and pthread in this way,
> it will create other interesting issues later.

what's the motivation to keep the limitation?

the current bahaviour would benefit applications which use i386 %gs register
for their own purposes.  is it your concern?

YAMAMOTO Takashi

> 
> Joerg

headers

Emmanuel Dreyfus | 13 Aug 2012 05:57

Re: [RFC][PATCH] _UC_TLSBASE for all ports

YAMAMOTO Takashi <yamt <at> mwd.biglobe.ne.jp> wrote:

> the current bahaviour would benefit applications which use i386 %gs register
> for their own purposes.  is it your concern?

Note I did not propose to alter the default behavior. I just want an
option to disable it.

--

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu <at> netbsd.org

headers

Joerg Sonnenberger | 10 Aug 2012 06:14

Re: swapcontext() around pthreads

On Fri, Aug 10, 2012 at 05:11:02AM +0200, Emmanuel Dreyfus wrote:
> Joerg Sonnenberger <joerg <at> britannica.bec.de> wrote:
> 
> > Is there a point? Our swapcontext is effectively a system call. 
> > If it wasn't, it is still quite expensive for a routine that supposedly
> > only changes %rsp followed by a call, at least for architectures that
> > don't use a register window algorithm.
> 
> It does more than that: it has to swap CPU register, signal mask.

Given that they likely want to implement some form of coroutines, that's
exactly the point of what you do *not* want. As soon as it involves
saving all CPU registers and the signal mask, you can just as well do a
full context switch. For coroutines, that is extremely wasteful on most
architectures as only callee-saved registers really have to be preserved
and the signal mask is constant as well. On x86 that excludes e.g. the
SSE and FPU state, which is quite large.

Joerg

headers

YAMAMOTO Takashi | 10 Aug 2012 06:21

Re: swapcontext() around pthreads

hi,

> On Thu, Aug 09, 2012 at 08:44:05PM +0200, Emmanuel Dreyfus wrote:
>> setjmp and longjmp are claim to not match the requirement because they
>> do not allow different stacks for each execution context.
> 
> But there is a strong correlation between thread and stack, you can not
> switch one without the other. So what is the intended semantics if you
> setcontext() in thread A with a context created by thread B while B is
> currently running on another CPU?

no problem if you played with uc_stack and makecontext appropriately.
i guess it's what glusterfs does.

iirc linux swapcontext excludes the thread base from the context
intentionally to allow this kind of usage.

YAMAMOTO Takashi

> 
> Martin