Joel Reymont | 25 Oct 17:51 2010
Picon

enif_send from a thread after NIF returns

Is it possible to use enif_send to asynchronously message back to Erlang after the NIF returns?

I'm wondering if enif_thread_create could be used to create a thread that continues running and
eventually invokes enif_send.

A practical example of using the above would be asynchronous io.

	Thanks for any guidance, Joel

---
http://twitter.com/wagerlabs

Rapsey | 25 Oct 17:58 2010
Picon

Re: enif_send from a thread after NIF returns

yes it is.
Env = enif_alloc_env();

enif_send(NULL,&Process, Env,Msg);
For every enif_send you need an enif_clear_env after it.
enif_clear_env(Env);

Sergej

On Mon, Oct 25, 2010 at 5:51 PM, Joel Reymont <joelr1 <at> gmail.com> wrote:

> Is it possible to use enif_send to asynchronously message back to Erlang
> after the NIF returns?
>
> I'm wondering if enif_thread_create could be used to create a thread that
> continues running and eventually invokes enif_send.
>
> A practical example of using the above would be asynchronous io.
>
>        Thanks for any guidance, Joel
>
> ---
> http://twitter.com/wagerlabs
>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe <at> erlang.org
>
(Continue reading)

Paul Davis | 25 Oct 18:00 2010
Picon

Re: enif_send from a thread after NIF returns

On Mon, Oct 25, 2010 at 11:51 AM, Joel Reymont <joelr1 <at> gmail.com> wrote:
> Is it possible to use enif_send to asynchronously message back to Erlang after the NIF returns?
>
> I'm wondering if enif_thread_create could be used to create a thread that continues running and
eventually invokes enif_send.
>
> A practical example of using the above would be asynchronous io.
>
>        Thanks for any guidance, Joel
>
> ---
> http://twitter.com/wagerlabs
>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe <at> erlang.org
>
>

Quite possible. There's a simple example of sending a term in my
nif-examples [1] repo and a fairly more complex example in Emonk [2].

[1] http://github.com/davisp/nif-examples
[2] http://github.com/davisp/emonk

Joel Reymont | 25 Oct 18:08 2010
Picon

Re: enif_send from a thread after NIF returns

With a combination of NIF, detaching a thread and enif_send, why would you ever need a port driver?

Are port drivers obsolete?

---
http://twitter.com/wagerlabs

Rapsey | 25 Oct 18:55 2010
Picon

Re: enif_send from a thread after NIF returns

I see no reason to use them over NIFs.

Sergej

On Mon, Oct 25, 2010 at 6:08 PM, Joel Reymont <joelr1 <at> gmail.com> wrote:

> With a combination of NIF, detaching a thread and enif_send, why would you
> ever need a port driver?
>
> Are port drivers obsolete?
>
> ---
> http://twitter.com/wagerlabs
>
>
> ________________________________________________________________
> erlang-questions (at) erlang.org mailing list.
> See http://www.erlang.org/faq.html
> To unsubscribe; mailto:erlang-questions-unsubscribe <at> erlang.org
>
>
Paul Davis | 25 Oct 19:51 2010
Picon

Re: enif_send from a thread after NIF returns

On Mon, Oct 25, 2010 at 12:08 PM, Joel Reymont <joelr1 <at> gmail.com> wrote:
> With a combination of NIF, detaching a thread and enif_send, why would you ever need a port driver?
>
> Are port drivers obsolete?
>
> ---
> http://twitter.com/wagerlabs
>
>

Its a different model really. NIF's have started to allow people to
re-implement almost anything a port driver can do, as well as things
they can't. Though, the question is more about technical debt at this
point.

For instance, if you had a situation that could be solved as a port
using the async thread pool, why would you want to re-implement that
as a NIF? NIF's are extremely awesome and its easy to assume they're
better for everything but there are definitely cases where a port
driver is a better fit.

HTH,
Paul Davis

Joel Reymont | 25 Oct 19:56 2010
Picon

Re: enif_send from a thread after NIF returns


On Oct 25, 2010, at 6:51 PM, Paul Davis wrote:

> For instance, if you had a situation that could be solved as a port
> using the async thread pool, why would you want to re-implement that
> as a NIF?

I think it boils down to ease of use and to "worse is better". 

Yes, NIF requires reimplementing an async thread pool but NIF is so much easier to use!

Want to debug your NIF? Attach a debugger and you are there. The layer between the Erlang VM and your C code is
thin and transparent! Compare this to trying to debug a port driver, even if you manage to fully understand
it's underlying mechanisms.

---
http://twitter.com/wagerlabs

Dave Smith | 25 Oct 20:02 2010
Picon

Re: enif_send from a thread after NIF returns

On Mon, Oct 25, 2010 at 11:56 AM, Joel Reymont <joelr1 <at> gmail.com> wrote:
>
> Want to debug your NIF? Attach a debugger and you are there. The layer between the Erlang VM and your C code is
thin and transparent! Compare this to trying to debug a port driver, even if you manage to fully understand
it's underlying mechanisms.

I've never had any problems debugging a port. As you say, attach a
debugger, set a breakpoint and wait for it to fire...

D.

Joel Reymont | 25 Oct 20:05 2010
Picon

Re: enif_send from a thread after NIF returns


On Oct 25, 2010, at 7:02 PM, Dave Smith wrote:
> I've never had any problems debugging a port. As you say, attach a
> debugger, set a breakpoint and wait for it to fire...

I'm talking about the way port drivers require you to structure and write your code. With NIF you just decode
your arguments and you are there. 

I, for one, have never managed to fully grasp port driver mechanisms so I welcome our NIF overlords!

---
http://twitter.com/wagerlabs

Joel Reymont | 26 Oct 14:08 2010
Picon

Re: enif_send from a thread after NIF returns


On Oct 25, 2010, at 5:00 PM, Paul Davis wrote:

> Quite possible. There's a simple example of sending a term in my
> nif-examples [1] repo and a fairly more complex example in Emonk [2].

It seems that the calling code must wait for the spawned thread to finish (join). The documentation states
that it's impossible to create detached threads and that all threads must be joined.

How do you spawn a thread from a NIF then, such that it continues to run after the spawning NIF returns?

	Thanks, Joel

---
http://twitter.com/wagerlabs

Rapsey | 26 Oct 15:09 2010
Picon

Re: enif_send from a thread after NIF returns

On Tue, Oct 26, 2010 at 2:08 PM, Joel Reymont <joelr1 <at> gmail.com> wrote:

>
> On Oct 25, 2010, at 5:00 PM, Paul Davis wrote:
>
> > Quite possible. There's a simple example of sending a term in my
> > nif-examples [1] repo and a fairly more complex example in Emonk [2].
>
> It seems that the calling code must wait for the spawned thread to finish
> (join). The documentation states that it's impossible to create detached
> threads and that all threads must be joined.
>
> How do you spawn a thread from a NIF then, such that it continues to run
> after the spawning NIF returns?
>

Where in the documentation does it say that? I launch my socket accept
thread in load.

Sergej
Joel Reymont | 26 Oct 15:12 2010
Picon

Re: enif_send from a thread after NIF returns


On Oct 26, 2010, at 2:09 PM, Rapsey wrote:

> Where in the documentation does it say that? I launch my socket accept thread in load.

man erl_driver

/erl_drv_thread_create

---
       The driver creating the thread has the  respon-
       sibility  of joining the thread, via erl_drv_thread_join() , before the
       driver is unloaded. It is not possible to  create  "detached"  threads,
       i.e., threads that don't need to be joined.

   Warning:
       All  created  threads  need  to  be  joined  by the driver before it is
       unloaded. If the driver fails to join all threads created before it  is
       unloaded,  the  runtime  system will most likely crash when the code of
       the driver is unloaded.
---

In the context of a NIF, "driver unloaded" seems to be "NIF exits". The VM consistently crashes when I skip
the join after creating threads. 

---
http://twitter.com/wagerlabs

Rapsey | 26 Oct 15:22 2010
Picon

Re: enif_send from a thread after NIF returns

On Tue, Oct 26, 2010 at 3:12 PM, Joel Reymont <joelr1 <at> gmail.com> wrote:

>
> On Oct 26, 2010, at 2:09 PM, Rapsey wrote:
>
> > Where in the documentation does it say that? I launch my socket accept
> thread in load.
>
> man erl_driver
>
> /erl_drv_thread_create
>
> ---
>       The driver creating the thread has the  respon-
>       sibility  of joining the thread, via erl_drv_thread_join() , before
> the
>       driver is unloaded. It is not possible to  create  "detached"
>  threads,
>       i.e., threads that don't need to be joined.
>
>   Warning:
>       All  created  threads  need  to  be  joined  by the driver before it
> is
>       unloaded. If the driver fails to join all threads created before it
>  is
>       unloaded,  the  runtime  system will most likely crash when the code
> of
>       the driver is unloaded.
> ---
>
(Continue reading)

Joel Reymont | 26 Oct 15:29 2010
Picon

Re: enif_send from a thread after NIF returns


On Oct 26, 2010, at 2:22 PM, Rapsey wrote:

> Does unload/reload/upgrade get called? I have printf in those so that I know what's going on.

I'm sure it does, although I'm not using it. Thanks for the tip, though. I now understand that the unload
"callback" corresponds to driver unload and that's where threads must be joined. 

I'm trying to implement asynchronous fsync, though, so I spawn a thread for each fsync. Joining all those
threads on unload in a long-running server would require me to keep a -huge- stack of thread ids, one per
fsync call. I realize that it's probably a bad idea to launch a thread for each fsync but I'm just experimenting.

How do you work around this in your network server? Do you use a thread pool?

	Thanks, Joel

---
http://twitter.com/wagerlabs

Rapsey | 26 Oct 15:37 2010
Picon

Re: enif_send from a thread after NIF returns

On Tue, Oct 26, 2010 at 3:29 PM, Joel Reymont <joelr1 <at> gmail.com> wrote:

>
> On Oct 26, 2010, at 2:22 PM, Rapsey wrote:
>
> > Does unload/reload/upgrade get called? I have printf in those so that I
> know what's going on.
>
> I'm sure it does, although I'm not using it. Thanks for the tip, though. I
> now understand that the unload "callback" corresponds to driver unload and
> that's where threads must be joined.
>
> I'm trying to implement asynchronous fsync, though, so I spawn a thread for
> each fsync. Joining all those threads on unload in a long-running server
> would require me to keep a -huge- stack of thread ids, one per fsync call. I
> realize that it's probably a bad idea to launch a thread for each fsync but
> I'm just experimenting.
>
> How do you work around this in your network server? Do you use a thread
> pool?
>

I don't use fsync anywhere :)
I have only one accept thread and sockets are non-blocking.

Sergej
Joel Reymont | 26 Oct 15:42 2010
Picon

Re: enif_send from a thread after NIF returns


On Oct 26, 2010, at 2:37 PM, Rapsey wrote:

> I don't use fsync anywhere :)
> I have only one accept thread and sockets are non-blocking.

Do you see unload being called?

Can you trigger it somehow and see if the VM crashes?

The manual says you must join before unloading (while unloading?). 

It's a bug if you are seeing different.

---
http://twitter.com/wagerlabs

Sverker Eriksson | 26 Oct 16:25 2010
Picon
Picon

Re: enif_send from a thread after NIF returns

Joel Reymont wrote:
> On Oct 26, 2010, at 2:09 PM, Rapsey wrote:
>
>   
>> Where in the documentation does it say that? I launch my socket accept thread in load.
>>     
>
> man erl_driver
>
> /erl_drv_thread_create
>
> ---
>        The driver creating the thread has the  respon-
>        sibility  of joining the thread, via erl_drv_thread_join() , before the
>        driver is unloaded. It is not possible to  create  "detached"  threads,
>        i.e., threads that don't need to be joined.
>
>    Warning:
>        All  created  threads  need  to  be  joined  by the driver before it is
>        unloaded. If the driver fails to join all threads created before it  is
>        unloaded,  the  runtime  system will most likely crash when the code of
>        the driver is unloaded.
> ---
>
> In the context of a NIF, "driver unloaded" seems to be "NIF exits". The VM consistently crashes when I skip
the join after creating threads. 
>   
You do not have to join a created thread before the nif returns. If the 
VM is crashing when a thread-creating nif returns then you are probably 
doing something wrong.
(Continue reading)

Joel Reymont | 26 Oct 16:40 2010
Picon

Re: enif_send from a thread after NIF returns


On Oct 26, 2010, at 3:25 PM, Sverker Eriksson wrote:

> A safe way to make sure that your thread is joined before the library is unloaded is to create a resource
object that acts as a handle to your thread. The destructor of the resource can then do join.

That makes total sense and I know understand why emonk does what it does.

	Thanks, Joel

[1] http://github.com/davisp/emonk

---
http://twitter.com/wagerlabs

Joel Reymont | 26 Oct 15:23 2010
Picon

Re: enif_send from a thread after NIF returns


On Oct 26, 2010, at 2:09 PM, Rapsey wrote:

> Where in the documentation does it say that? I launch my socket accept thread in load.

Do you launch your socket accept thread from a NIF?

Where do you join it?

---
http://twitter.com/wagerlabs

Rapsey | 26 Oct 15:29 2010
Picon

Re: enif_send from a thread after NIF returns

On Tue, Oct 26, 2010 at 3:23 PM, Joel Reymont <joelr1 <at> gmail.com> wrote:

>
> On Oct 26, 2010, at 2:09 PM, Rapsey wrote:
>
> > Where in the documentation does it say that? I launch my socket accept
> thread in load.
>
> Do you launch your socket accept thread from a NIF?
>
> Where do you join it?
>

I launch it in the load function of the nif (that gets called
automatically). I don't join it anywhere.

Sergej
Joel Reymont | 26 Oct 15:32 2010
Picon

Re: enif_send from a thread after NIF returns


On Oct 26, 2010, at 2:29 PM, Rapsey wrote:

> I launch it in the load function of the nif (that gets called automatically). I don't join it anywhere. 

What I'm trying to point out is that if you do this from a function and don't join then you'll crash the VM very quickly.

I don't understand why and how it works in your case. My guess is that unload is never called and thus no join is
called for. Again, the docs state that detached threads are impossible and joining is a must.

---
http://twitter.com/wagerlabs

Rapsey | 26 Oct 15:41 2010
Picon

Re: enif_send from a thread after NIF returns

On Tue, Oct 26, 2010 at 3:32 PM, Joel Reymont <joelr1 <at> gmail.com> wrote:

>
> On Oct 26, 2010, at 2:29 PM, Rapsey wrote:
>
> > I launch it in the load function of the nif (that gets called
> automatically). I don't join it anywhere.
>
> What I'm trying to point out is that if you do this from a function and
> don't join then you'll crash the VM very quickly.
>
> I don't understand why and how it works in your case. My guess is that
> unload is never called and thus no join is called for. Again, the docs state
> that detached threads are impossible and joining is a must.
>

Oh sorry, just noticed (it's been a while since I've implemented this). I
join the thread in unload function.

Sergej
Joel Reymont | 26 Oct 14:25 2010
Picon

Re: enif_send from a thread after NIF returns

Also, what is the point of launching threads from a NIF if it has to wait for launched threads to finish?

Does the VM scheduler continue running (not block) while the NIF is waiting in enif_thread_join?

	Thanks, Joel

---
http://twitter.com/wagerlabs


Gmane