Primary mailing list for the programming language Lua

Arseny Vakhrushev | 27 Jun 2012 19:31

Lua standard library - thread-unsafety

Hello, everyone!

I have encountered an issue with os.date(). As it's commonly known, Lua is pretty much thread-safe when
properly handled. However, you can't guarantee thread safety of some library functions which hold state
like os.date and math.random().

Personally, I totally am ok with math.random() and I can accept its thread-unsafety, i.e. issuing the same
random number in multiple threads. However, it's worse with os.date() which will issue completely
_WRONG_ results because the internal time structure is filled by racing calls.

The best way would be to use localtime_r() instead of localtime(). The manual states it is sufficient to
check for _POSIX_C_SOURCE >= 1 || _XOPEN_SOURCE || _BSD_SOURCE || _SVID_SOURCE || _POSIX_SOURCE.

The other way would be to write my own os.date(), although it looks the patch could be useful to many people
because the issue is really rare but devastating.

Your opinions are welcome. Thanks!

// Seny

headers

Daurnimator | 27 Jun 2012 20:37

Re: Lua standard library - thread-unsafety

On 28 June 2012 03:31, Arseny Vakhrushev <arseny.vakhrushev <at> gmail.com> wrote:
> The best way would be to use localtime_r() instead of localtime(). The manual states it is sufficient to
check for _POSIX_C_SOURCE >= 1 || _XOPEN_SOURCE || _BSD_SOURCE || _SVID_SOURCE || _POSIX_SOURCE.

lua 5.2 does.
http://www.lua.org/source/5.2/loslib.c.html#l_localtime

headers

Arseny Vakhrushev | 27 Jun 2012 21:00

Re: Lua standard library - thread-unsafety

True. I was talking about Lua 5.1. It would be great if these changes could be backported. LuaJIT is not going
to support Lua 5.2 in recent future, thus a program which links with either Lua or LuaJIT could benefit from
the both equally and symmetrically.

On Jun 27, 2012, at 22:37, Daurnimator wrote:

> On 28 June 2012 03:31, Arseny Vakhrushev <arseny.vakhrushev <at> gmail.com> wrote:
>> The best way would be to use localtime_r() instead of localtime(). The manual states it is sufficient to
check for _POSIX_C_SOURCE >= 1 || _XOPEN_SOURCE || _BSD_SOURCE || _SVID_SOURCE || _POSIX_SOURCE.
> 
> lua 5.2 does.
> http://www.lua.org/source/5.2/loslib.c.html#l_localtime
>

headers

Patrick Rapin | 27 Jun 2012 22:36

Re: Lua standard library - thread-unsafety

> True. I was talking about Lua 5.1. It would be great if these changes could be backported.

There are a number of people who wants to have Lua 5.2 features while
still keeping Lua 5.1 (this is by far not the first such request) !
Come on, you have to make a choice. Either you want new features and
upgrade to 5.2, or need stability and stay at 5.1
Backporting features does not make sense to me, and would be very
difficult to explain clearly in the documentation.
I like the fact that a Lua version (say 5.1) is associated with one
documentation and one API.
Lua release numbers (5.1.1, 5.1.2, etc) should remain bug fixes only
as they currently are.

headers

William Ahern | 28 Jun 2012 01:57

Re: Lua standard library - thread-unsafety

On Wed, Jun 27, 2012 at 10:36:17PM +0200, Patrick Rapin wrote:
> > True. I was talking about Lua 5.1. It would be great if these changes
> > could be backported.
> 
> There are a number of people who wants to have Lua 5.2 features while
> still keeping Lua 5.1 (this is by far not the first such request) !

It seems to me that people don't so much want to keep using Lua 5.1, as they
want to keep using LuaJIT. But those people have a real dilemma on their
hands, because Mike will never add _ENV, nor apparently the pcallk/yieldk
interfaces, which means the LuaJIT and Lua projects have officially parted
ways.

There's no way around this reality. It happened. People need to get over it.
My hypothetical PerlJIT project which compiles Perl 4 is just as much Perl
as LuaJIT is Lua, at this point.

> Come on, you have to make a choice. Either you want new features and
> upgrade to 5.2, or need stability and stay at 5.1
> Backporting features does not make sense to me, and would be very
> difficult to explain clearly in the documentation.
> I like the fact that a Lua version (say 5.1) is associated with one
> documentation and one API.
> Lua release numbers (5.1.1, 5.1.2, etc) should remain bug fixes only
> as they currently are.

I think LuaJIT is more likely to backport small features like this, so the
OP can just settle on LuaJIT.

People just need to stop confusing Lua with LuaJIT. Once they do that they

(Continue reading)

headers

Miles Bader | 28 Jun 2012 07:12

Re: Lua standard library - thread-unsafety

William Ahern <william <at> 25thandClement.com> writes:
> It seems to me that people don't so much want to keep using Lua 5.1,
> as they want to keep using LuaJIT. But those people have a real
> dilemma on their hands, because Mike will never add _ENV

Link?  I've never seen him say that....

-miles

--

-- 
Admiration, n. Our polite recognition of another's resemblance to ourselves.

headers

William Ahern | 28 Jun 2012 19:37

Re: Lua standard library - thread-unsafety

On Thu, Jun 28, 2012 at 02:12:02PM +0900, Miles Bader wrote:
> William Ahern <william <at> 25thandClement.com> writes:
> > It seems to me that people don't so much want to keep using Lua 5.1,
> > as they want to keep using LuaJIT. But those people have a real
> > dilemma on their hands, because Mike will never add _ENV
> 
> Link?  I've never seen him say that....
> 

From: http://www.freelists.org/post/luajit/LuaJIT-Roadmap-20122013

	But there's one important message: compatibility with Lua 5.1 is
	there to stay!

	Many users of LuaJIT, especially those with big code bases, have a
	heavy investment in Lua 5.1-compatible infrastructure, tools,
	frameworks and in-house knowledge. Understandably, they don't want
	to throw away their investment, but still keep up with the newest
	developments.

	As I've previously said, Lua 5.2 provides few tangible benefits.
	LuaJIT already includes the major new features, without breaking
	compatibility. Upgrading to be compatible with 5.2, just for the
	sake of a higher version number, is neither a priority nor a
	sensible move for most LuaJIT users.

	To protect the investment of my users and still provide them with
	new features, LuaJIT 2.1 will stay compatible with Lua 5.1.

From: http://permalink.gmane.org/gmane.comp.lang.lua.luajit/210

(Continue reading)

headers

Mike Pall | 28 Jun 2012 19:58

Re: Lua standard library - thread-unsafety

William Ahern wrote:
> [Quoting Mike Pall:]
>       Upgrading to be compatible with 5.2 [...] is neither a priority	[...]
>
> 	> [...] _ENV? [...]
>       [...] would break compatibility. [with 5.1]

My statements.

> > > [...] because Mike will never add _ENV

Your invalid conclusion.

--Mike

headers

William Ahern | 28 Jun 2012 21:12

Re: Lua standard library - thread-unsafety

On Thu, Jun 28, 2012 at 07:58:25PM +0200, Mike Pall wrote:
> William Ahern wrote:
> > [Quoting Mike Pall:]
> >       Upgrading to be compatible with 5.2 [...] is neither a priority	[...]
> >
> > 	> [...] _ENV? [...]
> >       [...] would break compatibility. [with 5.1]
> 
> My statements.
> 
> > > > [...] because Mike will never add _ENV
> 
> Your invalid conclusion.
> 

I apologize. Never is a long time, and I can't speak for you.

headers

steve donovan | 28 Jun 2012 08:01

Re: Lua standard library - thread-unsafety

On Thu, Jun 28, 2012 at 1:57 AM, William Ahern
<william <at> 25thandclement.com> wrote:
> People just need to stop confusing Lua with LuaJIT. Once they do that they
> won't feel like they're being left behind by Lua 5.2.

If you drop

#define LUAJIT_ENABLE_LUA52COMPAT 1

in LuaJIT's luaconf.h, then the metatables __pairs, __ipairs and __len
are available for tables.

As for re-entry safe standard functions, I'm sure he would be open to
that kind of fix.

Even __gc for tables is part of his roadmap for the next-generation
garbage collector.

steve d.

headers

Arseny Vakhrushev | 28 Jun 2012 07:23

Re: Lua standard library - thread-unsafety


> There are a number of people who wants to have Lua 5.2 features while
> still keeping Lua 5.1 (this is by far not the first such request) !
> Come on, you have to make a choice. Either you want new features and
> upgrade to 5.2, or need stability and stay at 5.1
> Backporting features does not make sense to me, and would be very
> difficult to explain clearly in the documentation.
> I like the fact that a Lua version (say 5.1) is associated with one
> documentation and one API.
> Lua release numbers (5.1.1, 5.1.2, etc) should remain bug fixes only
> as they currently are.

I might personally be confusing features and bug fixes, but I consider the lack of re-entrancy in such vital
functions as os.date() rather a bug than a lack of a feature.

I don't mind Lua and LuaJIT splitting. Moreover, it's natural since it would have been a nightmare for Mike
trying to walk the same way.
I will be considering adapting my projects to support Lua 5.2 in the future. I did thorough coding so it won't
be hard in the sense of API.

But, what I really do mind is stopping the support for Lua 5.1 backporting vital error fixes found while
developing Lua 5.2. That's it. I understand it could be difficult and time consuming particularly for the
Lua project. So, I can get over it looking at Lua 5.2 in the future. But, it seems vital for the LuaJIT project
since it will never inherit those kind of bug fixes entirely by inheriting the Lua 5.2 codebase. Thus, it is
going to be a hard work cherry-picking each of them one by one.

// Seny

headers

Luiz Henrique de Figueiredo | 28 Jun 2012 13:09

Re: Lua standard library - thread-unsafety

> what I really do mind is stopping the support for Lua 5.1 backporting vital error fixes found while
developing Lua 5.2.

Most if not all bug fixes included in 5.1.5 come from the 5.2 development.
Backport of features, however, are never done because minor releases are
bug-fix only:
	http://www.lua.org/versions.html#numbering

headers

liam mail | 28 Jun 2012 13:22

Re: Lua standard library - thread-unsafety

On 28 June 2012 12:09, Luiz Henrique de Figueiredo
<lhf <at> tecgraf.puc-rio.br> wrote:
>> what I really do mind is stopping the support for Lua 5.1 backporting vital error fixes found while
developing Lua 5.2.
>
> Most if not all bug fixes included in 5.1.5 come from the 5.2 development.
> Backport of features, however, are never done because minor releases are
> bug-fix only:
>        http://www.lua.org/versions.html#numbering
>

So am I to understand 5.1.5 has more fixes than listed [1], which is
just a makefile comment ?

[1] http://www.lua.org/bugs.html#5.1.5

Liam

headers

liam mail | 28 Jun 2012 13:26

Re: Lua standard library - thread-unsafety

On 28 June 2012 12:22, liam mail <liam.list <at> googlemail.com> wrote:
> On 28 June 2012 12:09, Luiz Henrique de Figueiredo
> <lhf <at> tecgraf.puc-rio.br> wrote:
>>> what I really do mind is stopping the support for Lua 5.1 backporting vital error fixes found while
developing Lua 5.2.
>>
>> Most if not all bug fixes included in 5.1.5 come from the 5.2 development.
>> Backport of features, however, are never done because minor releases are
>> bug-fix only:
>>        http://www.lua.org/versions.html#numbering
>>
>
>
> So am I to understand 5.1.5 has more fixes than listed [1], which is
> just a makefile comment ?
>
> [1] http://www.lua.org/bugs.html#5.1.5
>
> Liam

Sorry for the noise I see that is a bug in 5.1.5 not a bug fixed in
that release.

Liam

headers

Luiz Henrique de Figueiredo | 28 Jun 2012 13:27

Re: Lua standard library - thread-unsafety

> So am I to understand 5.1.5 has more fixes than listed [1], which is
> just a makefile comment ?
> 
> [1] http://www.lua.org/bugs.html#5.1.5

Lua 5.1.5 fixes bugs in 5.1.4, those listed in
	http://www.lua.org/bugs.html#5.1.4

The bugs listed in [1] are the ones remaining in 5.1.5.

headers

Arseny Vakhrushev | 28 Jun 2012 14:20

Re: Lua standard library - thread-unsafety


>> what I really do mind is stopping the support for Lua 5.1 backporting vital error fixes found while
developing Lua 5.2.
> 
> Most if not all bug fixes included in 5.1.5 come from the 5.2 development.
> Backport of features, however, are never done because minor releases are
> bug-fix only:
> 	http://www.lua.org/versions.html#numbering

Thank you, Luiz! That's what I was talking about. Then I am looking forward to the next bug fixing version.
Meanwhile I'll use a home-made patch.

// Seny

headers

Ignacio Burgueño | 28 Jun 2012 15:52

Re: Lua standard library - thread-unsafety

On Thu, Jun 28, 2012 at 8:09 AM, Luiz Henrique de Figueiredo <lhf <at> tecgraf.puc-rio.br> wrote:

Backport of features, however, are never done because minor releases are
bug-fix only:
http://www.lua.org/versions.html#numbering

So, will this issue will be considered as a bug or not? (will it be listed in http://www.lua.org/bugs.html#5.1.5 ?)

Regards,

Ignacio

headers

Roberto Ierusalimschy | 28 Jun 2012 16:22

Re: Lua standard library - thread-unsafety

> So, will this issue will be considered as a bug or not? (will it be listed
> in http://www.lua.org/bugs.html#5.1.5 ?)

I do not think it is. Standard Lua does not offer threads. Several
other functions in the libraries are not thread safe (and will not be
if C does not offer a proper implementation). The documentation does
not state they are thread safe. (It would be somewhat awkward for a
language that does not support threads to discuss thread safety in its
documentation, but we can add it.) So, to be thread safe is a feature;
not to be is not a bug. Programmers should be aware about that and code
accordingly (e.g., using mutual exclusion to call those functions).

-- Roberto

headers

Roberto Ierusalimschy | 28 Jun 2012 16:33

Re: Lua standard library - thread-unsafety

> I do not think it is. Standard Lua does not offer threads. Several
> other functions in the libraries are not thread safe (and will not be
> if C does not offer a proper implementation). The documentation does
> not state they are thread safe. (It would be somewhat awkward for a
> language that does not support threads to discuss thread safety in its
> documentation, but we can add it.) So, to be thread safe is a feature;
> not to be is not a bug. Programmers should be aware about that and code
> accordingly (e.g., using mutual exclusion to call those functions).

Actually, the documentation for Lua 5.2 already says that os.date may
not be thread safe.

-- Roberto

headers

Arseny Vakhrushev | 28 Jun 2012 16:45

Re: Lua standard library - thread-unsafety


> Actually, the documentation for Lua 5.2 already says that os.date may
> not be thread safe.

It's totally fine on systems without threads or without re-entrant versions of localtime(), i.e. non-POSIX.

The question is would the effort ALREADY MADE to Lua 5.2 to make os.date() re-entrant when possible (e.g. on
POSIX systems) be ported back to Lua 5.1 officially? It doesn't really matter whether it will be reflected
on the bug list or not. It matters whether the Lua 5.1 users may rely on the next bug fixing version to benefit
from it or should handle this case themselves.

// Seny

headers

Roberto Ierusalimschy | 28 Jun 2012 17:56

Re: Lua standard library - thread-unsafety

> The question is would the effort ALREADY MADE to Lua 5.2 to make
> os.date() re-entrant when possible (e.g. on POSIX systems) be ported
> back to Lua 5.1 officially? It doesn't really matter whether it will
> be reflected on the bug list or not. It matters whether the Lua 5.1
> users may rely on the next bug fixing version to benefit from it or
> should handle this case themselves.

It may be backported to 5.1, but there may not be a next bug
fixing version soon, as there are no significant bugs to fix (yet).

-- Roberto

headers

Ignacio Burgueño | 28 Jun 2012 17:23

Re: Lua standard library - thread-unsafety

On Thu, Jun 28, 2012 at 11:22 AM, Roberto Ierusalimschy <roberto <at> inf.puc-rio.br> wrote:

I do not think it is. Standard Lua does not offer threads. Several
other functions in the libraries are not thread safe (and will not be
if C does not offer a proper implementation). The documentation does
not state they are thread safe. (It would be somewhat awkward for a
language that does not support threads to discuss thread safety in its
documentation, but we can add it.) So, to be thread safe is a feature;

But Lua is advertised to be "reentrant and ready to be used in multithreaded code" (PIL, 2nd edition, page 219). In that case, the documentation should clarify which functions aren't thread safe.

If this is not considered a bug and it won't make it to Lua 5.1.6, at least the user that is embedding Lua in a multithreaded application would be aware of which functions needs to be overridden.

As you said in your other message, the docs for Lua 5.2 say that os.date may be not thread safe, but it would be great if it also stated which systems provides a thread safe implementation (or at least, which C function makes it unsafe).

Regards,

Ignacio