gmane.comp.lang.lua.general

Very large string loading into Lua

I was doing some benchmarking on a base64-encoding routine and was  
hoping to change it from the form:
   ptr = malloc(large-enough)
   for all data
      base64-decode

into
   init luaL_buffer
   for all data
       if not enough buffer left, luaL_addsize and get new ptr
       base64-decode

It turns out that this isn't so nice, particularly when the data size  
is much larger than LUAL_BUFFERSIZE  (on order of 10s of MB)

Has there been any thought on how one could create an arbitrary string- 
region that one could write to and then finalize?
Perhaps even be able to convert a userdata object into a string in  
place (and freeing it if the string value already existed due to  
interning)

Another feature that would probably be useful (particularly for string- 
building) is for userdata to be resizeable...

Any thoughts?

headers

Geoff Richards | 25 Sep 22:16

Re: Very large string loading into Lua

On Thu, Sep 25, 2008 at 04:05:23PM -0400, Thomas Harning wrote:
> ...
>
> Has there been any thought on how one could create an arbitrary string- 
> region that one could write to and then finalize?
> Perhaps even be able to convert a userdata object into a string in place 
> (and freeing it if the string value already existed due to interning)
>
> Another feature that would probably be useful (particularly for string- 
> building) is for userdata to be resizeable...
>
> Any thoughts?

I wrote a C module which has a resizeable buffer managed as a userdata,
and the finished buffer data can be converted into a Lua string with
'tostring()', but since it's just using the normal Lua API that requires
a new copy being made for Lua's string value.  I suppose if it was built
in then that could be avoided (if Lua just marked it as read-only and
added it to the string pool or whatever).

Anyway, if it helps:

    http://www.daizucms.org/lua/library/memoryfile/

You can write to it (and read data back) by treating the userdatas as file
handles.

--

-- 

--- Geoff Richards -------------><-------------- http://ungwe.org/ ---

(Continue reading)