Justin Paston-Cooper | 23 Jul 16:44 2013
Picon

Ideas on a fast and tidy CSV library

Dear All,

Recently I have been doing a lot of CSV processing. I initially tried to use the Data.Csv (cassava) library provided on Hackage, but I found this to still be too slow for my needs. In the meantime I have reverted to hacking something together in C, but I have been left wondering whether a tidy solution might be possible to implement in Haskell.

I would like to build a library that satisfies the following: 

1) Run a function <<f :: a_1 -> ... -> a_n -> m (Maybe (b_1, ..., b_n))>>, with <<m>> some monad and the <<a>>s and <<b>>s being input and output.

2) Be able to specify a maximum record string length and output record string length, so that the string buffers used for reading and outputting lines can be reused, preventing the need for allocating new strings for each record.

3) Allocate only once, the memory where the parsed input values, and output values are put.

4) The library's main function should take some kind of data structure describing the types of the function, the function itself and the filenames of input and output (could also be stdin/stdout).

I am not sure yet what would be that best value of <<m>>. I would like to most importantly efficiently, and if possible, purely allow changes in state to a number of variables, such as an aggregation over a certain field in the input. I do not currently have knowledge of the FFI, and how it might be used in this case. I would appreciate any suggestions as to where I should look further.

Regards,

Justin Paston-Cooper
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Ben Gamari | 23 Jul 17:45 2013
Picon

Re: Ideas on a fast and tidy CSV library

Justin Paston-Cooper <paston.cooper <at> gmail.com> writes:

> Dear All,
>
> Recently I have been doing a lot of CSV processing. I initially tried to
> use the Data.Csv (cassava) library provided on Hackage, but I found this to
> still be too slow for my needs. In the meantime I have reverted to hacking
> something together in C, but I have been left wondering whether a tidy
> solution might be possible to implement in Haskell.
>
Have you tried profiling your cassava implementation? In my experience
I've found it's quite quick. If you have an example of a slow path I'm
sure Johan (cc'd) would like to know about it.

> I would like to build a library that satisfies the following:
>
> 1) Run a function <<f :: a_1 -> ... -> a_n -> m (Maybe (b_1, ..., b_n))>>,
> with <<m>> some monad and the <<a>>s and <<b>>s being input and output.
>
> 2) Be able to specify a maximum record string length and output record
> string length, so that the string buffers used for reading and outputting
> lines can be reused, preventing the need for allocating new strings for
> each record.
>
> 3) Allocate only once, the memory where the parsed input values, and output
> values are put.
>
Ultimately this could be rather tricky to enforce. Haskell code
generally does a lot of allocation and the RTS is well optimized to
handle this.

I've often found that trying to shoehorn a non-idiomatic "optimal"
imperative approach into Haskell produces worse performance than the
more readable, idiomatic approach.

I understand this leaves many of your questions unanswered, but I'd give
the idiomatic approach a bit more time before trying to coerce C into
Haskell. Profile, see where the hotspots are and optimize
appropriately. If the profile has you flummoxed, the lists and #haskell
are always willing to help given the time.

Cheers,

- Ben

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Johan Tibell | 23 Jul 22:13 2013
Picon

Re: Ideas on a fast and tidy CSV library

On Tue, Jul 23, 2013 at 5:45 PM, Ben Gamari <bgamari.foss <at> gmail.com> wrote:
> Justin Paston-Cooper <paston.cooper <at> gmail.com> writes:
>
>> Dear All,
>>
>> Recently I have been doing a lot of CSV processing. I initially tried to
>> use the Data.Csv (cassava) library provided on Hackage, but I found this to
>> still be too slow for my needs. In the meantime I have reverted to hacking
>> something together in C, but I have been left wondering whether a tidy
>> solution might be possible to implement in Haskell.
>>
> Have you tried profiling your cassava implementation? In my experience
> I've found it's quite quick. If you have an example of a slow path I'm
> sure Johan (cc'd) would like to know about it.

I'm always interested in examples of code that is not running fast
enough. Send me a reproducible example (preferably as a bug on the
GitHub bug tracker) and I'll take a look.
Justin Paston-Cooper | 25 Jul 11:02 2013
Picon

Re: Ideas on a fast and tidy CSV library

I hadn't yet tried profiling the programme. I actually deleted it a few days ago. I'm going to try to get something new running, and I will report back. On a slightly less related track: Is there any way to use cassava so that I can have pure state and also yield CSV lines while my computation is running instead of everything at the end as would be with the State monad?


On 23 July 2013 22:13, Johan Tibell <johan.tibell <at> gmail.com> wrote:
On Tue, Jul 23, 2013 at 5:45 PM, Ben Gamari <bgamari.foss <at> gmail.com> wrote:
> Justin Paston-Cooper <paston.cooper <at> gmail.com> writes:
>
>> Dear All,
>>
>> Recently I have been doing a lot of CSV processing. I initially tried to
>> use the Data.Csv (cassava) library provided on Hackage, but I found this to
>> still be too slow for my needs. In the meantime I have reverted to hacking
>> something together in C, but I have been left wondering whether a tidy
>> solution might be possible to implement in Haskell.
>>
> Have you tried profiling your cassava implementation? In my experience
> I've found it's quite quick. If you have an example of a slow path I'm
> sure Johan (cc'd) would like to know about it.

I'm always interested in examples of code that is not running fast
enough. Send me a reproducible example (preferably as a bug on the
GitHub bug tracker) and I'll take a look.

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Johan Tibell | 25 Jul 17:53 2013
Picon

Re: Ideas on a fast and tidy CSV library

You can use the Incremental or Streaming modules to get more fine
grained control over when new parsed records are produced.

On Thu, Jul 25, 2013 at 11:02 AM, Justin Paston-Cooper
<paston.cooper <at> gmail.com> wrote:
> I hadn't yet tried profiling the programme. I actually deleted it a few days
> ago. I'm going to try to get something new running, and I will report back.
> On a slightly less related track: Is there any way to use cassava so that I
> can have pure state and also yield CSV lines while my computation is running
> instead of everything at the end as would be with the State monad?
>
>
> On 23 July 2013 22:13, Johan Tibell <johan.tibell <at> gmail.com> wrote:
>>
>> On Tue, Jul 23, 2013 at 5:45 PM, Ben Gamari <bgamari.foss <at> gmail.com>
>> wrote:
>> > Justin Paston-Cooper <paston.cooper <at> gmail.com> writes:
>> >
>> >> Dear All,
>> >>
>> >> Recently I have been doing a lot of CSV processing. I initially tried
>> >> to
>> >> use the Data.Csv (cassava) library provided on Hackage, but I found
>> >> this to
>> >> still be too slow for my needs. In the meantime I have reverted to
>> >> hacking
>> >> something together in C, but I have been left wondering whether a tidy
>> >> solution might be possible to implement in Haskell.
>> >>
>> > Have you tried profiling your cassava implementation? In my experience
>> > I've found it's quite quick. If you have an example of a slow path I'm
>> > sure Johan (cc'd) would like to know about it.
>>
>> I'm always interested in examples of code that is not running fast
>> enough. Send me a reproducible example (preferably as a bug on the
>> GitHub bug tracker) and I'll take a look.
>
>
Justin Paston-Cooper | 21 Aug 14:09 2013
Picon

Re: Ideas on a fast and tidy CSV library

Dear All,

I now have some example code. I have put it on: http://pastebin.com/D9MPmyVd

vectorBinner is simply of type Vector Int -> Int. I am inputting a 1.5GB CSV on stdin, and would like vectorBinner to run over every single record, outputting results as computed, thus running in constant memory. My programme instead quickly approaches full memory use. Is there any way to work around this?

Justin


On 25 July 2013 17:53, Johan Tibell <johan.tibell <at> gmail.com> wrote:
You can use the Incremental or Streaming modules to get more fine
grained control over when new parsed records are produced.

On Thu, Jul 25, 2013 at 11:02 AM, Justin Paston-Cooper
<paston.cooper <at> gmail.com> wrote:
> I hadn't yet tried profiling the programme. I actually deleted it a few days
> ago. I'm going to try to get something new running, and I will report back.
> On a slightly less related track: Is there any way to use cassava so that I
> can have pure state and also yield CSV lines while my computation is running
> instead of everything at the end as would be with the State monad?
>
>
> On 23 July 2013 22:13, Johan Tibell <johan.tibell <at> gmail.com> wrote:
>>
>> On Tue, Jul 23, 2013 at 5:45 PM, Ben Gamari <bgamari.foss <at> gmail.com>
>> wrote:
>> > Justin Paston-Cooper <paston.cooper <at> gmail.com> writes:
>> >
>> >> Dear All,
>> >>
>> >> Recently I have been doing a lot of CSV processing. I initially tried
>> >> to
>> >> use the Data.Csv (cassava) library provided on Hackage, but I found
>> >> this to
>> >> still be too slow for my needs. In the meantime I have reverted to
>> >> hacking
>> >> something together in C, but I have been left wondering whether a tidy
>> >> solution might be possible to implement in Haskell.
>> >>
>> > Have you tried profiling your cassava implementation? In my experience
>> > I've found it's quite quick. If you have an example of a slow path I'm
>> > sure Johan (cc'd) would like to know about it.
>>
>> I'm always interested in examples of code that is not running fast
>> enough. Send me a reproducible example (preferably as a bug on the
>> GitHub bug tracker) and I'll take a look.
>
>

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Johan Tibell | 21 Aug 14:14 2013
Picon

Re: Ideas on a fast and tidy CSV library

As I mentioned, you want to use the Streaming (or Incremental) module.
As the program now stands the call to `decode` causes 1.5 GB of CSV
data to be read as a `Vector (Vector Int)` before any encoding starts.

-- Johan

On Wed, Aug 21, 2013 at 1:09 PM, Justin Paston-Cooper
<paston.cooper <at> gmail.com> wrote:
> Dear All,
>
> I now have some example code. I have put it on: http://pastebin.com/D9MPmyVd
> .
>
> vectorBinner is simply of type Vector Int -> Int. I am inputting a 1.5GB CSV
> on stdin, and would like vectorBinner to run over every single record,
> outputting results as computed, thus running in constant memory. My
> programme instead quickly approaches full memory use. Is there any way to
> work around this?
>
> Justin
>
>
> On 25 July 2013 17:53, Johan Tibell <johan.tibell <at> gmail.com> wrote:
>>
>> You can use the Incremental or Streaming modules to get more fine
>> grained control over when new parsed records are produced.
>>
>> On Thu, Jul 25, 2013 at 11:02 AM, Justin Paston-Cooper
>> <paston.cooper <at> gmail.com> wrote:
>> > I hadn't yet tried profiling the programme. I actually deleted it a few
>> > days
>> > ago. I'm going to try to get something new running, and I will report
>> > back.
>> > On a slightly less related track: Is there any way to use cassava so
>> > that I
>> > can have pure state and also yield CSV lines while my computation is
>> > running
>> > instead of everything at the end as would be with the State monad?
>> >
>> >
>> > On 23 July 2013 22:13, Johan Tibell <johan.tibell <at> gmail.com> wrote:
>> >>
>> >> On Tue, Jul 23, 2013 at 5:45 PM, Ben Gamari <bgamari.foss <at> gmail.com>
>> >> wrote:
>> >> > Justin Paston-Cooper <paston.cooper <at> gmail.com> writes:
>> >> >
>> >> >> Dear All,
>> >> >>
>> >> >> Recently I have been doing a lot of CSV processing. I initially
>> >> >> tried
>> >> >> to
>> >> >> use the Data.Csv (cassava) library provided on Hackage, but I found
>> >> >> this to
>> >> >> still be too slow for my needs. In the meantime I have reverted to
>> >> >> hacking
>> >> >> something together in C, but I have been left wondering whether a
>> >> >> tidy
>> >> >> solution might be possible to implement in Haskell.
>> >> >>
>> >> > Have you tried profiling your cassava implementation? In my
>> >> > experience
>> >> > I've found it's quite quick. If you have an example of a slow path
>> >> > I'm
>> >> > sure Johan (cc'd) would like to know about it.
>> >>
>> >> I'm always interested in examples of code that is not running fast
>> >> enough. Send me a reproducible example (preferably as a bug on the
>> >> GitHub bug tracker) and I'll take a look.
>> >
>> >
>
>

Gmane