11 Dec 16:14 2012
Large, numerical calculations in Haskell
Emil Hedevang <emilhedevang <at> gmail.com>
2012-12-11 15:14:23 GMT
2012-12-11 15:14:23 GMT
Hi Haskell Cafe,
I need to perform very large numerical computations requiring tens of GB of memory. The computations consist essentially of generation of random numbers and discrete convolutions of large arrays of random numbers with somewhat smaller kernel arrays. Should I use Haskell and call some C libraries when necessary? Should I just write everything in C? Or should I use e.g. Chapel (chapel.cray.com)?
To be more specific, a very large array X is filled with random numbers drawn independently from some distribution. Then, the discrete convolution Y = K * X of X with a somewhat smaller kernel array K is calculated.
Typical sizes of X and K could be 1000 x 1000 x 1000 and 100 x 100 x 100, respectively. The storage of X alone requires 8 GB of memory. Luckily I have recently been blessed with a well-equipped workstation with 128 GB RAM and two 8-core Xeon E5-2650 running at 2.0 GHz.
I see immediate opportunities for parallelism. Firstly, parallel random number generators could be employed to populate the array X (e.g. SPRNG). Secondly, since the size of the kernel K is much smaller than the size of the array X, the discrete convolution can split into several smaller convolutions (see e.g. Kim and Strintzis (1980)). Finally, the discrete convolutions can be computed efficiently using Fourier methods (using e.g. FFTW).
It seems fairly straightforward to implement it all in C, perhaps with some OpenMP pragmas scattered around in the code, since the heavy numerics would be performed in preexisting libraries (SPRNG and FFTW),
I would like to use Haskell, since I like Haskell and find it tremendously fascinating, but I will not use it for the sake of just using it. I am willing to pay a small price in terms of performance, say, twice the run time, if I in turn will get more convenience and less suffering while coding. Therefore, would you claim it to be feasible to use Haskell? I suppose it very much depends on the quality of the Haskell FFI bindings to parallel random number generators and fast Fourier transforms. Unfortunately, I find it difficult to assess these qualities.
_______________________________________________ Haskell-Cafe mailing list Haskell-Cafe <at> haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe