diego souza | 6 May 01:30 2013

help understanding zlib space leak

Dear haskellers,

I'd like assistance figuring out a strange space leak using zlib
package.

To make it easier to reproduce I've come up with the following snippet
that pretty much resumes up the problem I'm trying to solve:

> -- omitting imports and function signatures
> encode = compress . L.fromChunks
>  
> main = do { hSetBinaryMode stdout True
>           ; hSetBinaryMode stdin True
>           ; hSetBuffering stdin NoBuffering
>           ; hSetBuffering stdout NoBuffering
>           ; loop []
>           }
>  where loop buff
>          | length buff == 64 = L.hPut stdout (encode buff) >> loop []
>          | otherwise         = do { eof <- hIsEOF stdin
>                                   ; when (not eof) (fmap (: buff) (B.hGetSome stdin 512) >>= loop)
>                                   }

N.B.: Removing the `compress' function from the above code also removes the
      space leak.

Now, feeding the above program with ~8GB worth of binary data:

> $ uname -a
Linux mephisto.localhost.localdomain 3.8.7-1-ARCH #1 SMP PREEMPT Sat Apr 13 09:01:47 CEST 2013 x86_64 GNU/Linux
(Continue reading)

diego souza | 6 May 02:08 2013

Re: help understanding zlib space leak


Sorry, I should've removed the pid number from the output. The
following should be correct:

> $ sudo dd if=/dev/sda bs=4K count=2048K | ./test +RTS -M1M -s >/dev/null                                                   
...
8589934592 bytes (8.6 GB) copied, 243.525 s, 35.3 MB/s
  41,942,119,192 bytes allocated in the heap
     228,827,904 bytes copied during GC
         104,048 bytes maximum residency (6 sample(s))
          24,408 bytes maximum slop
               2 MB total memory in use (0 MB lost due to fragmentation)
...

> $ while pidof test >/dev/null; do ps -o rss= -p $(pidof test); sleep 1; done | tail
32056
32408
32832
33264
33684
34100
34560
34900
35384
35816 # ~ 35MB

Thanks!
~dsouza

Gmane