Anatoly Yakovenko | 16 Apr 20:03 2013
Picon

trying to understand out of memory exceptions

-- So why does this code run out of memory?

import Control.DeepSeq
import System.IO
import qualified Data.ByteString.Char8 as BS

scanl' :: NFData a => (a -> b -> a) -> a -> [b] -> [a]
scanl' f q ls =  q : (case ls of
                        []   -> []
                        x:xs -> let q' = f q x
                                in q' `deepseq` scanl' f q' xs)


main = do
   file <- openBinaryFile "/dev/zero" ReadMode
   chars <- BS.hGetContents file
   let rv = drop 100000000000 $ scanl' (+) 0 $ map fromEnum $ BS.unpack chars
   print (head rv)

-- my scanl' implementation seems to do the right thing, because 

main = print $ last $ scanl' (+) (0::Int) [0..]

-- runs without blowing up.  so am i creating a some thunk here?  or is hGetContents storing values?  any way to get the exception handler to print a trace of what caused the allocation?

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Clark Gaebel | 16 Apr 20:29 2013
Picon
Picon

Re: trying to understand out of memory exceptions

See the comment for hGetContents:

"This function reads chunks at a time, doubling the chunksize on each read. The final buffer is then realloced to the appropriate size. For files > half of available memory, this may lead to memory exhaustion. Consider using readFile in this case."


Maybe try lazy bytestrings?

  - Clark

On Tuesday, April 16, 2013, Anatoly Yakovenko wrote:
-- So why does this code run out of memory?

import Control.DeepSeq
import System.IO
import qualified Data.ByteString.Char8 as BS

scanl' :: NFData a => (a -> b -> a) -> a -> [b] -> [a]
scanl' f q ls =  q : (case ls of
                        []   -> []
                        x:xs -> let q' = f q x
                                in q' `deepseq` scanl' f q' xs)


main = do
   file <- openBinaryFile "/dev/zero" ReadMode
   chars <- BS.hGetContents file
   let rv = drop 100000000000 $ scanl' (+) 0 $ map fromEnum $ BS.unpack chars
   print (head rv)

-- my scanl' implementation seems to do the right thing, because 

main = print $ last $ scanl' (+) (0::Int) [0..]

-- runs without blowing up.  so am i creating a some thunk here?  or is hGetContents storing values?  any way to get the exception handler to print a trace of what caused the allocation?

_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Anatoly Yakovenko | 16 Apr 20:33 2013
Picon

Re: trying to understand out of memory exceptions

ah, doh, my mistake.  i accidently pulled in Strict version of bytestring.  the Lazy works file :).  I have a much more complex program that isn't working correctly which i was trying to simplify and looks like i added an error :)


On Tue, Apr 16, 2013 at 11:29 AM, Clark Gaebel <cgaebel <at> uwaterloo.ca> wrote:
See the comment for hGetContents:

"This function reads chunks at a time, doubling the chunksize on each read. The final buffer is then realloced to the appropriate size. For files > half of available memory, this may lead to memory exhaustion. Consider using readFile in this case."


Maybe try lazy bytestrings?

  - Clark

On Tuesday, April 16, 2013, Anatoly Yakovenko wrote:
-- So why does this code run out of memory?

import Control.DeepSeq
import System.IO
import qualified Data.ByteString.Char8 as BS

scanl' :: NFData a => (a -> b -> a) -> a -> [b] -> [a]
scanl' f q ls =  q : (case ls of
                        []   -> []
                        x:xs -> let q' = f q x
                                in q' `deepseq` scanl' f q' xs)


main = do
   file <- openBinaryFile "/dev/zero" ReadMode
   chars <- BS.hGetContents file
   let rv = drop 100000000000 $ scanl' (+) 0 $ map fromEnum $ BS.unpack chars
   print (head rv)

-- my scanl' implementation seems to do the right thing, because 

main = print $ last $ scanl' (+) (0::Int) [0..]

-- runs without blowing up.  so am i creating a some thunk here?  or is hGetContents storing values?  any way to get the exception handler to print a trace of what caused the allocation?


_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe <at> haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe

Gmane