headers
karme | 26 Jan 11:33
Picon

stream question / unbound memory?

Hi,

why does the following example seem to need unbound memory?

sh$ { cat << EOF
#!/bin/sh
#|
GC_PRINT_STATS=1
export GC_PRINT_STATS
exec gosh -- "\$0" "\$@"
|#

(use util.stream)

(define (main args)
  (stream-for-each (lambda(x)
                     (write x)
                     (newline))
                   (port->stream (current-input-port) read))
  0)
EOF
} > test.scm \
&& chmod +x test.scm \
&& seq 10000000\
|./test.scm 2>&1 > /dev/null \
| grep --line-buffered heapsize |cut -f 2 -d ">"

output should be something like:
heapsize = 262144 bytes
heapsize = 786432 bytes
(Continue reading)

Permalink | Reply |
headers
karme | 26 Jan 12:13
Picon

Re: stream question / unbound memory?

sorry, my fault
if i put a (gc) in there everything looks alright

{ cat << EOF
#!/bin/sh
#|
GC_PRINT_STATS=1
export GC_PRINT_STATS
exec gosh -- "\$0" "\$@"
|#

(use util.stream)

(define (main args)
  (stream-for-each (lambda(x)
                     (gc)
                     (write x)
                     (newline))
                   (port->stream (current-input-port) read))
  0)
EOF
} > test.scm \
&& chmod +x test.scm \
&& seq 10000000\
|./test.scm 2>&1 > /dev/null \
| grep --line-buffered heapsize |cut -f 2 -d ">"

output:
 heapsize = 3338240 bytes
 heapsize = 3338240 bytes
(Continue reading)

Permalink | Reply |
headers
Stas Boukarev | 26 Jan 12:39
Picon

Re: stream question / unbound memory?

karme <at> karme.de writes:

> sorry, my fault
> if i put a (gc) in there everything looks alright
Doesn't look like your fault, the purpose of the garbage collector is to
automatically reclaim memory.

--

-- 
With best regards, Stas.

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-d2d
Permalink | Reply |
headers
Shiro Kawai | 26 Jan 12:53
Favicon

Re: stream question / unbound memory?

Yeah, I just found the same thing.  However, it's supposed to work
without explicit gc, so there's something wrong somewhere.

Since explicit gc eliminates the leak, I think it's not a "hard"
pointer, meaning there's a live object that points to the head
of the stream.   Rather I think what's happening is the failure
scenario described in Hans Boehm's paper
( www.hpl.hp.com/personal/Hans_Boehm/gc/bounds.ps ).
That is, at some point a false pointer that points to the
stream head is created somehow; if you collect the stream head
by explicit gc call before the false pointer appears, it runs
in bounded memory.

What's exactly causing the false pointer is still unknown, though.

--shiro

From: karme <at> karme.de
Subject: Re: [Gauche-devel] stream question / unbound memory?
Date: Thu, 26 Jan 2012 12:13:16 +0100

> sorry, my fault
> if i put a (gc) in there everything looks alright
[snip]

------------------------------------------------------------------------------
Keep Your Developer Skills Current with LearnDevNow!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
(Continue reading)

Permalink | Reply |
headers
KOGURO, Naoki | 27 Jan 15:03

Re: stream question / unbound memory?

It seems to me that explicit gc doesn't free memory. A lot of
"heapsize = 3338240 bytes" messages in each gc cheat us. Let's try to
add "| uniq" and wait for a while.

% seq 1000000 | ./test.scm 2>&1 >/dev/null | grep --line-buffered
heapsize |cut -f 2 -d ">" | uniq
 heapsize = 262144 bytes
 heapsize = 786432 bytes
 heapsize = 1875968 bytes
 heapsize = 3338240 bytes
 heapsize = 4452352 bytes
 heapsize = 5939200 bytes
 heapsize = 7921664 bytes
...

2012/1/26 Shiro Kawai <shiro <at> lava.net>:
> Yeah, I just found the same thing.  However, it's supposed to work
> without explicit gc, so there's something wrong somewhere.
>
> Since explicit gc eliminates the leak, I think it's not a "hard"
> pointer, meaning there's a live object that points to the head
> of the stream.   Rather I think what's happening is the failure
> scenario described in Hans Boehm's paper
> ( www.hpl.hp.com/personal/Hans_Boehm/gc/bounds.ps ).
> That is, at some point a false pointer that points to the
> stream head is created somehow; if you collect the stream head
> by explicit gc call before the false pointer appears, it runs
> in bounded memory.
>
> What's exactly causing the false pointer is still unknown, though.
(Continue reading)

Permalink | Reply |
headers
KOGURO, Naoki | 27 Jan 21:47

Re: stream question / unbound memory?

Hmm, I think the current implementation of util.stream doesn't work
correctly in gc point of view.

stream-for-each is defined like this.
(define (stream-for-each proc . strms)
  (cond ((null? strms)
         (error "no stream arguments to stream-for-each"))
        ((not (every stream? strms))
         (error "non-stream argument to stream-for-each"))
        (else (let loop ((strms strms))
                (if (not (any stream-null? strms))
                  (begin (apply proc (map stream-car strms))
                         (loop (map stream-cdr strms))))))))

The passed stream list is consumed in let loop, but the first element
of the list remains in stack. So we need to wait the end of
stream-for-each for the unbound memory.

I've misunderstood something?

2012/1/27 KOGURO, Naoki <naoki <at> koguro.net>:
> It seems to me that explicit gc doesn't free memory. A lot of
> "heapsize = 3338240 bytes" messages in each gc cheat us. Let's try to
> add "| uniq" and wait for a while.
>
> % seq 1000000 | ./test.scm 2>&1 >/dev/null | grep --line-buffered
> heapsize |cut -f 2 -d ">" | uniq
>  heapsize = 262144 bytes
>  heapsize = 786432 bytes
>  heapsize = 1875968 bytes
(Continue reading)

Permalink | Reply |
headers
KOGURO, Naoki | 27 Jan 22:06

Re: stream question / unbound memory?

If we define stream-for-each like this, the heap size looks like constant.

(define (stream-for-each proc . strms)
  (cond ((null? strms)
         (error "no stream arguments to stream-for-each"))
        ((not (every stream? strms))
         (error "non-stream argument to stream-for-each"))
        (else (let loop ((ss strms))
                (set! strms #f)  ; a quick hack for unbound memory
                (if (not (any stream-null? ss))
                  (begin (apply proc (map stream-car ss))
                         (loop (map stream-cdr ss))))))))

2012/1/28 KOGURO, Naoki <naoki <at> koguro.net>:
> Hmm, I think the current implementation of util.stream doesn't work
> correctly in gc point of view.
>
> stream-for-each is defined like this.
> (define (stream-for-each proc . strms)
>  (cond ((null? strms)
>         (error "no stream arguments to stream-for-each"))
>        ((not (every stream? strms))
>         (error "non-stream argument to stream-for-each"))
>        (else (let loop ((strms strms))
>                (if (not (any stream-null? strms))
>                  (begin (apply proc (map stream-car strms))
>                         (loop (map stream-cdr strms))))))))
>
> The passed stream list is consumed in let loop, but the first element
> of the list remains in stack. So we need to wait the end of
(Continue reading)

Permalink | Reply |
headers
Shiro Kawai | 27 Jan 23:15
Favicon

Re: stream question / unbound memory?

You're right about the current implemntation, but I've already
tried the modification you suggested and saw it didn't solve the
leak in my environment.   I also tried to clear garbages in the stack
for each iteration, which didn't work either.  Inserting gc
inside the loop sometimes solves, so I'm inclined to believe
it's a false pointer (as opposed to your hypothesis, which does
have a real live pointer to the stream head).

--shiro

From: "KOGURO, Naoki" <naoki <at> koguro.net>
Subject: Re: [Gauche-devel] stream question / unbound memory?
Date: Sat, 28 Jan 2012 06:06:53 +0900

> If we define stream-for-each like this, the heap size looks like constant.
> 
> (define (stream-for-each proc . strms)
>   (cond ((null? strms)
>          (error "no stream arguments to stream-for-each"))
>         ((not (every stream? strms))
>          (error "non-stream argument to stream-for-each"))
>         (else (let loop ((ss strms))
>                 (set! strms #f)  ; a quick hack for unbound memory
>                 (if (not (any stream-null? ss))
>                   (begin (apply proc (map stream-car ss))
>                          (loop (map stream-cdr ss))))))))
> 
> 
> 2012/1/28 KOGURO, Naoki <naoki <at> koguro.net>:
>> Hmm, I think the current implementation of util.stream doesn't work
(Continue reading)

Permalink | Reply |
headers
KOGURO, Naoki | 27 Jan 23:50

Re: stream question / unbound memory?

It's weird. I have a question about your false pointer hypothesis.

If there is a false pointer, explicit gc unbind memory, I think.
Because implicit gc and explicit gc calls the same function
(GC_try_to_collect_inner).

Or if the calling (gc) disturbs the state in gauche and it removes a
false pointer, does calling other function like (print "Hello, world")
instead of (gc) unbind memory? (I tried it, but I saw the increasing
heap size.)

2012/1/28 Shiro Kawai <shiro <at> lava.net>:
> You're right about the current implemntation, but I've already
> tried the modification you suggested and saw it didn't solve the
> leak in my environment.   I also tried to clear garbages in the stack
> for each iteration, which didn't work either.  Inserting gc
> inside the loop sometimes solves, so I'm inclined to believe
> it's a false pointer (as opposed to your hypothesis, which does
> have a real live pointer to the stream head).
>
> --shiro
>
>
> From: "KOGURO, Naoki" <naoki <at> koguro.net>
> Subject: Re: [Gauche-devel] stream question / unbound memory?
> Date: Sat, 28 Jan 2012 06:06:53 +0900
>
>> If we define stream-for-each like this, the heap size looks like constant.
>>
>> (define (stream-for-each proc . strms)
(Continue reading)

Permalink | Reply |
headers
Shiro Kawai | 28 Jan 00:07
Favicon

Re: stream question / unbound memory?

My hypothesis is that false pointer is *created* at some moment
during iteration, but not before the point where, if I insert gc,
the leak stops.  So at the moment explicit gc runs there's no
false pointer, which causes the stream head to be collected.
Otherwise the explicit gc won't stop the leak as you suggested.

It does sound unlikely, though.  I'll take a look again to understand
what's going on.

--shiro

From: "KOGURO, Naoki" <naoki <at> koguro.net>
Subject: Re: [Gauche-devel] stream question / unbound memory?
Date: Sat, 28 Jan 2012 07:50:03 +0900

> It's weird. I have a question about your false pointer hypothesis.
> 
> If there is a false pointer, explicit gc unbind memory, I think.
> Because implicit gc and explicit gc calls the same function
> (GC_try_to_collect_inner).
> 
> Or if the calling (gc) disturbs the state in gauche and it removes a
> false pointer, does calling other function like (print "Hello, world")
> instead of (gc) unbind memory? (I tried it, but I saw the increasing
> heap size.)
> 
> 2012/1/28 Shiro Kawai <shiro <at> lava.net>:
>> You're right about the current implemntation, but I've already
>> tried the modification you suggested and saw it didn't solve the
>> leak in my environment.   I also tried to clear garbages in the stack
(Continue reading)

Permalink | Reply |

Gmane