headers
squeak414 | 2 May 22:35
Picon
Favicon

Linux locks up when handling large data sets

Hi, I've been testing Squeak's ability to handle large amounts of data. This
snippet:

testArrayFilling
	| startTime endTime iArray jArray kArray |
	iArray := Array ofSize: 100.
	1
		to: 100
		do: [:i |
			jArray := Array ofSize: 1000.
			1
				to: 1000
				do: [:j |
					startTime := Time millisecondClockValue.
					kArray := Array ofSize: 1000.
					1
						to: 1000
						do: [:k | kArray at: k put: Object new].
					jArray at: j put: kArray.
					endTime := Time millisecondClockValue.
					Transcript cr; show: i asString , ',' , j asString , ' ' , (endTime -
startTime) asFloat asString.
					startTime := Time millisecondClockValue].
			iArray at: i put: jArray].
	Transcript cr; show: 'Finished'

creates about 5 million objects , then the image freezes. When I run on the same
machine under Windows, it happily continues until the short of memory warning (
about 70 Million objects in my case). The VM is 3.7 in both cases, the image
Damien's 3.9 development image.
(Continue reading)

Permalink | Reply |
headers
johnps11 | 2 May 23:58
Favicon

Re: Linux locks up when handling large data sets

Hi Stan!

Have you tried using the -mmap option when starting Squeak?  I notice that
according to the squeakvm man page:

  squeak uses a dynamic heap by default with the maximum size  set  to
  75%  of the available virtual memory or 1 gigabyte, whichever is smaller.

Perhaps Windows doesn't have this limit.  Is one instance of Object bigger
than 200 bytes? If so you'd expect it to oom.

Assuming you have 2 gig of virtual memory, try starting the VM with

-mmap  1500m

as an option and see if it goes further.

Note you can temporarily increase your virtual memory by running (as root):

dd if=/dev/zero of=/tmp/swapfile bs=1024 count=1M
mkswap /tmp/swapfile
swapon /tmp/swapfile

This will increase your VM by 1 Gigabyte - note that on an intel machine
your VM is usually restricted to a max of about 2gig (unless you start the
kernel with the right options, and you have a large memory machine, and
you have a kernel built to support over 4gig of memory).  I'm also
assuming you have a spare gig of space in /tmp . This lasts till you
reboot or until you run
(Continue reading)

Permalink | Reply |
headers
squeak414 | 3 May 21:50
Picon
Favicon

Re: Linux locks up when handling large data sets

Quoting johnps11 <at> bigpond.com:

> Hi Stan!
>
> Have you tried using the -mmap option when starting Squeak?  I notice that
> according to the squeakvm man page:
>
>   squeak uses a dynamic heap by default with the maximum size  set  to
>   75%  of the available virtual memory or 1 gigabyte, whichever is smaller.
>
> Perhaps Windows doesn't have this limit.  Is one instance of Object bigger
> than 200 bytes? If so you'd expect it to oom.
>
> Assuming you have 2 gig of virtual memory, try starting the VM with
>
> -mmap  1500m
>
> as an option and see if it goes further.
>
> Note you can temporarily increase your virtual memory by running (as root):
>
> dd if=/dev/zero of=/tmp/swapfile bs=1024 count=1M
> mkswap /tmp/swapfile
> swapon /tmp/swapfile
>
> This will increase your VM by 1 Gigabyte - note that on an intel machine
> your VM is usually restricted to a max of about 2gig (unless you start the
> kernel with the right options, and you have a large memory machine, and
> you have a kernel built to support over 4gig of memory).  I'm also
> assuming you have a spare gig of space in /tmp . This lasts till you
(Continue reading)

Permalink | Reply |
headers
johnps11 | 4 May 00:22
Favicon

Re: Linux locks up when handling large data sets

> Quoting johnps11 <at> bigpond.com:
>
>> Hi Stan!
<snipped large post>
>
> Hi John, not confusing- an excellent response, thanks.
>
> With the memory option it also cruises on under Linux, until it freezes at
> 70
> million objects.
>
> While it's still loading vmstat shows:
>
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id
> wa
>  2  0  32528  38552  22792 758560    0    0   183   520  295 2399 66  9 17
>  7
>  3  0  32528  38420  22800 758568    0    0     0     3  254 1779 92  8  0
>  0
>  4  0  32528  38420  22804 758568    0    0     0     6  234 2100 93  7  0
>  0
>  4  0  32528  38420  22804 758568    0    0     0     0  239 2125 93  7  0
>  0
>  2  0  32528  38420  22808 758568    0    0     0     0  233 1379 95  5  0
>  0
>  2  0  32528  38420  22816 758568    0    0     0     1  251 2255 92  8  0
>  0
>  2  0  32528  38296  22824 758568    0    0     0     3  244 1963 94  6  0
(Continue reading)

Permalink | Reply |
headers
David T. Lewis | 3 May 15:20
Picon
Favicon

Re: Linux locks up when handling large data sets

On Fri, May 02, 2008 at 10:36:36PM +0200, squeak414 <at> free.fr wrote:
> Hi, I've been testing Squeak's ability to handle large amounts of data. This
> snippet:
> 
> testArrayFilling
> 	| startTime endTime iArray jArray kArray |
> 	iArray := Array ofSize: 100.
> 	1
> 		to: 100
> 		do: [:i |
> 			jArray := Array ofSize: 1000.
> 			1
> 				to: 1000
> 				do: [:j |
> 					startTime := Time millisecondClockValue.
> 					kArray := Array ofSize: 1000.
> 					1
> 						to: 1000
> 						do: [:k | kArray at: k put: Object new].
> 					jArray at: j put: kArray.
> 					endTime := Time millisecondClockValue.
> 					Transcript cr; show: i asString , ',' , j asString , ' ' , (endTime -
> startTime) asFloat asString.
> 					startTime := Time millisecondClockValue].
> 			iArray at: i put: jArray].
> 	Transcript cr; show: 'Finished'
> 
> creates about 5 million objects , then the image freezes. When I run on the same
> machine under Windows, it happily continues until the short of memory warning (
> about 70 Million objects in my case). The VM is 3.7 in both cases, the image
(Continue reading)

Permalink | Reply |
headers
stan shepherd | 3 May 22:22
Picon
Favicon

Re: Linux locks up when handling large data sets


David T. Lewis wrote:
> 
> 
> 
> You are probably just growing your image to the point where the operating
> system starts swapping. The image will appear to be unresponsive, but if
> you interrupt it with <alt>period, it will eventually wake up and return
> control to you.
> 

David, my image doesn't respond to <alt>period, although it is still using
processor. It didn't sort itself out when left overnight.

David T. Lewis wrote:
> 
> The Unix VM automatically allocates memory from the operating system
> as the Squeak object memory grows. This has the nice property of making
> the image just as big as it needs to be without you worrying about it,
> but it also means that if you write code that allocates far more memory
> that is physically available, the operating system is going to start
> thrashing to the point where Squeak becomes unusable. It will not really
> be frozen though; if you are patient enough you can still save the image,
> install more memory on your computer, and restart it ;-)
> 
> johnps11 <at> bigpond.com gave so good tips on how to control VM memory
> allocation for the Unix VM.
> 
>
(Continue reading)

Permalink | Reply |

Gmane