headers
Doruk Fisek | 29 Oct 08:54
Picon

memory consumption problem as collections get bigger

Hi,

 I'm using Tellico for a music collection, my .tc file is about 2 MB.
When I unzip it, it's about 16 MB as xml data.

 When I open the file, Tellico consumes about 250 MB of my computer's
memory. I notice that as the collection grows, so does the memory
Tellico takes up.

 I can't open this collection on my 512 MB machine with other
applications running since it causes huge swap activity.

 Is this normal behavior? If so, is there anyway to improve Tellico to
consume less memory?

                   Doruk

--
FISEK INSTITUTE - http://www.fisek.org.tr
Permalink | Reply |
headers
Robby Stephenson | 29 Oct 15:03

Re: memory consumption problem as collections get bigger

*



On Wednesday 29 October 2008, Doruk Fisek wrote:
> Hi,
>
>  I'm using Tellico for a music collection, my .tc file is about 2 MB.
> When I unzip it, it's about 16 MB as xml data.
>
>  When I open the file, Tellico consumes about 250 MB of my computer's
> memory. I notice that as the collection grows, so does the memory
> Tellico takes up.
>
>  I can't open this collection on my 512 MB machine with other
> applications running since it causes huge swap activity.
>
>  Is this normal behavior? If so, is there anyway to improve Tellico to
> consume less memory?


It's been a chronic problem for some folks, and I've tried a few things with 
reducing the memory usage, but like you point out, it seems to grow too 
much with respect to the file size.

Tellico holds all the collection data in RAM once the file is opened. That's 
obviously not going to work well with really big collections, it's just the 
way I wrote it. If you delete your data file once it's loaded, Tellico can 
save it again completely.

Are there images in the tc file? That's what really bogs things down, too.

Robby
Permalink | Reply |
headers
Doruk Fisek | 29 Oct 17:05
Picon

Re: memory consumption problem as collections get bigger

Wed, 29 Oct 2008 07:03:54 -0700, Robby Stephenson
<robby@...> :

> Tellico holds all the collection data in RAM once the file is opened.
> That's obviously not going to work well with really big collections,
> it's just the way I wrote it. If you delete your data file once it's
> loaded, Tellico can save it again completely.
Even if you store all of the collection in RAM, 250 MB consumption for
a 16 MB xml file (not zipped) seems too much. It's nearly 15 times the
original size.

> Are there images in the tc file? That's what really bogs things down,
> too.
There are none. I know it does, that's why I didn't put any (though I
wanted to).

                   Doruk

--
FISEK INSTITUTE - http://www.fisek.org.tr
Permalink | Reply |
headers
Robby Stephenson | 30 Oct 07:23

Re: memory consumption problem as collections get bigger

On Wednesday 29 October 2008, Doruk Fisek wrote:
> Even if you store all of the collection in RAM, 250 MB consumption for
> a 16 MB xml file (not zipped) seems too much. It's nearly 15 times the
> original size.

A 16 MB XML text file is pretty huge. How many entries?

How are you measuring the memory usage? The virtual column in top includes 
all the KDE libraries, etc. 

I have a test file with 10k entries that's 5 mb unzipped. The shared column 
in top shows 20 m usage.

> > Are there images in the tc file? That's what really bogs things down,
> > too.
>
> There are none. I know it does, that's why I didn't put any (though I
> wanted to).

Are you loading the images at all? I mean, do you have a visible column with 
the images, or using the icon view or anything? It's possible the image 
cache is not working correctly, and the loaded pixmaps, etc. aren't being 
freed, I guess.

Robby
Permalink | Reply |
headers
Doruk Fisek | 30 Oct 07:31
Picon

Re: memory consumption problem as collections get bigger

Wed, 29 Oct 2008 23:23:59 -0700, Robby Stephenson
<robby@...> :

> A 16 MB XML text file is pretty huge. How many entries?
~9000 albums.

I'm not saying it's small but even if you load the whole 16 MB file to
RAM, why does it take up so much space?

> How are you measuring the memory usage? The virtual column in top
> includes all the KDE libraries, etc. 
> I have a test file with 10k entries that's 5 mb unzipped. The shared
> column in top shows 20 m usage.
My file is 1.9M, that's 16 MB unzipped. In KDE System Guard's process
table, Vmsize 277988, Vmrss 243480. (I double checked with top command
too).

> > > Are there images in the tc file? That's what really bogs things
> > > down, too.
> > There are none. I know it does, that's why I didn't put any (though
> > I wanted to).
> Are you loading the images at all? I mean, do you have a visible
> column with the images, or using the icon view or anything? It's
> possible the image cache is not working correctly, and the loaded
> pixmaps, etc. aren't being freed, I guess.
No images, no pixmaps, nothing at all.

                   Doruk

--
FISEK INSTITUTE - http://www.fisek.org.tr
Permalink | Reply |
headers
Robby Stephenson | 30 Oct 07:41

Re: memory consumption problem as collections get bigger

*



On Wednesday 29 October 2008, Doruk Fisek wrote:
> Wed, 29 Oct 2008 23:23:59 -0700, Robby Stephenson
>
> <robby@...> :
> > A 16 MB XML text file is pretty huge. How many entries?
>
> ~9000 albums.
>
> I'm not saying it's small but even if you load the whole 16 MB file to
> RAM, why does it take up so much space?


Certainly a valid question. I'll poke at it some. There's overhead with the 
data structures, but a 10x+ memory usage is way excessive.

Robby
Permalink | Reply |
headers
Benny Malengier | 29 Oct 22:59
Picon

Re: memory consumption problem as collections get bigger



2008/10/29 Doruk Fisek <dfisek-8ahlAQKSdoL+ojMbLx6GBw@public.gmane.org>
Wed, 29 Oct 2008 07:03:54 -0700, Robby Stephenson
<robby-9lFPeden07UgsBAKwltoeQ@public.gmane.org> :

*
> Tellico holds all the collection data in RAM once the file is opened.
> That's obviously not going to work well with really big collections,
> it's just the way I wrote it. If you delete your data file once it's
> loaded, Tellico can save it again completely.
Even if you store all of the collection in RAM, 250 MB consumption for
a 16 MB xml file (not zipped) seems too much. It's nearly 15 times the
original size.

I don't know how tellico does it, but I work for an app, and eg an xml file is  6.9 Mb, however, in reality this is a compressed file (like eg odf or docx format, also xml), uncompressed it is 48.7 Mb. In database format (xml is for export and import), this takes up two times 178 Mb on your hard disk (one for failsafe is a backup of last successfull data). However, because it is a database, one does not need to keep it in memory.
This example just to show things are not so easy to calculate and determine from the size of an xml file.

Benny

_______________________________________________
tellico-users mailing list
tellico-users@...
http://forge.novell.com/mailman/listinfo/tellico-users
Permalink | Reply |
headers
Benny Malengier | 29 Oct 15:25
Picon

Re: memory consumption problem as collections get bigger



2008/10/29 Robby Stephenson <robby-9lFPeden07UgsBAKwltoeQ@public.gmane.org>
On Wednesday 29 October 2008, Doruk Fisek wrote:
*
> Hi,
>
>  I'm using Tellico for a music collection, my .tc file is about 2 MB.
> When I unzip it, it's about 16 MB as xml data.
>
>  When I open the file, Tellico consumes about 250 MB of my computer's
> memory. I notice that as the collection grows, so does the memory
> Tellico takes up.
>
>  I can't open this collection on my 512 MB machine with other
> applications running since it causes huge swap activity.
>
>  Is this normal behavior? If so, is there anyway to improve Tellico to
> consume less memory?

It's been a chronic problem for some folks, and I've tried a few things with
reducing the memory usage, but like you point out, it seems to grow too
much with respect to the file size.

Tellico holds all the collection data in RAM once the file is opened. That's
obviously not going to work well with really big collections, it's just the
way I wrote it. If you delete your data file once it's loaded, Tellico can
save it again completely.

If things are in an sqlite embedded database as eg Amarok collection, you can avoid this. Undo is possible by going back in the database logs. Anyway, with all the database based apps now, people get used to atomic saves in apps that handle collections

Benny

_______________________________________________
tellico-users mailing list
tellico-users@...
http://forge.novell.com/mailman/listinfo/tellico-users
Permalink | Reply |
headers
Robby Stephenson | 29 Oct 15:55

Re: memory consumption problem as collections get bigger

*



Benny Malengier wrote:
> 2008/10/29 Robby Stephenson <robby@... 
>     Tellico holds all the collection data in RAM once the file is
>     opened. That's
>     obviously not going to work well with really big collections, it's
>     just the
>     way I wrote it. If you delete your data file once it's loaded,
>     Tellico can
>     save it again completely.
> 
> 
> If things are in an sqlite embedded database as eg Amarok collection, 
> you can avoid this. Undo is possible by going back in the database logs. 
> Anyway, with all the database based apps now, people get used to atomic 
> saves in apps that handle collections


True. I wish I spoke SQL. I've taken a shot at it before, and had 
problems getting the performance.

Robby
Permalink | Reply |
headers
Benny Malengier | 29 Oct 16:18
Picon

Re: memory consumption problem as collections get bigger



2008/10/29 Robby Stephenson <robby-9lFPeden07UgsBAKwltoeQ@public.gmane.org>
Benny Malengier wrote:
2008/10/29 Robby Stephenson <robby-9lFPeden07UgsBAKwltoeQ@public.gmane.org    Tellico holds all the collection data in RAM once the file is

   opened. That's
   obviously not going to work well with really big collections, it's
   just the
   way I wrote it. If you delete your data file once it's loaded,
   Tellico can
   save it again completely.


If things are in an sqlite embedded database as eg Amarok collection, you can avoid this. Undo is possible by going back in the database logs. Anyway, with all the database based apps now, people get used to atomic saves in apps that handle collections

True. I wish I spoke SQL. I've taken a shot at it before, and had problems getting the performance.

If you can create a model in Django, then you can generate the SQL tables from the code Django creates, and have a first tool to edit it (the Django admin module). Once you have the database tables correct, well, I suppose the real work only starts then ;-)

Benny

_______________________________________________
tellico-users mailing list
tellico-users@...
http://forge.novell.com/mailman/listinfo/tellico-users
Permalink | Reply |
headers
Robby Stephenson | 30 Oct 07:24

Re: memory consumption problem as collections get bigger

*



On Wednesday 29 October 2008, Benny Malengier wrote:
> > True. I wish I spoke SQL. I've taken a shot at it before, and had
> > problems getting the performance.
>
> If you can create a model in Django, then you can generate the SQL tables
> from the code Django creates, and have a first tool to edit it (the
> Django admin module). Once you have the database tables correct, well, I
> suppose the real work only starts then ;-)


Then I'd need to learn python in addition to sql. :)

Robby
Permalink | Reply |

Gmane