Frederik Ramm | 23 Feb 12:34 2011

Very Large PNGs

Dear all,

    I posted this to the cairo list but it occurred to me that this list 
is the appropriate place.

I tried to convert an SVG file to a very large PNG with rsvg-convert.
If the bitmap requested is larger than about 23k x 23k pixels, it is
returned an invalid surface (where if you query the state it says "out
of memory"), happily ignores that and continues to "render" which then
results in no output.

I then delved into the cairo library and found that it uses pixman to 
create the image; and pixman refuses to create an image that takes more 
than INT32_MAX raw bytes. With one pixel using 4 bytes, that means that 
an image with 23.000 x 23.000 pixels is still ok, but if it becomes any 
larger, that was it.

Which was a bit sad for me since the machine I was running this on could 
easily have created an image ten times as big, if only pixman/cairo had 
supported that.

I briefly toyed with adding 64-bit support to the whole bundle but it
became quickly apparent that this is more than an afternoon's job. I
also tried to squeeze at least a little more out of the combo by using
24-bit images instead of 32-bit (no difference since 24-bit images are
internally stored with 32 bit), and even the deprecated 16-bit data type 
(doesn't work since you cannot create a surface for that).

I resorted to dicing my image into smaller chunks but that caused
considerable manual work.
(Continue reading)

Soeren Sandmann | 24 Feb 18:38 2011
Picon
Picon

Re: Very Large PNGs

Frederik Ramm <frederik@...> writes:

> Dear all,
>
>    I posted this to the cairo list but it occurred to me that this
> list is the appropriate place.
>
> I tried to convert an SVG file to a very large PNG with rsvg-convert.
> If the bitmap requested is larger than about 23k x 23k pixels, it is
> returned an invalid surface (where if you query the state it says "out
> of memory"), happily ignores that and continues to "render" which then
> results in no output.
>
> I then delved into the cairo library and found that it uses pixman to
> create the image; and pixman refuses to create an image that takes
> more than INT32_MAX raw bytes. With one pixel using 4 bytes, that
> means that an image with 23.000 x 23.000 pixels is still ok, but if it
> becomes any larger, that was it.

What pixman is trying to avoid is integer overflows. If you pass
an overflowed number to malloc(), you'll get buffer overruns which could
become security issues

Given that size_t on a 32 bit machine is 32 bits, the limit would be 4
GB in any case, but just to be on the safe side, pixman limits it to 2GB
so that the number will fit in a signed integer.

I think it should be safe to use size_t in create_bits() along with new
functions pixman_multiply/add_overflows_size() if you want to fix this.

(Continue reading)

Frederik Ramm | 25 Feb 01:43 2011

Re: Very Large PNGs

Soeren,

Soeren Sandmann wrote:
> I think it should be safe to use size_t in create_bits() along with new
> functions pixman_multiply/add_overflows_size() if you want to fix this.

I guess I'll try my hand at that.

> However, not that pixman is still limited to 16.16 bits of coordinate
> space in various places, so you wouldn't gain that much. This 16 bit
> limitation is something that we would like to get rid of, however. 
> 
> Out of curiosity, exactly how large are the images you are trying to create?

The first use case I had was actually only very slightly above the 
limit; I tried to generate something that was about 24000x32000 pixels.

> In cairo 1.10, you actually can create a 16 bit image, and it is stored
> internally in a 16 bit data type.

I'll have a look at that too and perhaps use whichever way gives me the 
least pain. The 16-bit data type should allow bitmaps of roughly 32k x 
32k without hitting the coordinate space limit, that would already be an 
advantage, and the loss of colour space would probably not be too bad 
for my application.

Just to give some background, what I'm doing is creating raster street 
map images from OpenStreetMap data; I generate an SVG and have it 
rasterized with rsvg/cairo. If you want a full map of a large city where 
all the road names are still readable, you quickly approach the sizes 
(Continue reading)


Gmane