[reportlab-users] ValueError: no memory after 4GB

[Shaw, Theodore]

Hi,

I'm trying to convert an SVG to a BMP that's about 100,000 pixels by
100,000 pixels. My code looks like

drawing = svg2rlg( 'infile.svg')
renderPM.drawToFile(drawing, 'outfile.bmp', fmt='BMP')

When I run it I get "ValueError: _renderPM.gstate: no memory."

By attempting this conversion with progressively larger files and
monitoring the memory usage, I've found that the issue occurs when the
memory consumption exceeds about 4GB. I've checked that I'm using a 64
bit version of Python and have been able to create large arrays that
consume up to 64GB of memory (my total amount of RAM), so it seems like
the issue is specific to reportlab.

Is this expected? Is there any workaround?

Thank you very much.

_______________________________________________
reportlab-users mailing list
reportl...@lists2.reportlab.com
https://pairlist2.pair.net/mailman/listinfo/reportlab-users

[Robin Becker]

Hi Theodore,

sorry you are having a problem.

At the very least I expect the BMP to need 100000 x 100000 x 3 bytes which is at least 30Gb; since the image is
converted by PIL it may well require twice that if things go well.

However, looking in the code that message is output by the initial allocation of the in memory image buffer for the
gstate object) which is trying to allocate w * h * nchan bytes (ie 30Gb) using this code

p->buf = PyMem_Malloc(n=w*h*nchan);

so the python memory manager returns NULL and the gstate creation fails and returns NULL to python which handles the
error. Given that python is telling you that it cannot allocate the memory probably means that your process is either
constrained in some way or that this additional 30Gb is too much.

If you are using a really modern version of reportlab you can try to use the other available backend which we built
recently rlPyCairo (needs version >=3.5.61). You can pip install rlPyCairo and follow the instructions see
https://pypi.org/project/rlPyCairo/. I doubt that this will actually improve things as you will have even more memory
used by additional python code and modules.

To assist you in your debugging it might be useful to use methods from this page

https://stackoverflow.com/questions/938733/total-memory-used-by-python-process

although even you should be able to allocate the required memory it may not be possible if the memory has become
fragmented ie the 30Gb may not be contiguous.

In addition PIL has limits on the image size and will issue warnings and/or fail when trying to deal with such images.

You could try simple experiments in your environment ie

$ python -c"from reportlab.graphics._renderPM import gstate;gs=gstate(10000,10000)"
$ python -c"from reportlab.graphics._renderPM import gstate;gs=gstate(100000,100000)"

on my machine the first is OK, but the second fails, but I only have 16Gb memory. I suppose the above is the minimal
case ie no extra memory except that used by the import and extension.


--
Robin Becker

On 06/04/2021 20:04, Shaw, Theodore wrote:
> Hi,
>
> I'm trying to convert an SVG to a BMP that's about 100,000 pixels by 100,000 pixels. My code looks like
>
> drawing = svg2rlg( 'infile.svg')
> renderPM.drawToFile(drawing, 'outfile.bmp', fmt='BMP')
>
> When I run it I get "ValueError: _renderPM.gstate: no memory."

............

[Tim Roberts]

Shaw, Theodore wrote:
>
> I'm trying to convert an SVG to a BMP that's about 100,000 pixels by
> 100,000 pixels. My code looks like

> ...

> By attempting this conversion with progressively larger files and
> monitoring the memory usage, I've found that the issue occurs when the
> memory consumption exceeds about 4GB. I've checked that I'm using a 64
> bit version of Python and have been able to create large arrays that
> consume up to 64GB of memory (my total amount of RAM), so it seems
> like the issue is specific to reportlab.

No, it's actually an issue in Windows.  The Windows GDI BMP engine
cannot handle bitmaps with coordinates larger than 15 bits (32767).

--
Tim Roberts, ti...@probo.com
Providenza & Boekelheide, Inc.

[Robin Becker]

On 07/04/2021 19:58, Tim Roberts wrote:
> Shaw, Theodore wrote:
>>
........

>> issue occurs when the memory consumption exceeds about 4GB. I've checked that I'm using a 64 bit version of Python and
>> have been able to create large arrays that consume up to 64GB of memory (my total amount of RAM), so it seems like the
>> issue is specific to reportlab.
>
> No, it's actually an issue in Windows.  The Windows GDI BMP engine cannot handle bitmaps with coordinates larger than 15
> bits (32767).

....
I would expect the error message to be something other than "ValueError: _renderPM.gstate: no memory." if the error
happens in PIL or windows. Perhaps Theodore was delving into the renderPM.py code to search for the error origin.
--
Robin Becker

[Henning von Bargen]

Shaw, Theodore wrote:
> I'm trying to convert an SVG to a BMP that's about 100,000 pixels by
> 100,000 pixels. My code looks like

I'm just curious:
WHY do you try to generate such a huge² image?
I mean, practically every image viewer program would fail with OutOfMemory if a user wants to look at this image.
Why don't you just keep the SVG?

Just to demonstrate the dimension:
Say, a colour laser printer uses 600 dpi (which is good enough for photos).
Then printing your image in this natural dpi resolution would result in a 4.2 m x 4.2 m paper...
I really cannot imagine a use-case...

Henning

[Mark De Wit]

Some modern game engines use MegaTextures to enhance/describe the 3D environment, they are easily this kind of size (first iterations were 32'000 by 32'000, they're now up to 128k in size). The engine would use paging to manage memory consumption though...

But from a reportlab perspective, it must be quite the PDF indeed!

Mark

[Shaw, Theodore]

That's a good question. The BMP is the input filetype for a
microfabrication tool that uses 128,814 dpi. 600 dpi is good enough for
photos but far from good enough for photolithography!

>No, it's actually an issue in Windows.  The Windows GDI BMP engine
cannot handle bitmaps with coordinates larger than 15 bits (32767).

I have managed to generate an image that's 10,143 x 32,840, so I don't
think it's as simple as whether either dimension exceeds 2^15.

It seems like this use case is fairly far outside of reportlab's typical
use case, so I'm gravitating towards using ImageMagick, which comes with
support for caching areas to disk instead of trying to fit everything in
RAM.

Thanks for the help!

>>> This message is from an external sender. Learn more about why this <<
>>> matters at https://links.utexas.edu/rtyclf. <<