Drawing millions of images on a bigger canvas

[krypticallusion]

Hi, what would be the correct way to composite millions of images on a larger canvas?
I am trying to composite a lot of 32x35 pngs to a larger canvas but the method I am using, draw.Draw() seems to be too hungry for resources choking the GC again and again.

CPUProfile and MEMProfile added below. Thanks!

[Tamás Gulácsi]

A large canvas needs large amount of memory.

32x35x4 byte is about 4.4Kib, so one million is 4272Mib.

Just as your MemProfile says.

Either use some image format that can be written in chunks (AFAIK PNG can do this, but that's quite low-level),

or mmap the region of the pixels, or leave it to tho OS (swapping).

[Howard C. Shaw III]

One more options to add to  Tamás' suggestions - due to Go's interface design, implementing a draw.Image, which is an image.Image with an additional Set(x, y int, c color.Color) method, is sufficient to use draw.Draw for compositing.

As such, you can pre-allocate a flat-file with sufficient space to store 4 x 8 x width x height bytes

(4 for RGBA, 8 for float64 in bytes), and have your implementation of draw.Image reading and writing

to the file by offset. This would minimize memory usage in exchange for a massive slowdown, as

every *pixel* operation would now be a read or write to a file, but only one of the small files needs to be open

at a time. MMapping this file can get some of the performance back, as Tamás suggested, or you can have the file be on a RAMdisk - this alleviates less of the performance issue, as it still requires filesystem operations per-pixel, but is not as slow as dealing with actual spinning media. 

You would still at some point need to pull in the entire large file to write it into a usable format, so whether there is any significant advantage depends on whether your large file has an area equal to or larger than the sum of the areas of the individual images - that is, to what degree the images are being composited with overlap.

 

[Robert Engels]

Many image formats support tiling so you don’t need to have the entire image in memory (since a very large image is usually scaled or only a portion used at a time). 

On Jan 24, 2022, at 10:07 AM, Howard C. Shaw III <howar...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/2710cbd7-ed44-4343-830b-1b661d0b70d7n%40googlegroups.com.