squares in heatmaps

[Lo_Pan]

G'day,

I see that the readme notes that someone once noted faint squares
while using pygame, but it still exists, and also happens in PIL. You
can see what I mean here: http://geoip.arpatubes.net/heatmap_squares.png

Fixing the squares in PIL is pretty simple, simply use the gen-dots.py
from 0.0.1 that uses PIL instead of pygame like 0.0.2. Examining the
images 0.0.2's gen-dots.py generates, you can see that the white's
aren't ffffff, they are fdfdfd etc... Running the old gen-dots.py
fixes this.

When using the pygame backend, it seems that during alpha blending
blits the alpha value is increased across all pixels (which is why the
squares, when visible, are coloured). Having a fart around it looks
like the culprit is pygame's surface.blit BLEND_MULTI special flag,
and quite possibly SDLs implementation alpha to alpha blend blits.
Also I think the reason you haven't been able to get surface.set_alpha
working is because surface alphas are ignored when per pixel alphas
are available.

If you comment out the colouring code in pygame_.py, the tiles become
white and dots become grayscale. It is very easy to see the squares
when you do this.

[Lo_Pan]

forgot to mention, using pygame 1.8.1 and 1.9pre, same results.

i might see if there's a newer sdl later.

[Chad Whitacre]

Quick note to say that this is on my radar and I'll look into it when I get a chance.

chad

[Chad Whitacre]

Fixing the squares in PIL is pretty simple, simply use the gen-dots.py

from 0.0.1 that uses PIL instead of pygame like 0.0.2.  Examining the
images 0.0.2's gen-dots.py generates, you can see that the white's
aren't ffffff, they are fdfdfd etc...  Running the old gen-dots.py
fixes this.

FWIW, the old gen-dots.py has a different problem: you can see a certain graininess in the dots. That's what motivated the switch to Pygame.

 

When using the pygame backend, it seems that during alpha blending
blits the alpha value is increased across all pixels (which is why the
squares, when visible, are coloured).  Having a fart around it looks
like the culprit is pygame's surface.blit BLEND_MULTI special flag,
and quite possibly SDLs implementation alpha to alpha blend blits.

 

Is blit called as part of gen-dots.py? That code just resizes the dots down from the largest one. The goal, I think, is to get pure whites in the dots produced by Pygame, no? Perhaps we could do some color correction in there. I haven't noticed to date because I think it only shows up when you have multiple data points at the same exact geographic location.

chad

[Chad Whitacre]

Also, I did confirm that the whites in dots*.png are not pure when using the Pygame backend. I think that is the heart of the problem, no?

chad

[Chad Whitacre]

Also, I did confirm that the whites in dots*.png are not pure when using the Pygame backend. I think that is the heart of the problem, no?

I just checked in new dots and a new dot-generation script that's actually written in JavaScript and uses PhotoShop's scripting features to generate the dots using PhotoShop's resizing algorithm, which preserves the pure whites as we want, unlike Pygame's.

Could you confirm whether this solves the problem?

chad

[Lo_Pan]

> Is blit called as part of gen-dots.py? That code just resizes the dots down
> from the largest one. The goal, I think, is to get pure whites in the dots
> produced by Pygame, no? Perhaps we could do some color correction in there.
> I haven't noticed to date because I think it only shows up when you have
> multiple data points at the same exact geographic location.

Unsure, but I saw that the transform it was using to resize dot 30
used averaging, so I assume it's a related or similar problem.

It probably only occurs when two or more dots intersect, but I haven't
tested. It definitely easily visible when a LOT of dots overlap.

[Lo_Pan]

> I just checked in new dots and a new dot-generation script that's actually
> written in JavaScript and uses PhotoShop's scripting features to generate
> the dots using PhotoShop's resizing algorithm, which preserves the pure
> whites as we want, unlike Pygame's.
>
> Could you confirm whether this solves the problem?

Hi Chad,

The dots produced are quite nice, and work fine in PIL, as expected.
In pygame the problem still exists, as the problem is not related to
the generation of the dots.
As noted previously, it appears to be the alpha blend blit that
changes the alpha value of the entire tile.

You can see a new version of the previously posted map here:
http://geoip.arpatubes.net/heatmap_squares_2009033001.png
Interestingly there appears to be some kind of weird inversion or
miscolouring going on now, as can be seen clearly in some of the
squares here: http://geoip.arpatubes.net/heatmap_squares_2009033002.png

The tiles generated in the above links are not cached from a previous
render, aspen is running on a completely fresh gheat directory.

DZ

[Lo_Pan]

> Interestingly there appears to be some kind of weird inversion or
> miscolouring going on now, as can be seen clearly in some of the
> squares here:http://geoip.arpatubes.net/heatmap_squares_2009033002.png

Actually, that was already happening and isn't a new problem, sorry!

[Lo_Pan]

Here is my points.txt file for you to test with, so you can see the
squares yourself:

http://geoip.arpatubes.net/points.txt

[Chad Whitacre]

[Chad Whitacre]

David,

> Here is my points.txt file for you to test with, so you can see the
> squares yourself:
>
> http://geoip.arpatubes.net/points.txt

Perfect, thanks. I'll take a look.

chad

[Chad Whitacre]

David,

Have you ever used nose for testing? It would be awesome if you were able to write a test case for this bug. Do an image generation and then sample a point to see if the transparency/color is right. Trying to find time here to look at this but thought I'd drop the suggestion in the mean time. :^)

chad

[Chad Whitacre]

Josh,

I believe David Z (Lo_Pan) was working on this. You can find him on-list or possibly in #gheat on freenode.

chad

On Tue, Jul 21, 2009 at 1:52 AM, Josh Gross <endt...@gmail.com> wrote:

Was a solution to the squares problem in pygame ever found? I've
attempted to find a solution and still cannot find one.

- Josh

On Apr 2, 11:45 am, Chad Whitacre <c...@zetaweb.com> wrote:
> David,
> Have you ever used nose for testing? It would be awesome if you were able to
> write a test case for this bug. Do an image generation and then sample a
> point to see if the transparency/color is right. Trying to find time here to
> look at this but thought I'd drop the suggestion in the mean time. :^)
>
> chad
>

[Josh Gross]

Chad,

I did some research (and by research I mean asking on StackOverflow,
in addition to finding the solution myself, simultaneously) and you
can read the solution here:
http://stackoverflow.com/questions/1157385/pygame-and-blitting-white-on-white-gray

In summary, the basic is that pygame has an 'off by one' bug wherein
blitting two colors using BLEND_MULT causes the R, G, and B values to
be one less than expected. Therefore, 255 * 255 = 254. Doing the same
thing over and over causes it to continue to be reduced by 1
(BLEND_MULT the same dot 50 times, you get 205, which is obviously 255
- 50), and producing the ugly gray square.

The solution is either use BLEND_ADD (1, 1, 1) over the final result,
if you don't want to recompile pygame or (as I did in my instance)
manually patch and recompile the surface.h file in pygame's source to
add + 1 to the R, G, and B results of BLEND_MULT.

Either method works fine and completely resolves the issue, allowing
pygame to be preferable heatmap generation choice.

- Josh

On Jul 21, 12:34 pm, Chad Whitacre <c...@zetaweb.com> wrote:
> Josh,
>
> I believe David Z (Lo_Pan) was working on this. You can find him on-list or
> possibly in #gheat on freenode.
>
> chad
>

[Josh Gross]

I should say, rather, you would use BLEND_ADD (1, 1, 1) over the dot's
specific area each time you draw a new dot or dot repetition, *not*
after you finish drawing all the dots.

On Jul 21, 12:41 pm, Josh Gross <endtw...@gmail.com> wrote:
> Chad,
>
> I did some research (and by research I mean asking on StackOverflow,
> in addition to finding the solution myself, simultaneously) and you

> can read the solution here:http://stackoverflow.com/questions/1157385/pygame-and-blitting-white-...

[Chad Whitacre]

That's awesome, good work. :^)

David, did you catch that?

[David Zielezna]

yep, thats exactly what i was expecting.

Chad Whitacre wrote:
> That's awesome, good work. :^)
>
> David, did you catch that?
>
>
>
> chad
>
>
>
>
>
> On Tue, Jul 21, 2009 at 12:43 PM, Josh Gross <endt...@gmail.com
> <mailto:endt...@gmail.com>> wrote:
>
>
> I should say, rather, you would use BLEND_ADD (1, 1, 1) over the dot's
> specific area each time you draw a new dot or dot repetition, *not*
> after you finish drawing all the dots.
>
> On Jul 21, 12:41 pm, Josh Gross <endtw...@gmail.com

> <mailto:c...@zetaweb.com>> wrote:
> >
> > > Josh,
> >
> > > I believe David Z (Lo_Pan) was working on this. You can find
> him on-list or
> > > possibly in #gheat on freenode.
> >
> > > chad
> >
> > > On Tue, Jul 21, 2009 at 1:52 AM, Josh Gross
> <endtw...@gmail.com <mailto:endtw...@gmail.com>> wrote:
> > > > Was a solution to the squares problem in pygame ever found? I've
> > > > attempted to find a solution and still cannot find one.
> >
> > > > - Josh
> >
> > > > On Apr 2, 11:45 am, Chad Whitacre <c...@zetaweb.com