ThreadLocal and RangeLookupTable

[Michael Bedward]

Simone, Andrea,

You mentioned that you decided against using a ThreadLocal field in
RangeLookupTable in favour of caching within the ImageOp class. What
were the reasons for that ?

I ask in the context of adding pass-through vs default value vs
exception options to the table class... If there is a strong reason
not to use ThreadLocal within a table object, then I guess these
options would have to be implemented within the image op instead.
However, it seems a lot neater to keep such behaviour in the table and
it may be useful for other applications later.

Michael

[Andrea Aime]

On Thu, Jan 17, 2013 at 11:50 AM, Michael Bedward <michael...@gmail.com> wrote:

Simone, Andrea,

You mentioned that you decided against using a ThreadLocal field in
RangeLookupTable in favour of caching within the ImageOp class. What
were the reasons for that ?

There is no clear way to remove the thread locals, and this is a serious problem

in web applications, it might generate serious leaks if the application gets

un-deployed (mostly because the thread local makes the un-deploy impossible,

as it is part of the container, clobbering the permanent generation of the web container

and resulting in the short or long run into a permgen OOM).

More details: the threads that we are talking about are the tile scheduler

ones, which are out of the control of the image op itself.

Once the op has finished running and it's deallocated we would need to be

able to de-allocate all thread locals to perform the cleanup, unfortunately

the deallocation:

- can only be done withing the thread in which the thread local is allocated

- requires a point in which we are sure, withing that thead, that the operation is done

Long story short, it's not possible to do such deallocation in the JAI context

(or better, we could not find a way to do so).

That's why Simone went for keeping the state as a local variable in the

operation itself, that variable is stack allocated, so private to the thread

running the tile computation, and does not need any particular cleanup

operation, as it gets removed the moment the method call ends.

Cheers

Andrea

 

--

==

Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.

==

Ing. Andrea Aime 

@geowolf

Technical Lead

GeoSolutions S.A.S.

Via Poggio alle Viti 1187

55054  Massarosa (LU)

Italy

phone: +39 0584 962313

fax: +39 0584 1660272

mob: +39  339 8844549

http://www.geo-solutions.it

http://twitter.com/geosolutions_it

-------------------------------------------------------

[Michael Bedward]

Hi Andrea,

Yuk... I see. Certainly sounds painful.

Trying to think laterally... What about if the table has a method to
lookup multiple values, perhaps like:

public List<U> getDestValues( List<T> srcValues )

If that was called per tile row (for example), source value caching
could be done with a local var in the method.

Michael

> --
> You received this message because you are subscribed to the Google Groups
> "jai-tools" group.
> To post to this group, send email to jai-...@googlegroups.com.
> To unsubscribe from this group, send email to
> jai-tools+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/jai-tools?hl=en.

[Andrea Aime]

On Thu, Jan 17, 2013 at 12:32 PM, Michael Bedward <michael...@gmail.com> wrote:

Hi Andrea,

Yuk... I see. Certainly sounds painful.

Trying to think laterally... What about if the table has a method to
lookup multiple values, perhaps like:

public List<U> getDestValues( List<T> srcValues )

If that was called per tile row (for example), source value caching
could be done with a local var in the method.

I see what you mean, could work.

I don't have the code in front of me, so pardon if this was already

discussed/implemented, but another option that might work with minimal 

changes is that the table returns not the value, but both the range and the value, so that

the method can easily check if the next value is still in range, and

go back to the table only when getting outside of the range.

It would be faster against constant value areas and would not require

allocating so many lists.

Cheers

Andrea

 

[Michael Bedward]

On 17 January 2013 22:46, Andrea Aime <andre...@geo-solutions.it> wrote:
> I see what you mean, could work.
>
> I don't have the code in front of me, so pardon if this was already
> discussed/implemented, but another option that might work with minimal
> changes is that the table returns not the value, but both the range and the
> value, so that
> the method can easily check if the next value is still in range, and
> go back to the table only when getting outside of the range.
> It would be faster against constant value areas and would not require
> allocating so many lists.
>

Yep - this is essentially what Simone implemented. So the table
functions as just a collection of range / value tuples (LookupItems)

So, then I start to think do we need the table class at all ? The
alternative would be the client constructing a Collection of
LookupItems and passing this as a parameter to the operator. But I
guess the table (or now, the table Builder class) at least takes care
of handling the case where overlapping lookup ranges are added which
lightens the load for client code.

Thanks for the input - very helpful. I'll commit some new code tomorrow.

Michael

[Andrea Aime]

On Thu, Jan 17, 2013 at 1:58 PM, Michael Bedward <michael...@gmail.com> wrote:

Yep - this is essentially what Simone implemented. So the table
functions as just a collection of range / value tuples (LookupItems)

So, then I start to think do we need the table class at all ? The
alternative would be the client constructing a Collection of
LookupItems and passing this as a parameter to the operator. But I
guess the table (or now, the table Builder class) at least takes care
of handling the case where overlapping lookup ranges are added which
lightens the load for client code.

Right, and it should also try to use a indexed search to speed up finding

the candidate range, e.g., using sorting + binary search (e.g., against

the beginning of the range, assuming the overlapping issues

have already been taken care of).

Again, haven't looked at the code, maybe that's there already too.

 

Thanks for the input - very helpful.  I'll commit some new code tomorrow.

Cheers

Andrea

 

[Simone Giannecchini]

I have the indexed search in my todo (wish?) list.

At a certain point I want to replace some code in the rastersymbolizer and I really need this has we might thousands of lookup items.

--

[Michael Bedward]

On 18 January 2013 00:52, Simone Giannecchini

<simone.gi...@geo-solutions.it> wrote:
>
> At a certain point I want to replace some code in the rastersymbolizer and I
> really need this has we might thousands of lookup items.

Some progress on this...

RangeLookupTable now maintains a sorted list of LookupItems (ordered
by source range) and uses binary search to match input source values.
Seems to be working but needs testing with some real-world data.

Using the table builder is not as neat as I was planning. I had hoped
clients could do:

RangeLookupTable.Builder<Double, Integer> builder = RangeLookupTable.builder()

But javac type inference doesn't seem able to handle the complex type
parameter bounds, so we're stuck with:

RangeLookupTable.Builder<Double, Integer> builder = new
RangeLookupTable.Builder<Double, Integer>();

(I've been using Scala a lot lately and it's spoiled me).

Wondering if there is any need to store default value and pass-through
option in the lookup table now that we just use it to retrieve
LookupItems ? I'd vote to just have these as parameters for the image
op instead.

It's been 46C in Sydney today and still warm no so I'm not doing a lot
more this evening :)

Michael