static init gotcha

[Roedy Green]

I think I have discovered a new Java gotcha. I don't recall ever
reading about it. It make perfect sense though it completely baffled
me for hours.

The symptom was after a while, every time a executed a static method,
no matter how trivial, synchronized or not, it would just hang.
IntelliJ trace would just go to sleep.

I was doing threading, and I had some locks internal to two
ConcurrentHashMaps in the class, but they should lock on the
Collection, not the whole class.

I turned on -verbose mode and discovered some system locking code was
being loaded just prior to the freeze.

I think what happens is this. Static init code does a secret lock of
the class object. You thus cannot call any methods of the class on a
separate thread until the static init code is complete. My static
init code was spawning threads.

The rule of thumb is, don't use static init so fire up complicated
code, just init a few fields.
--
Roedy Green Canadian Mind Products
http://mindprod.com
When you were a child, if you did your own experiment
to see if it was better to put to cocoa into your cup first
or the hot milk first, then you likely have the programmer gene..

[Patricia Shanahan]

On 3/30/2012 12:29 PM, Roedy Green wrote:
...

> I think what happens is this. Static init code does a secret lock of
> the class object. You thus cannot call any methods of the class on a
> separate thread until the static init code is complete. My static
> init code was spawning threads.

...

The system for ensuring that a class gets fully initialized by one
thread before any other thread does anything that needs it to have been
initialized is documented in the JLS, in the section on initialization
of classes and interfaces. See
http://docs.oracle.com/javase/specs/jls/se7/html/jls-12.html#jls-12.4.2

That seems a very strange approach to keeping it secret.

Patricia

[Roedy Green]

On Fri, 30 Mar 2012 12:43:30 -0700, Patricia Shanahan <pa...@acm.org>
wrote, quoted or indirectly quoted someone who said :

>That seems a very strange approach to keeping it secret.

"secret" was the wrong word. What I meant is "behind the scenes".
There is no explicit locking code in your program.

[Lew]

It's a well-known antipattern in Java to spawn threads during an initialization phase.

Nutshell arguments:
- Visibility guarantees don't apply because you haven't hit an edge yet.
- Things aren't completely built yet.
- Strange shit inevitably happens.

--
Lew

[Daniel Pitts]

On 3/30/12 12:50 PM, Roedy Green wrote:
> On Fri, 30 Mar 2012 12:43:30 -0700, Patricia Shanahan<pa...@acm.org>
> wrote, quoted or indirectly quoted someone who said :
>
>> That seems a very strange approach to keeping it secret.
>
> "secret" was the wrong word. What I meant is "behind the scenes".
> There is no explicit locking code in your program.

In general, having "static" resources which have any kind of complicated
initialization (IO, threading, etc...) seems to me to be a strong
anti-pattern. If something needs more than simple initialization, it
should be initialized separately from the class initialization.
Dependency injection is your friend. This is often true for simple
classes as well. There are very few use-cases that indicate a static
method call as the best approach IMO. YMMV.

[Patricia Shanahan]

On 3/30/2012 12:50 PM, Roedy Green wrote:
> On Fri, 30 Mar 2012 12:43:30 -0700, Patricia Shanahan<pa...@acm.org>
> wrote, quoted or indirectly quoted someone who said :
>
>> That seems a very strange approach to keeping it secret.
>
> "secret" was the wrong word. What I meant is "behind the scenes".
> There is no explicit locking code in your program.

Perhaps "implicit"?

Incidentally, I do agree with the conclusion that static initializers
should not be over-used.

I do sometimes use them, but only for tasks that are logically static
variable initialization but do not fit in initializer expressions, such
as putting the entries in a constant map. I want the locking behavior -
I don't want any chance of another thread trying to use the map before
it is completely initialized.

Patricia

[markspace]

On 3/30/2012 12:29 PM, Roedy Green wrote:

> I think what happens is this. Static init code does a secret lock of
> the class object. You thus cannot call any methods of the class on a
> separate thread until the static init code is complete. My static
> init code was spawning threads.

Actually, if you look at a running Java program in a debugger, there are
sever threads that you don't spawn. I assume at least one is the "class
initializer." However, it is not spawned, but initialized once and then
kept around. It's a bit like the EDT, or the garbage collection thread.
Those are re-used also, not "spawned."

> The rule of thumb is, don't use static init so fire up complicated
> code, just init a few fields.

I think "so fired up complicated" could use some qualification here.
The point about avoiding multi-threading in a static initializer is
probably good however.

[markspace]

On 3/30/2012 3:35 PM, markspace wrote:
>
> Actually, if you look at a running Java program in a debugger, there are
> sever threads that you don't spawn. I assume at least one is the "class

"serveral"

[Arved Sandstrom]

On 12-03-30 06:57 PM, Patricia Shanahan wrote:
> On 3/30/2012 12:50 PM, Roedy Green wrote:
>> On Fri, 30 Mar 2012 12:43:30 -0700, Patricia Shanahan<pa...@acm.org>
>> wrote, quoted or indirectly quoted someone who said :
>>
>>> That seems a very strange approach to keeping it secret.
>>
>> "secret" was the wrong word. What I meant is "behind the scenes".
>> There is no explicit locking code in your program.
>
> Perhaps "implicit"?
>
> Incidentally, I do agree with the conclusion that static initializers
> should not be over-used.

I'd argue that the logical conclusion is then that static (class)
variables themselves should not be over-used. It's not a distinction
between static initializer blocks and class variable initializers at
all, since all of those are executed consecutively in lexical order.
Whether you use a variable initializer or a static block really comes
down to the nature of the initialization: as you point out in your
example below you sometimes have no choice but to use a static block.

So if there is some objection to be raised I would think it has to do
with whether you use class variables in the first place. Here I think
we'd draw a distinction between implementation-private read-only data
that is common to all objects of a class (and that cannot be construed
as part of the public API), and shared data. With respect to the latter,
the bicycle instance count example used in the Java tutorial is a good
example of when *not* to use class variables. IMHO.

> I do sometimes use them, but only for tasks that are logically static
> variable initialization but do not fit in initializer expressions, such
> as putting the entries in a constant map. I want the locking behavior -
> I don't want any chance of another thread trying to use the map before
> it is completely initialized.
>
> Patricia

Your example is what at least three quarters of all my static
initializers do: initializing read-only maps, for implementation-private
mechanics common to all objects of that class. I'm comfortable with that
because all of the instances actually get the values through a static
method, and conceptually there could be no map at all.

AHS
--
Last week I helped my friend stay put. It's a lot easier'n helpin' 'em
move. I just went over to his house and made sure that he did not start
to load shit into a truck.
-- Mitch Hedberg

[Roedy Green]

On Fri, 30 Mar 2012 14:18:06 -0700, Daniel Pitts
<newsgrou...@virtualinfinity.net> wrote, quoted or indirectly
quoted someone who said :

>Dependency injection

Could you suggest a good place to learn about this?

[Roedy Green]

On Fri, 30 Mar 2012 15:35:45 -0700, markspace <-@.> wrote, quoted or

indirectly quoted someone who said :

>

>> The rule of thumb is, don't use static init so fire up complicated
>> code, just init a few fields.
>
>
>I think "so fired up complicated" could use some qualification here.
>The point about avoiding multi-threading in a static initializer is
>probably good however.

The problem is if code gets complicated you can inadvertently use
structures before your initialisation is complete. Your static init
code might improperly call methods you plan to use once all is built.

In my own case, I was mildly surprised to find the code causing
trouble was being run during a static init. It too far removed from
the initial trigger to be obvious.

[Arved Sandstrom]

On 12-03-31 04:40 AM, Roedy Green wrote:
> On Fri, 30 Mar 2012 14:18:06 -0700, Daniel Pitts
> <newsgrou...@virtualinfinity.net> wrote, quoted or indirectly
> quoted someone who said :
>
>> Dependency injection
>
> Could you suggest a good place to learn about this?

You could do worse than starting with
http://docs.oracle.com/javaee/6/tutorial/doc/giwhb.html.

[markspace]

On 3/31/2012 12:44 AM, Roedy Green wrote:

> The problem is if code gets complicated you can inadvertently use
> structures before your initialisation is complete. Your static init
> code might improperly call methods you plan to use once all is built.

What I'm lacking is insight into how and why this occurred. Did you,
for example, cause the problem yourself somehow?

static {
// let's get busy!
for( i=0; i< 200; i++) new MyThread().start();
}

Or did you instantiate a complex object:

static {
everything = new Everything() // boot the whole system
}

Or something else?

In other words, in some detail, what's the actual anti-pattern here?

[markspace]

On 3/31/2012 12:40 AM, Roedy Green wrote:
> On Fri, 30 Mar 2012 14:18:06 -0700, Daniel Pitts
> <newsgrou...@virtualinfinity.net> wrote, quoted or indirectly
> quoted someone who said :
>
>> Dependency injection
>
> Could you suggest a good place to learn about this?

Any web search for Dependency Injection, or for CDI Dependency
Injection, seems to turn up a lot of good hits. (CDI is a popular
implementation of Dependency Injection for Java.)

In a Java/JEE context (no pun intended), this is a decent introduction:

<http://horstmann.com/corejsf/>

[Roedy Green]

On Sat, 31 Mar 2012 09:28:23 -0700, markspace <-@.> wrote, quoted or

indirectly quoted someone who said :

>

>Or something else?
>
>In other words, in some detail, what's the actual anti-pattern here?

This will take a while.

The basic project is probing bookstores, online electronics stores and
DVD stores to find out which of the books, electronics, and DVDs in
mention on my website they have in stock. I generate links to where
you can find that item on their store, greyed out if I think they do
not have it. See http://mindprod.com/book/books.html to see what the
links look like.

To speed this up I decided to use 30 threads. It turns out that I
then had to slow it down again, because most stores stop listening to
you if you probe them too rapidly. Some stores like O"Reilly and
Barnes and Noble don't mind you hammering them.

The problem was I was doing this via screen scraping the same
interface that humans use. That interface is in constant flux. Trying
to decide whether a book is in or out of stock turns out to require
looking for dozens of strings, each of which has positive or negative
weight. I had to keep tweaking these magic strings whenever my logic
could not decide based on its current set of strings. Sometimes I
would look with my own eyes at the page for 20 minutes, sifting
through the hidden text, the Javascript, ... and still could not
decide, I have been trying to talk stores into putting in/out of stock
icons on their pages for humans to make it clear.

It is all quite crazy. One store you must probe with both a 10 and 13
digit ISBN. They file some books one way and others the other, and
some both.

Amazon advertised a SOAP-XML interface to get this information in a
stable, efficient way. It turned out all their docs are incorrect/out
of date. However, I eventually got it going, (now documented it at
http://mindprod.com/jgloss/amazon.html) only to discover that a probe
even once every 20 seconds is overwhelming to amazon.com though the
other stores are more forgiving.. (How on earth do they survive DOS
attacks?) However, the API works on the European Amazon stores too and
in English, unlike the screen scraping which is customised for each
language/store.

The important thing to understand is how this gradually grew in
compexity. The mistake I made was firing up the whole process of
probing a set of bookstores to refresh in-out-of-stock data with a
static init. This all worked fine until the camel straw when I, in
one of my threads, called a very vanilla method to convert a ISBN to
an ASIN via HashMap lookup, where the method just happened to be in
the same class as the static init fireup. Up until that point that,
by some fluke, never happened. That delay in the penalty for me error
threw me off the scent.

What I do now is have a fireup method that gets called at the start,
and everything that needs to happen at the start of a run, gets a
method inserted there.

In the FORTH days I was able to build both fireup and shutdown chains
as a side effect of compilation, which kept the code more modular.
Today I can build shutdown chains dynamically at class load but not
fireup chains.

[markspace]

On 4/1/2012 6:56 PM, Roedy Green wrote:
> static init. This all worked fine until the camel straw when I, in
> one of my threads, called a very vanilla method to convert a ISBN to
> an ASIN via HashMap lookup, where the method just happened to be in
> the same class as the static init fireup. Up until that point that,

This just sounds like a poor design choice. Initializing a map
statically from disc or worse over a network is just asking for trouble
somewhere. I'm still advising a Properties object (at runtime,
obviously, not a static init block).

> In the FORTH days I was able to build both fireup and shutdown chains
> as a side effect of compilation, which kept the code more modular.
> Today I can build shutdown chains dynamically at class load but not
> fireup chains.

I'd be interested in seeing what you have done to support your app. It
sounds like you are building an application framework. With SAF dead
and NetBeans Platform being fairly complex, most folks still just roll
their own, afaict. A generic, simple app framework might be a useful
thing to a lot of folks.

[Roedy Green]

On Tue, 17 Apr 2012 07:45:28 -0700, markspace <-@.> wrote, quoted or

indirectly quoted someone who said :

>

>I'd be interested in seeing what you have done to support your app. It
>sounds like you are building an application framework. With SAF dead
>and NetBeans Platform being fairly complex, most folks still just roll
>their own, afaict. A generic, simple app framework might be a useful
>thing to a lot of folks.

That's what I though originally, but I don't think anyone but me is
using static generation, much less my framework which is available at
http://mindprod.com/products1.html#HTMLMACROS.

Dynamic hosting is now much more easily had cheaply than when I
started.

My framework has grown too big and complex for someone to use as a
simpler alternative to one of the dynamic frameworks.

The advantage of what I do is caching. My pages can be cached or
easily mirrored. When change code or content there is no delay on the
server for recompilation.Pages are regenerated off line and uploaded.
I don't have any problem with the server code getting bigger and
bigger. Debugging is simpler. I look at the generated pages. I run
them through a validator. There none of the complicated changing code
on the fly complication.

The main thing that does not work well is inserting random quotations
at the foot of each page. They have to be gradually changed and
uploaded, which creates a torrent of somewhat illegitimate uploads.
In put them in iframes and have complex logic to shuffle them, but the
best way would be doing it dynamically.

There is the big advantage of all the code being mine, all understood,
all commented enough for me to understand it, all modifiable. The
parser and macro framework expander are quite simple. All the
cleverness is in the macro application code. I know how I write code,
so I don't really have to study docs. I just ask myself "how would I
have done that?"