Possible incompt between CF9+10 & Railo

[Adam Cameron]

G'day all:

Hey, I've got this code:

<cfscript>
one = "tahi";
two = "rua";
three = "toru";

st1= {one=one, two="rua", three=three};
st2 = structNew();
st2.three = st1.three;
st2.two = two;
st2.one = "tahi";

writeOutput("#st1.hashcode()#<br />");
writeOutput("#st2.hashcode()#<br />");
</cfscript>

On CF (both versions I tested), I get this:
87017274
87017274

And it is always that answer.

On Railo 4.0.2, I get this sort of thing:
1652000677
952790250

IE: different numbers for both, and they change very time.

I'm guessing that Railo is returning a hashcode of the reference, rather than the value? But, yeah, just a guess.

I dunno what the correct behaviour should be, but the CF behaviour shure is more convenient (for what I'm doing).

Thoughts?

Cheers.

--
Adam

[Chris Blackwell]

Yet in Railo

st1.equals(st2) --> true

It looks like equals() has been overridden to provide value based equality, but hascode() is inherited from java.lang.Object which uses the objects memory address to calculate the hash

in the past when i've needed a cross platform test for equality i've used

arraycontains([st1], st2)

Chris

[Adam Tuttle]

I guess I'll chime in because I was Adam's inspiration and I happen to be paying attention to the thread right now.

Chris, your solution works... in the exact case that Adam outlined previously. However, I'm trying to use .hashCode() as a general solution value for HTTP Etags data. The object in question could be an array, but it could just as easily be a query, string, boolean, number, or structure (am I forgetting anything?)...

The nice thing about hashCode (at least on ACF) is that it's available for all data types, has the exact same API (e.g. just call obj.hashCode()) and it's consistent across requests. On the other hand, Railo's results change every time you refresh the code sample Adam provided.

Since the value changes on every request even though the data does not, it's not useful for Etags.

Think of it like this:

hash(serializeJson(someObj))

That should return the same string every time, right? (depending on serializeJson to always list structure keys in the same order).

Well .hashCode() should do the same thing: provide a unique(ish?) value based on the data it contains; but without having to resort to the slowness of serialization and then the slowness of MD5 hashing. And it should be the same value as long as the data doesn't change.

-- (the other) Adam

[Igal @ getRailo.org]

when you call st1.hashCode() then the CFML engine first attempts to resolve that call to a member function with the name hashCode(), and when that fails, it is deferred to Java via reflection.

the hashCode() method is intrinsic to Java and like equals() and toString() there is a base implementation in the java.lang.Object class which can be overridden in inheriting classes.  http://docs.oracle.com/javase/7/docs/api/java/lang/Object.html#hashCode()

when it makes sense -- hashCode() should be overridden.  when it doesn't -- it shouldn't ;)

either way, Railo strives to maintain ACF compatibility in CFML matters, but definitely not on implementation and Java matters, so IMO this is not an issue.

I am not sure what it is that "for what [you're] doing" that makes it an important feature.  but keep in mind that if your code relies on undocumented implementation it might break some day in the future without notice since it is not part of the "contract" that is defined by the public API.

also keep in mind that deferring to Java methods from CFML, can result in a slower process than using the BIFs due to the use of reflection and runtime evaluation (as opposed to compile time) -- at least currently -- there are plans to improve on that soon.


Igal

[Igal @ getRailo.org]

another thing to keep in mind is that hashCode() is used in gazillions of places in the Java code that makes Railo run -- far beyond your use in CFML. 

so ad-hoc implementations (where they are not needed) can degrade performance and have other unintended consequences.


Igal

[Bruce Kirkpatrick]

ACF doesn't have documentation for hashcode usage, so I don't see how Railo needs to obey any specific behavior here.

I didn't know arraycontains existed yet.  That's neat.   However, I don't think I've ever needed to compare complex objects for equality.  There might be some inefficient code in place if that is what you're doing.   When I have a very long string, I just use hash with md5 or sha1 to save space in the database mostly.  For example, before I import real estate listings into the 300 column table, I run a hash on the entire row, and compare it with a hash stored in the database for that listing id.   This allows me to avoid the complex work of converting the row data and saving it to several tables.

[Bruce Kirkpatrick]

I agree about the use of Java in CFML.  You need to run a single java method that does huge amount of work and returns a simple value, rather then run a bunch of small java methods for best performance.   I tested the difference to be something ridiculous like 100 or 1000 times faster without the reflection overhead.

[Adam Tuttle]

>You need to run a single java method that does huge amount of work and returns a simple value, rather then run a bunch of small java methods for best performance

This is exactly what I'm trying to do.

Consider a large CF array that resembles: x = [{key1:val1,key2:val2,...,key10:val10},{...},............]

What I'm doing is x.hashCode() to get a unique ("hash") of that object. Except the value is different every request for Railo, so it's not useful in this way. Contrast that with hash(serializeJson(x)), which will be the same across every request.

Adam

[Igal @ getRailo.org]

as I wrote before -- the hashCode() method is used in many different places in the java code.

so now imagine that for the sake of consistency we will "serialize" the Struct or Array each time the hashCode() method is called -- that will cause a major performance degradation because it will affect all of the calls to hashCode(), and that would be for a feature that in 99.8% of the case is not needed. 

for the 0.2% of times when it is needed you have the solution that you already cited ;)


Igal

[Bruce Kirkpatrick]

I don't understand why you'd want an ETag calculation done on an object when ETags are used for HTTP caching purposes and the actual data being output will always be a string, won't it?

Here is an alternative HTTP caching approach that doesn't require ETag:

Change this:

http://yourdomain/style.css

to this:

http://yourdomain/style.css?version=1-123

I built a file tracking system into my CFML app that indexes the filesystem for each web site in onApplicationStart (specific domain) and onServerStart (shared global files). You could use hash for this, but I just use file + date since that is compatible for my app.    If your output is dynamically generated, then you'd still need to be versioning it somehow.

This allows me to output the max-age:3600 header, so that content doesn't expire for an hour for some of the file types.   

When you append a unique query string, the browser is forced to re-download the file even if it is identical.  Won't an ETag approach still require a smaller packet to verify it is identical with the server?   According to the wiki, yes: http://en.wikipedia.org/wiki/HTTP_ETag

Using Expires+max-age actually eliminates a large numbers of round-trip requests which will be noticeable anytime there is a slower internet connection or mobile device. It's also possible to add rewrite rules that eliminate the version number from being in the query string (an optimization that only helps certain proxy servers).  I did this too, but I decided not to use since I want developers to understand where the file is more easily.  A url like /style.1-1231.css is a bit strange.

Expires + max-age headers are perhaps the most efficient way to handle http caching.

Almost note that my version string is a combination of a file ID and a file version ID from the database and these values auto increment.  I also have a scheduled task that deletes old versions periodically to avoid waste.

In addition to this, I built concat/minification and css sprite map automation into the same code in my app so that the minified file is also versioned.  The sprite map was done by writing my own CSS parser, which also inserts query strings for the images in the CSS, so that they are versioned as well.   Eventually, I may find the time to implement this across every feature that outputs to an embedded url so the entire site could be cached.   At the moment, I find a 1 hour expiration an acceptable delay for the features which aren't versioned yet.   Most web developers have to depend on CTRL + R or clearing cache, but you can eliminate that if you automate versioning.    I've noticed that google even shows a message with ajax now on its apps when they release a new version.  My system could later allow such a notification to appear which encourages people to run the new version.   

Check out my company site to see all this in action.  My site does virtually everything in terms of pagespeed recommendations, minify/concat, css sprite maps,  ajax partial page transitions with html 5 push state, responsive web design,  adjustable font sizes, and maximum caching performance with Railo, yet all requests are dynamic.  I also patched nginx with the SPDY protocol and use SSL so that the requests are multi-plexed through a single connection.   My site should seem faster then google because of all this.  SPDY would be very noticeable if your page has hundreds of small images, etc and it works in current versions of Firefox and Chrome for SSL sites only.  There is a mod_spdy for apache as well, which I previously was running before migrating to nginx.

https://www.farbeyondcode.com/

I plan on making my source code for all this open source once I clean it up some more.

[Adam Cameron]

Just to try to circumvent some of this "Railo's doing it right" discussion, and move on to getting this fixed, consider this code:

<cfscript>
one = "tahi";
two = "rua";
three = "toru";

st1= {one=one, two="rua", three=three};
st2 = structNew();
st2.three = st1.three;
st2.two = two;
st2.one = "tahi";

writeOutput("st1.equals(st2): #st1.equals(st2)#<br />");
writeOutput("st2.equals(st1): #st2.equals(st1)#<br />");
writeOutput("st1.hashCode(): #st1.hashcode()#<br />");
writeOutput("st2.hashCode(): #st2.hashcode()#<br />");
</cfscript>

This outputs:
st1.equals(st2): true
st2.equals(st1): true
st1.hashCode(): 18796902
st2.hashCode(): 4565548

OK, so according to this, the two objects do equal each other, however the hashCodes are different. This is illegal according to how hashCode() & equals() are supposed to be implemented?

So whether it's a CF compat issue or not (the compat issue would be whether the two systems come up with the same hashCode, I guess. But that's unimportant, I agree), it's just bung.

Adam (the other one) is trying to use hashCode() in exactly the way it was intended, and it ain't working right.

Ref: http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/Object.html#hashCode%28%29
Extract: "If two objects are equal according to the equals(Object) method, then calling the hashCode method on each of the two objects must produce the same integer result."

Micha, would you like a bug raised for this? For the purposes of the fix, despite the spec saying "This integer need not remain consistent from one execution of an application to another execution of the same application" it would be handy if it did stay the same, so if everything else is equal (like there's no performance degradation, or the code would just be unfeasibly complex to write), would it be possible for the hashCode() method to... err... "coincidentally" return the same result for the "same" struct each time?  CF manages to achieve this (even across CF versions), but who knows what shenanigans they get up to to effect it.

Cheers.

--
Adam

[Adam Cameron]

Oops I accidentally worded this as a question "This is illegal according to how hashCode() & equals() are supposed to be implemented?" It's a statement, not a question.

--
Adam

[Michael Offner]

First of all, it is impossible to be 100% ACF compatible on Java level, only ACF can be 100% compatible on Java level.

we try to be compatible when it comes to popular functionality on this level.

what's about the Object.hashCode method, compatibility here is not my highest concern, but compatiblity to the java defintion of this method is

http://docs.oracle.com/javase/1.4.2/docs/api/java/lang/Object.html#hashCode()

in the defintion of this method (see link above) is clearly written the following:

If two objects are equal according to the equals(Object) method, then calling the hashCode method on each of the two objects must produce the same integer result.

i have this also tested with the class HashMap, to see how a java implementation handle this situation

<cfscript>
hm1=createObject('java','java.util.HashMap').init();
hm1.one=one;
hm1.two="rua";
hm1.three="three";

hm2 = createObject('java','java.util.HashMap').init();
hm2.three = hm1.three;
hm2.two = two;
hm2.one = "tahi";

writeOutput("hm1.equals(hm2): #hm1.equals(hm2)#<br />");
writeOutput("hm2.equals(hm1): #hm2.equals(hm1)#<br />");
writeOutput("hm1.hashCode(): #hm1.hashcode()#<br />");
writeOutput("hm2.hashCode(): #hm2.hashcode()#<br />");
</cfscript>

in that case i get the result you get with ACF structs, so this is in fact a incompatibility to the Java interface definition and we have to fix this.

can you please open a ticket for this, we will check all railo types (array,struct,query ...) for that.  

Micha

2013/1/18 Adam Cameron <adamcamero...@gmail.com>

<cfscript>
one = "tahi";
two = "rua";
three = "toru";

st1= {one=one, two="rua", three=three};
st2 = structNew();
st2.three = st1.three;
st2.two = two;
st2.one = "tahi";

writeOutput("st1.equals(st2): #st1.equals(st2)#<br />");
writeOutput("st2.equals(st1): #st2.equals(st1)#<br />");
writeOutput("st1.hashCode(): #st1.hashcode()#<br />");
writeOutput("st2.hashCode(): #st2.hashcode()#<br />");
</cfscript>

This outputs:
st1.equals(st2): true
st2.equals(st1): true
st1.hashCode(): 18796902
st2.hashCode(): 4565548

--

/micha

Michael Offner CTO Railo Technologies GmbH

[Michael Offner]

if have done a ticket

https://issues.jboss.org/browse/RAILO-2255

and solved the problem

Micha

2013/1/18 Michael Offner <mic...@getrailo.com>

[Adam Cameron]

Hi Micha

First of all, it is impossible to be 100% ACF compatible on Java level, only ACF can be 100% compatible on Java level.

we try to be compatible when it comes to popular functionality on this level.

Absolutely, yes. I agree with the others that my wording was poor in that this wasn't really an incompat between CF & Railo in hindsight. I had not looked closely at the hashCode() docs when I raised the question, so was really more seeking clarification of what the expectations should be, at that point.

> in that case i get the result you get with ACF structs, so this is in fact a incompatibility to the Java interface definition and we have to fix this.

> can you please open a ticket for this, we will check all railo types (array,struct,query ...) for that.  

Done: https://issues.jboss.org/browse/RAILO-2256.

Thanks mate!

--
Adam

[Adam Cameron]

Damn. You got in just before me!

You can close that other one then :-)

--
Adam

[Adam Tuttle]

I'm glad that you can think of ways that what I'm asking is irrelevant for what you want to do, but the fact remains that ETags would be useful for me. I'm sorry you disagree, but your disagreement doesn't make my use case irrelevant.

I am a framework author -- a REST framework. People write API's with my framework, and ETags are useful to APIs. (Sure, there are other caching mechanism to explore, but ETags are perfectly valid). Since the data that the API returns could potentially change very frequently, or possibly not at all, ETags are a good balance. Semi-frequent requests with little or no payload are preferable to long-outdated data in the client. But since the data is coming out of a database, we can't rely on apache/etc to handle etags and the appropriate responses when a request with an if-none-match header is made. Clearly it's a job for the framework.

Thanks, Michael, for validating the issue and addressing it. I just wish that some community members weren't so quick to dismiss a request because they, themselves, wouldn't find it useful.

Adam

[Adam Cameron]

G'day:
In addition to what Adam (the other one) said, I can't help but thinking that his approach is kinda the "industry standard" approach to this sort of thing, whereas you seem to be reinventing the wheel a bit, Bruce? I have to say I did not read your comment as thoroughly as it perhaps warranted, but the impression I got is that you had "invented" an approach that is already covered via using etags..?

Either way, what Adam is doing is completely legit, and indeed I think it's a pretty slick thing to do in a REST situation.

And it's always important to remember - as Adam points out - the fact that you wouldn't do something a given way does not mean that way is somehow invalid, and it's kinda irrelevant to the issue on this thread anyhow (not that I necessarily see an issue with threads straying off the initial topic they were raised as... it's all interesting stuff!).

--
Adam

[Bruce Kirkpatrick]

Integrating 2 systems where you don't have the ability to store the information for when it changed sounds like you do need something like ETags.  I saw facebook graph api uses ETags for this.   

A specific user could allow the API to have a field that lets you search for new data perhaps, but this ETAG feature does make your framework look more complete.   It seems like the ETag could make it easier to know whether specific responses are the same without having to modify the rest of your system to support it.    I'm still confused on why an object would be hashed instead of the string.

[Adam Tuttle]

I could very easily do hash(serializeJson(data)), but that's a lot of processing at the CFML layer that would be made easier and orders of magnitude faster (and functionally equivalent as far as etags are concerned) by doing data.hashCode(). That's all, and I said as much in my first post in this thread.

I'm not "storing" the hash anywhere. I simply return it as part of the api response (etag header). Then in subsequent requests it can be checked against the value sent by the client to see if the data that the client already has cached. It's an entirely transient data point.

I see your point about letting developers do their own thing with getting "new" data. I myself have written api's that were inspired by the twitter api's "since_id" option that only returns tweets newer than the latest one you have on file. But again, this is different. Say you're writing a conference-schedule app, and the details of a session change. It's not a new record, it's an updated record. That doesn't really fit the "since_id" model.

Adam

[Igal]

IMHO this is not a bug in Struct and StructImpl should not be modified.

please see my comments in the JIRA ticket for details:

https://issues.jboss.org/browse/RAILO-2255?focusedCommentId=12748999#comment-12748999

Igal

[Chris Blackwell]

Igal,

StructImpl does override equals()

StructImpl extends StructSupport, which overrides equals using CollectionUtil.equals()

https://github.com/getrailo/railo/blob/master/railo-java/railo-core/src/railo/runtime/type/util/StructSupport.java#L211

>>

https://github.com/getrailo/railo/blob/master/railo-java/railo-core/src/railo/runtime/type/util/CollectionUtil.java#L16

So if its both or none, hashCode() should be overridden, If hashCode is overridden to provide a consistent  value based on the value of the struct then the equals() implementation can be reduced to st1.hashCode() == st2.hashCode()

Chris

[Igal @ getRailo.org]

Chris,

StructImpl DOES NOT override equals()
https://github.com/getrailo/railo/blob/4.0/railo-java/railo-core/src/railo/runtime/type/StructImpl.java

if StructSupport overrides equal() then StructSupport should also override hashCode() and it doesn't so that is exactly where the "fix" should go.


Igal

[Bruce Kirkpatrick]

You guys are both right you know.  Igal is referring to the development branch which was changed 7 hours ago,  but the master branch still overrides equals.   You'll have to wait for the next official release or build the dev 4.0 version of railo now.

[Chris Blackwell]

splitting hairs..

[Igal @ getRailo.org]

Not at all.

as a matter of fact -- now this can definitely cause problems -- because StructImpl has a specific implementation of hashCode() that is not based on different "content" from the equals() implementation in StructSupport.


https://github.com/getrailo/railo/blob/4.0/railo-java/railo-core/src/railo/runtime/type/StructImpl.java

	@Override
	public int hashCode() {
		return map<
span class="o" style="margin: 0px; padding: 0px; border: 0px; font-weight: bold;">.hashCode();
	}

https://github.com/getrailo/railo/blob/4.0/railo-java/railo-core/src/railo/runtime/type/util/StructSupport.java

	
@Override
	public boolean equals(Object obj){
		if(!(obj instanceof Collection)) return false;
		return CollectionUtil.equals(this,(Collection)obj);
	}

 @Override
 public int hashCode() {
	return CollectionUtil.hashCode(this);
 } 

Igal

[Igal @ getRailo.org]

there was a typo there.  the word "not" should not be there.  it should read:  "now this can definitely cause problems -- because StructImpl has a specific implementation of hashCode() that is  based on different "content" from the equals() implementation in StructSupport"

also, it's better to split hairs now than scratch our heads later trying to figure out weird, elusive, bugs.


Igal

On 1/18/2013 12:39 PM, Igal @ getRailo.org wrote:

Not at all.

as a matter of fact -- now this can definitely cause problems -- because StructImpl has a specific implementation of hashCode() that is not based on different "content" from the equals() implementation in StructSupport.


https://github.com/getrailo/railo/blob/4.0/railo-java/railo-core/src/railo/runtime/type/StructImpl.java

	@Override
	public int hashCode() {
		return map&
lt;
span class="o" style="margin: 0px; padding: 0px; border: 0px; font-weight: bold;">.hashCode();
	}

https://github.com/getrailo/railo/blob/4.0/railo-java/railo-core/src/railo/runtime/type/util/StructSupport.java

	
@Override
	public boolean equals(Object obj){

		<
span c="" lass="k" style="margin: 0px; padding: 0px; border: 0px; font-weight: bold;">if(!(obj instanceof Collection)) return false;
		return CollectionUtil.equals(this,(Collection)obj);
	}

[Denny]

On 1/18/13 6:53 AM, Adam Cameron wrote:
> G'day:
> In addition to what Adam (the other one) said, I can't help but thinking
> that his approach is kinda the "industry standard" approach to this sort of
> thing, whereas you seem to be reinventing the wheel a bit, Bruce? I have to
> say I did not read your comment as thoroughly as it perhaps warranted, but
> the impression I got is that you had "invented" an approach that is already
> covered via using etags..?

ETags are more for versioning than caching, per se (though not
unrelated). Commonly used to prevent a user from overwriting another
user's modifications, or a time-to-live-less cache (better than none,
but TTL is a huge part of caching, as even minimal IO can have serious
overhead).

Bruce outlined a common way to handle implementing cache control. The
important part isn't the implementation, it's the use-- there can be
pretty decent performance gains, which are especially nice w/mobile, as
he pointed out (several cache layers can take advantage).

The fastest request is the one not requested. ;]

According to this (which is quite nice), "all of the above" is the
correct approach, and the logic seems sound to me:

http://my.safaribooksonline.com/book/-/9781449317904/4dot-metadata-design/id2829455

:Denny

--
Railo Technologies: getrailo.com Professional Open Source
Skype: valliantster (505)510.1336 de...@getrailo.com
GnuPG-FP: DDEB 16E1 EF43 DCFD 0AEE 5CD0 964B B7B0 1C22 CB62

[Bruce Kirkpatrick]

BTW: I noticed wordpress was using the query string approach which was why I favored that approach for my app.

It's the fact he is making a framework that he can't make a specific decision for other people on how they want to do things.   Their might be specific APIs that benefit more from using max-age and expires instead of ETag.   I definitely favor putting more of the validation and processing work on the third party system or on the client (browser) so that the service itself is screaming fast.

[Bruce Kirkpatrick]

In my reading today, I noticed several articles mention that many hashing systems are machine specific or have collision problems.  Since ETag relies on hashing, I checked to see if Java's hash code is able to work between machines for this purpose and perhaps it can't according to this URL:  http://martin.kleppmann.com/2012/06/18/java-hashcode-unsafe-for-distributed-systems.html

What does this mean for your framework?  Perhaps, you should mention in your REST framework documentation that ETag is not able to be unique across multiple machines (when relying on java hashcode) or you could consider an alternative hash algorithm, which may be more expensive to calculate.  Perhaps, your framework could have an option to configure the hash algorithm.

The article suggests that serialization of the data into a bytestream and then running something like md5 hash on it is the correct solution for a distributed system, but also warns that it may be too slow depending on the volume of requests.  This might require the api to pre-process more of the data to make it efficient.  hashcode is fine for a single system running the API.

The java docs kind of state that it wouldn't work with multiple java processes as well since there is no guarantee of the same result "This integer need not remain consistent from one execution of an application to another execution of the same application."

[Adam Tuttle]

Before I jump to optimize for clustering, I'm going to wait and see what the Railo implementation looks like. I think that ACF is consistent cross-machine, even cross CF version. If Railo ends up the same, then it would all be for nothing.

Adam

[Denny]

On 1/18/13 5:00 PM, Bruce Kirkpatrick wrote:
> In my reading today, I noticed several articles mention that many hashing
> systems are machine specific or have collision problems. Since ETag relies
> on hashing, I checked to see if Java's hash code is able to work between
> machines for this purpose and perhaps it can't according to this URL:
> http://martin.kleppmann.com/2012/06/18/java-hashcode-unsafe-for-distributed-systems.html

Good point!

Apache httpd in a clustered environment has the same problem, because
it's based off the inode or some such.

...

> Perhaps, your framework could have an option to configure the hash
> algorithm.

I like the sound of that! HA-safe mode, yo! =)


And again, I dig your posts, Bruce. While in general I favour a more
abstracted approach, I'll undoubtedly be leveraging some of the
information you've shared for my raspberry pi application (the code will
probably be generated for me though. Yay abstractions! ;)).

[Denny]

On 1/18/13 7:20 PM, Adam Tuttle wrote:
> Before I jump to optimize for clustering, I'm going to wait and see what
> the Railo implementation looks like. I think that ACF is consistent
> cross-machine, even cross CF version. If Railo ends up the same, then it
> would all be for nothing.

Perhaps:

http://stackoverflow.com/questions/1516843/java-object-hashcode-result-constant-across-all-jvms-systems

But it seems a bit like programming by accident, or whatever the term
they used in "The Pragmatic Programmer" was. (a must read IMHO!)