cf incorrectly thinks compiled java byte code is correct

[hofar...@houseoffusion.com]

Hey guys,

I am wondering if many others have run into this issue. Basically, when we deploy certain "high volume" .cfm files (i.e. index.cfm), they are sometimes incorrectly compiled to java byte code.

What I believe is happening is when several requests come in while ColdFusion is trying to compile the .cfm file to java byte code, the byte code gets created, but the process bombs out before it is able to complete successfully.

Since CF in fact DID generate the byte code (albeit not correctly), on subsequent requests, CF "thinks" it already compiled the byte code successfully so it doesn't try to compile it on future requests. What we end up with is a .cfm file on the server with all the source code in place, and some incorrect byte code somewhere.

In order to fix the problem, we need to deploy the file again (sometimes one or two more times) in order to "get lucky" in that when the high traffic file is hit, another request doesn't come in just then and interrupt the compilation process.

Has anyone else come across this problem?

Here is a related article: http://www.neiland.net/blog/article/compile-cfml-to-java-bytecode-plus-fusebox-fix/



~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion
Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:359311

[hofar...@houseoffusion.com]

> I am wondering if many others have run into this issue. Basically, when we deploy certain "high volume" .cfm
> files (i.e. index.cfm), they are sometimes incorrectly compiled to java byte code.

How do you know this? What do you mean exactly by "incorrectly
compiled"? What version of CF are you running? What are the exact
error messages you're seeing?

> What I believe is happening is when several requests come in while ColdFusion is trying to compile the .cfm file
> to java byte code, the byte code gets created, but the process bombs out before it is able to complete successfully.
>
> Since CF in fact DID generate the byte code (albeit not correctly), on subsequent requests, CF "thinks" it already
> compiled the byte code successfully so it doesn't try to compile it on future requests. What we end up with is a .cfm
> file on the server with all the source code in place, and some incorrect byte code somewhere.

I've never seen this problem, even in many high-volume environments -
some of which had their own problems. And it doesn't really make
sense, given my (admittedly limited) knowledge of CF internals. So, I
kind of think this is unlikely. Something else is probably happening.

> In order to fix the problem, we need to deploy the file again (sometimes one or two more times) in order to "get lucky"
> in that when the high traffic file is hit, another request doesn't come in just then and interrupt the compilation process.
>
> Has anyone else come across this problem?
>
> Here is a related article: http://www.neiland.net/blog/article/compile-cfml-to-java-bytecode-plus-fusebox-fix/

That article isn't really related, although it could solve the problem
you describe. You can precompile CF code, then deploy that instead of
the source code. You could try doing this to solve your problem, but
again I don't think that's really the problem you're having.

Dave Watts, CTO, Fig Leaf Software
1-202-527-9569
http://www.figleaf.com/
http://training.figleaf.com/

Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on
GSA Schedule, and provides the highest caliber vendor-authorized
instruction at our training centers, online, or onsite.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:359312

[hofar...@houseoffusion.com]

do you have cf set set to save compiled files to disk ? this is the default
so probably yes.
if so, then try disabling this and deleting all those compiled class files
from the server.
The only benefit having this option enabled gives you is that when CF is
restarted, it loads all those files into memory again so that they do not
need to be compiled the first time the page is executed. This is only
really helpful on BIG apps which take a long time to load/initialise, for
everything else, the compile overhead is minimal and barely noticeable.

It also causes cf to take longer to restart, a lot longer if you have a lot
of files and can cause timeouts. Unless you have a specific reason to
enable this, you are better off with it disabled.

Also in your cfadmin do you have trusted cache enabled ?

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:359317

[hofar...@houseoffusion.com]

> How do you know this? What do you mean exactly by "incorrectly compiled"? What version of CF are you running? What are the exact error messages you're seeing?

The reason I know it's not being compiled correctly is because when this issue occurs, the errors that happen are indicative of the .cfm file being in place, but the byte code being empty.

What I mean is, let's say page.cfm does a cfinclude on a high volume file called set_first_name.cfm which simply sets a variable called "firstName".

If that file is "corrupted" in the way I am describing, a "firstName variable is undefined" error would occur when firstName was referenced back in page.cfm (after the cfinclude). Because even though the included set_first_name.cfm file does indeed exist, the byte code that was generated is behaving as if it is "blank" (for lack of a better term).

I haven't personally examined the byte code, but 1) I don't have access to the live server, and 2) I wouldn't really know how to read it.

> I've never seen this problem, even in many high-volume environments - some of which had their own problems. And it doesn't really make sense, given my (admittedly limited) knowledge of CF internals. So, I kind of think this is unlikely. Something else is probably happening.

You may be right, but this is what we're experiencing. I also admit a limited knowledge of CF internals.

> That article isn't really related, although it could solve the
> problem you describe. You can precompile CF code, then deploy that instead of
> the source code. You could try doing this to solve your problem, but
> again I don't think that's really the problem you're having.

I agree that the article is only somewhat related, but it is related in the fact that, yes, there seems to be some sort of problem with the compilation process. We have thought about deploying the compiled code, but wanted to see if anyone else had experienced this problem first.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:359318

[hofar...@houseoffusion.com]

> > How do you know this? What do you mean exactly by "incorrectly compiled"? What version of CF are you running?
> > What are the exact error messages you're seeing?
>
> The reason I know it's not being compiled correctly is because when this issue occurs, the errors that happen are
> indicative of the .cfm file being in place, but the byte code being empty.
>
> What I mean is, let's say page.cfm does a cfinclude on a high volume file called set_first_name.cfm which simply
> sets a variable called "firstName".
>
> If that file is "corrupted" in the way I am describing, a "firstName variable is undefined" error would occur when
> firstName was referenced back in page.cfm (after the cfinclude). Because even though the included
> set_first_name.cfm file does indeed exist, the byte code that was generated is behaving as if it is "blank" (for lack
> of a better term).
>
> I haven't personally examined the byte code, but 1) I don't have access to the live server, and 2) I wouldn't really
> know how to read it.

Well, my take on this is that you wouldn't see this problem if the
file is corrupted. Instead, the entire page would simply fail to
render unless you had the CFINCLUDE in a CFTRY/CFCATCH. The generated
byte code has to be a fully-formed Java class that conforms to the
Java Servlet API, and it seems to me to be unlikely that the class
would be created correctly but just without your custom code. Again, I
don't know enough about the internals of CF to guarantee this, but I
do know a little bit about compilers and Java, and it seems unlikely.

It would be useful if you could simply compare the bytecode size
between "good" and "bad" versions of the same compiled class, if you
could access those from the live server. They may be written to disk,
in the cfclasses directory, if you have "Save cache files to disk"
enabled in the CF Administrator.

> > I've never seen this problem, even in many high-volume environments - some of which had their own problems.
> > And it doesn't really make sense, given my (admittedly limited) knowledge of CF internals. So, I kind of think this
> > is unlikely. Something else is probably happening.
>
> You may be right, but this is what we're experiencing. I also admit a limited knowledge of CF internals.

I guess what I'm getting at here is: you definitely have a problem.
But I don't think your description of the cause of the problem is
accurate at all, and you could easily go down a rabbit-hole here by
focusing on what you imagine is the cause of the problem, rather than
the problem itself. The basic description of the problem is that
you're not getting the output under load that you get without load,
right?

What happens if you block incoming traffic to the server, stop and
restart the CF service, and rerun the same pages, with "Save cache
files to disk" enabled? My guess is that you'll get the correct
output.

> > That article isn't really related, although it could solve the problem you describe. You can precompile CF code,
> > then deploy that instead of the source code. You could try doing this to solve your problem, but again I don't
> > think that's really the problem you're having.
>
> I agree that the article is only somewhat related, but it is related in the fact that, yes, there seems to be some sort of
> problem with the compilation process. We have thought about deploying the compiled code, but wanted to see if anyone
> else had experienced this problem first.

You could certainly try to just deploy compiled classes and see what
happens. It's generally a good thing to do in production anyway.

Another approach you could try is to force the server to precompile
everything by running a test script that generates an HTTP request for
each CF file before you open access to users.

Dave Watts, CTO, Fig Leaf Software
1-202-527-9569
http://www.figleaf.com/
http://training.figleaf.com/

Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on
GSA Schedule, and provides the highest caliber vendor-authorized
instruction at our training centers, online, or onsite.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:359320

[hofar...@houseoffusion.com]

Can you post the code for the set_first_name.cfm?

"firstName variable is undefined".

I have seen this type of error before when the variable is set to null.
Something like a web service call that fails or really does not return
anything. In this instance "setName" might be a function that returns
'void', so firstName would be null.

<cfinvoke method="setName" webservice="...?wsdl" returnvariable="firstname">
.....
</cfinvoke>

You could test this by checking isNull(firstname).

I would think if the byte code was incorrect you would not be getting CF
error details about what is going on and probably get some sort of java
servlet 503 error instead.

Byron

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:359324

[hofar...@houseoffusion.com]

>Can you post the code for the set_first_name.cfm?

Thanks for chiming in Byron. set_first_name.cfm is not a real file, but if it were, it would just look like this:

<cfset firstName = "Brian">

And it would work in all cases, except for the scenario that I'm describing (where the byte code is "incorrectly generated").

In that case, the <cfinclude> for set_first_name.cfm would not fail, but the firstName variable would never get set to "Brian", so any code later in the request lifecycle that expected the value to be "Brian" would not execute as expected.

In other words, in the scenario I'm describing, none of the cfml code in the included file would ever be executed, unless we redeploy the exact same file, which will cause CF to regenerate the byte code (hopefully correctly this time) and fix the issue.

I realize this may sound "unbelievable", so I am going to try to recreate this behavior on my local dev server and see if I can get my hands on the byte code in both the working state and the broken state.

From there, perhaps I'll be in a better position to provide you guys more information to work with. I really appreciate everyone sharing their thoughts!

Brian

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:359327

[hofar...@houseoffusion.com]

did you check the settings I suggested previously ? as this do affect the
generation of the class files which contain the java byte code

On Fri, Sep 19, 2014 at 2:37 PM, Brian FitzGerald <bmfitz...@gmail.com>
wrote:

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:359328