JError has been getting in my way.

[Paladin]

JError has been causing trouble for testing, and part of the issue is
it's making an unsound assumption. It's apparently assuming it will
never ever be called when JApplication is not instantiated. These are
some notes about how I'm presently trying to work around this. Feel
free add to this or correct me where I'm wrong.

To get down to specifics, in the particular piece of code I'm working
on now, I feed it bad data to make sure it can detect it and handle it
gracefully, and it detects the bad input and throws an error. (If it
wasn't hardcoded to make a static call to JError, but rather looked up
an error object in a factory and called it, I could intercept the call
at this point and route it to another handler. I can stub the current
error class out with a test error object, but then we get obnoxious
interactions between the test object and the real object that can only
be solved by process isolation, which doesn't seem to work for us yet
in phpunit, as I think we've amply demonstrated. I should also note
that I'm aware of issues in doing it that way -- what happens if the
error object isn't available, for one -- so this isn't a plea for
doing it another way, at least at the moment. I'm sure there's a
better way, but I haven't come up with an alternative and I'm not
going to even try to open that can of worms for quite a while, yet.)

Anyway, to get back on point, it throws an error. Which launches us
into JError, and the first thing it does is check with JFactory to get
the JApplication object reference, except that doesn't exist in the
testing environment (I'd like to keep it that way, BTW). And, since
JError calls it without specifying an Application ID, JFactory tosses
another error, which results in an infinite loop (and not the one in
Cupertino). This is the bad assumption I mentioned at the top.

I've tried a few different error handlers for it (even 'ignore') and
they all run aground in what appears to be this same infinite loop.

It strikes me the most appropriate handler for testing would probably
be callback, so the testing system can log the errors encountered for
comparison with expected errors. By returning false from the callback,
I think I'm managing to duck the infinite loop problem. I'm too tired
to finish the code and check it in to the tree tonight, so I'll do
that tomorrow to give a more concrete example of what I'm talking
about. Anyway, the thing is to log the appropriate info into a
variable in the testing class, and then compare that with what you
expect. My current local version of JSimpleXMLTest is passing now,
without throwing errors even if run without isolation, and I think
this approach will work for other tests that are running afoul of
JError.

I can't dig up a lot of info on JError's options and what they do and
what the requirements for each are (I tried both true, false, and void
for returns from callback, for example, before I found one that worked
for me) so if anyone can toss me a bone on this, I'd appreciate it.
What I'm looking for is a way to tell JError "don't bother your pretty
head about this, just pass the error back to me and let me take care
of the big, bad trouble for you." I think I've found it, but I could
be wrong.

A larger question might be whether it's a good idea for JError to just
throw in the towel if Application is not instantiated, but that's
beyond my current scope.

[Ian MacLennan]

On Fri, Nov 20, 2009 at 1:32 AM, Paladin <arlen....@gmail.com> wrote:

JError has been causing trouble for testing, and part of the issue is
it's making an unsound assumption. It's apparently assuming it will
never ever be called when JApplication is not instantiated. These are
some notes about how I'm presently trying to work around this. Feel
free add to this or correct me where I'm wrong.

To get down to specifics, in the particular piece of code I'm working
on now, I feed it bad data to make sure it can detect it and handle it
gracefully, and it detects the bad input and throws an error. (If it
wasn't hardcoded to make a static call to JError, but rather looked up
an error object in a factory and called it, I could intercept the call
at this point and route it to another handler. I can stub the current
error class out with a test error object, but then we get obnoxious
interactions between the test object and the real object that can only
be solved by process isolation, which doesn't seem to work for us yet
in phpunit, as I think we've amply demonstrated.

I had process isolation working fine for the cases where it was required using @runInSeparateProcess.  The way I had this implemented was specifically for the where we load mocks, so that we could run tests that need mocks in a separate process and not have the mock classes interfere with the real classes.

That being said, being able to inject an object rather than a class would be useful, but I don't know if it is something we can change in 1.6.  Would be something to see about doing for either 1.7 or 2.0.

 

I should also note
that I'm aware of issues in doing it that way -- what happens if the
error object isn't available, for one -- so this isn't a plea for
doing it another way, at least at the moment. I'm sure there's a
better way, but I haven't come up with an alternative and I'm not
going to even try to open that can of worms for quite a while, yet.)

Anyway, to get back on point, it throws an error. Which launches us
into JError, and the first thing it does is check with JFactory to get
the JApplication object reference, except that doesn't exist in the
testing environment (I'd like to keep it that way, BTW). And, since
JError calls it without specifying an Application ID, JFactory tosses
another error, which results in an infinite loop (and not the one in
Cupertino). This is the bad assumption I mentioned at the top.

Infinite loops are certainly bad, and yes, I have noticed that it happens.  Was looking at it last night.  We should perhaps look at how we might fix JError so that this doesn't happen.
 

I've tried a few different error handlers for it (even 'ignore') and
they all run aground in what appears to be this same infinite loop.

It strikes me the most appropriate handler for testing would probably
be callback, so the testing system can log the errors encountered for
comparison with expected errors. By returning false from the callback,
I think I'm managing to duck the infinite loop problem. I'm too tired
to finish the code and check it in to the tree tonight, so I'll do
that tomorrow to give a more concrete example of what I'm talking
about. Anyway, the thing is to log the appropriate info into a

Okay, will look at the example when it's ready.
 

variable in the testing class, and then compare that with what you
expect. My current local version of JSimpleXMLTest is passing now,
without throwing errors even if run without isolation, and I think
this approach will work for other tests that are running afoul of
JError.

Okay, that might be a worthwhile approach as an alternative to running in separate processes.

I can't dig up a lot of info on JError's options and what they do and
what the requirements for each are (I tried both true, false, and void
for returns from callback, for example, before I found one that worked
for me) so if anyone can toss me a bone on this, I'd appreciate it.
What I'm looking for is a way to tell JError "don't bother your pretty
head about this, just pass the error back to me and let me take care
of the big, bad trouble for you." I think I've found it, but I could
be wrong.

I have attached the wiki source for the only docs I could really find that we had.  We had done something based on patError at one point.  We don't have the patError library anymore, but I don't know how closely we used the concepts.  The forum thread is at http://forum.joomla.org/viewtopic.php?f=304&t=24910.  The mentioned wiki source is attached to this email.
 

A larger question might be whether it's a good idea for JError to just
throw in the towel if Application is not instantiated, but that's
beyond my current scope.

It seems to me that throwing an error inside JError is in general not the best idea.  I would be inclined to just die or something if something inside JError itself fails, or at least handle it in a way that it won't invoke itself (I have a hard time imagining that if JError failed the first time through that it would pass the second time through).

Ian
 

[Paladin]

On Nov 20, 8:30 am, Ian MacLennan <ian.maclen...@joomla.org> wrote:
> I had process isolation working fine for the cases where it was required
> using @runInSeparateProcess.  The way I had this implemented was
> specifically for the where we load mocks, so that we could run tests that
> need mocks in a separate process and not have the mock classes interfere
> with the real classes.

Yes, but I ran into difficulty with it while testing the changes I was
making to move the JUtility code. Specifically, the problem was that
JSessionTest loaded session.php, and after that was loaded, I could no
longer use getMock in JUtilityTest to create a mock JSession object
with a method overriding the static method I had added to JSession.
(If the method were not declared static, the test and the mock object
would work fine. Static, and the method in JSession would supplant the
mock method.) The tests would pass only so long as no other test in
the entire suite loaded session.php, but as soon as any test loaded
that file, the tests failed, despite the fact that the tests
themselves were not changing, nor were the data the tests were being
performed upon.

This happened despite @runInSeparateProcesses, and when it happened it
became apparent to me that I was never really achieving isolation. At
best I was getting a "sort of" isolation which I couldn't trust
because I had an example right in front of me of one test corrupting
the results of another, supposedly isolated, test.

> I have attached the wiki source for the only docs I could really find that
> we had.  We had done something based on patError at one point.  We don't
> have the patError library anymore, but I don't know how closely we used the

> concepts.  The forum thread is athttp://forum.joomla.org/viewtopic.php?f=304&t=24910.  The mentioned wiki

> source is attached to this email.

I'll check that over. I've wondered about maybe some pages in the wiki
to capture live for all of us the wisdom we each glean from our
trials. Testing tips that we've found useful or things we've found
especially problematic. But maybe that sort of thing is best left for
this medium instead, and only at the end gathered up and made sense of
for posterity.

I've committed the new version of JSimpleXMLTest to the unittest tree.
The new touches are above the setup method. In a nutshell, setup saves
the current state of the error handler, sets it to callback for
everything and points it at the callback method provided by
JSimpleXMLTest to record the error message, initializes the error
recording location, then instantiates the class and begins testing.
Teardown then restores the error handlers on the way out, back to what
they were before the test began.

Whenever you start exploring black boxes like I did, you invariably
come up with some "superstitious" rules, rules that don't really exist
but only appear to be rules because of the way you poked around the
box. I've tried to clean all the superstition out of the test code; if
you find some still there, let me know.

> It seems to me that throwing an error inside JError is in general not the
> best idea.  I would be inclined to just die or something if something inside
> JError itself fails, or at least handle it in a way that it won't invoke
> itself (I have a hard time imagining that if JError failed the first time
> through that it would pass the second time through).

It's not so much that it's throwing an error inside JError itself, but
rather that JError is calling a JFactory method as part of its error
handling, which in turn is throwing another error. JError, under the
error handling mode it's been told to follow, then detects that this
latest error has been thrown while handling a previous error, declares
it's in an infinite loop and quits.

While poking around the black box of JError, I found a couple of
options that managed to escape the loop detection, but not the loop,
and it was only this particular callback combination that seemed to
escape the loop as well.

It's always a bit iffy to include the normal code of a system in error
handling, because once an error happens you can't count on the system
being fit to do any processing at all, but I'm going to wait until far
more than the current 10% of the system is under test before I start
doing more than idly speculating about major rework efforts like that.
I want a more accurate picture of where we are before we even think
about changing course.