Connection error
Jason Striegel
spitting out the following error:
Phirehose: Failed to connect to stream: Connection refused (111).
Sleeping for 16 seconds.
It kept attempting to connect every 16 seconds and outputting the same
error. After killing and relaunching the process, it connected on the
first shot. What's odd is that I didn't see any indication that the
connection failure limit was reached, and the script hadn't exited.
I'm digging through the code to see what might be happening, but does
anyone know off hand why (or under what circumstances) this might have
happened? Any recommendations for having catastrophic errors fail and
restart gracefully?
Thanks!
Jason (@jmstriegel)
Fenn
Hmmm - well, first off - a "Connection refused" error is a TCP level
error, not HTTP. Phirehose is behaving (largely) as designed. From the
twitter docs:
---
When a network error (TCP/IP level) is encountered, back off linearly.
Perhaps start at 250 milliseconds and cap at 16 seconds. Network layer
problems are generally transitory and tend to clear quickly.
---
In your case, something is obviously going wrong, and the problem was
not "clearing". The fact that when you terminated it and restarted it
worked first time is seems to point at a problem with the code (rather
than twitter) but I can't really see how this could be happening,
unless PHP's fsockopen() is setting the 'conn' attribute to something
very odd.
Phirehose is designed to throw an exception once connectFailuresMax is
reached, but by the looks of it (in your case), this isn't happening.
I'll have a look at the code and see if there's some type edge case
where the connection failure could be being missed.
Thanks for your email,
Cheers!
Fenn.
Adam Green
I have also started getting failures in the last couple of days. Where are the logs, so I can check them?
Adam Green
CEO, Grazr Corp
ad...@alertrank.com
781-879-2960
Our sites:
http://alertrank.com
http://grazr.com
http://topstocktweets.com
Fenn
recently and you've both started getting these errors at around the
same time, my guess is that something is going on at Twitter with the
connection handling.
It may be worth moving this discussion to the twitter development talk
mailing list (http://groups.google.com/group/twitter-development-
talk).
The logging goes to wherever your log() function is set to go - By
default, it uses the PHP error_log() function, but you can override to
whatever you like.
I'll post a new version of Phirehose that has some logging
enhancements to help with this sort of thing.
Cheers!
Fenn.
> a...@alertrank.com
> 781-879-2960
>
> Our sites:http://alertrank.comhttp://grazr.comhttp://topstocktweets.com
Fenn
I've release a new version (0.2.2) that has better error logging on
connection failures and does definitely disconnect after a maximum
number of errors (or at least, it does for me).
This should help you track down where the problem may be.
It also contains a consuming/processing example for the ghetto-queue.
Cheers,
Fenn.
att...@gmail.com
I have also been using your nice Phirehose library for several months
and until last week it was stable and I was quite happy with it.
I'm mainly following user_ids...and the average status rate is between
300 to 1000 status/min with peeks at 5000.
Now, I have to face more instability:
Like some others, I have the following error message:
Phirehose: Failed to connect to stream: Connection refused (111).
And sometimes, no details like:
Phirehose connection error occured:
Phirehose: Closing connection.
I'm wondering if the problem is not on the Twitter side?
By the way, I noticed a little bug:
at the beginning when I implemented Phirehose, I gave a wrong user_id
within the checkFilterPredicates() function.
Of course, the Tweeter API returned an error. That error was handled
by Phirehose. The problem is that, even if I get rid off the wrong id
at the next checkFilterPredicates() call, Phirehose still sumbitted
the indentical list (with the wrong value) to the Twitter API!
and thanks again for the phirehose library!
cheers,
Kim LELOUP
Fenn
Glad the library is helping you out. Regarding the disconnections, I'm
really thinking that there's something going on on the Twitter side,
as several people have started reporting this at the same time (and
the library hasn't changed).
I also notice that Twitter have added the following note in bold in
the documentation page:
---
The Streaming API may disconnect your connection at any time, perhaps
multiple times per day. Proper client coding will prevent data loss
due to occasional disconnections.
---
Whilst Phirehose _should_ handle this, it appears there may be
something that's stopping it (ie: continued refused connections).
Kim, can I ask that you try the updated version (0.2.2) which has much
more informative logging and see what happens?
Regarding the issue with invalid user_id's - You're correct. Filter
predicates are setup at the start of the connect() method, which is
outside of the error handling loop. If the predicates cause an error,
Phirehose will continue to retry until the maxFailures is reached.
I'll need to restructure how this works to make it deal with parameter
specific HTTP errors.
Cheers and thanks so much for the feedback,
Fenn.
Adam Green
I have time this weekend to swap in the new version of the library,
and then send you copies of any logged errors. For now the
connectivity is still good, but not as good as before. Until this past
week or two it has been able to stay live for many weeks without a
problem. Now it fails about once every 2-3 days. The problem is that
it fails silently. the daemon that calls it doesn't stop running or
throw a visible error. I have a separate cron job that looks for new
tweets in the MySQL cache every 5 minutes and emails me if they fail
to appear, so I know about failures. If I kill and then restart the
deamon, it starts collecting again.
BTW, you can see the app that uses the library at
http://topstocktweets.com. It is already getting about 75,000 page
views per month after 2 months, so I am very happy.
--
Adam Green
CEO, Grazr Corp
att...@gmail.com
Yesterday I moved to the last version 0.2.2.
Unfortunately, I faced today exactly the same symptomes as Adam Green
did.
It failes silently...I also have a separte cron task that checks
incoming tweets every 5min. The only solution I have found so far is
to kill and restart de task.
Kim
Adam Green
failure times as a way of isolating the problem? I'm guessing that the
Twitter servers are doing something that is causing all Phirehose
instances to fail.
Today there was a failure at about 11:52AM ET.
On Feb 9 there was a failure at 9:32AM ET.
On Feb 7 there was a failure at 4:07AM ET.
Fenn, are you running an instance of the library? Has it failed
lately? If you aren't, you probably should. This error is likely
systemic for all instances of Phirehose.
I'm happy with the library, so starting tomorrow I will begin testing
with 0.2.2 to capture the Twitter error message, if there is one.
--
Adam Green
CEO, Grazr Corp
Jason Striegel
connection errors, without exiting or showing the retry limit error.
Killing the process and re-executing allowed it to immediately
reconnect. Odd, hunh?
I recall reading somewhere in the documentation that Twitter may
change DNS throughout the day to divert resources between servers. Is
there any chance that the socket calls in PHP are somehow caching the
IP address that the DNS name resolves to? Perhaps Twitter kicks a
connection, changes DNS, and Phirehose attempts to reconnect to the
old IP? It still doesn't explain the odd lock-up or why the exception
isn't thrown, but I'm sort of grasping at straws. :)
Jason
Fenn
Thanks heaps for your incredibly helpful reports. There's definitely
something weird going on :)
Jason - Like you, I'm wondering whether it's something strange like
DNS updates which the long-lived PHP process is caching DNS responses
(unlikely, but definitely possible).
Just a quick question for you all - When Phirehose fails does it:
- 1) Just keep trying to connect forever (ie: connecting messages
coming up in logs)?
- 2) Stop receiving tweets (so no tweets, but tweet rate summaries
still coming up in logs)?
- 3) Just go completely silent (ie: no output, no log entries,
nothing)?
Thanks heaps!
Fenn.
Fenn
The more I think about Jason's DNS changing theory, the more I think
it's correct. I noticed that Twitter has their DNS TTL on
stream.twitter.com set VERY low (60 seconds), so I'm willing to bet
that's for a reason.
Consequently, I've built/uploaded a special experimental release of
Phirehose just for you guys, that contains a few key updates - You can
find it here:
http://phirehose.googlecode.com/files/phirehose_experimental.tar.gz
There are two major changes in this:
1) DNS is now resolved explicitly on every (re)connect - You also get
improved log output to see what's going on. A connect now looks
something like this:
---
Phirehose: Connecting to twitter stream: http://stream.twitter.com/1/statuses/filter.json
with params: array ( 'delimited' => 'length', 'track' =>
'sldkjfdsfdssss',)
Phirehose: Resolved host stream.twitter.com to 168.143.162.55
Phirehose: Connecting to 168.143.162.55
---
2) The concept of an $idleReconnectTimeout, which can be set upon
initialization. This tries to make a library-native version of what
you guys have implemented manually. Basically, if no tweets are
received for $idleReconnectTimeout seconds, the client will
automatically terminate/reconnect to attempt to recover.
Would be very keen on hearing from you guys whether this appears to
fix the problem (I haven't managed to replicate it myself reliably
yet) so I can make these changes official and integrated into the main
build.
Cheers/thanks,
Fenn.
Adam Green
This is now at the top of my queue, since I can no longer leave the
house while it is happening. I'll report back my experiences with the
new test code.
Adam Green
with the new experimental library with no additional changes to my
code. Phirehose did get the tweets, and my code stored it into a MySQL
cache correctly. Unfortunately, my code failed to parse the cached
tweets, even though it is emitting no errors. I've reverted to 0.1.0.
I'd rather not take apart my parsing code right now.
Fenn, I'm capturing the status with consume(), but my parsing code is
failing silently. Has the format of a tweet status resulting from the
following code changed?
class FilterTrackConsumer extends Phirehose
{
public function enqueueStatus($status)
{
global $oParse;
$oParse->insert_cache($status);
}
}
// Start streaming
$sc = new FilterTrackConsumer('****', '****', Phirehose::METHOD_FILTER);
$sc->setTrack($stock_list);
$sc->consume();
Adam Green
JSON, while I had it set to XML in 0.1.0. I changed to XML in the new
code, and everything is good. I'll report any disconnects and errors
that show up.
Fenn, I'm reluctant to be one of the first people to test
$idleReconnectTimeout for fear of being banned by twitter. I know its
needed, but they don't always respond in a thoughtful way. They often
ban first, and talk later. How much testing have you given this?
Jason Striegel
happens.
To answer your earlier question, I'm seeing the behavior in your 3rd
example. Logs stop outputting, things go silent, but the process is
still running. Perhaps it's blocking on reading something from the
socket?
Here's my most recent crash output:
Phirehose: Consume rate: 18 status/sec (1056 total), avg
enqueueStatus(): 0.02ms, avg checkFilterPredicates(): 0ms (12 total)
over 60 seconds.
Phirehose: Successfully rotated active stream to queue file: ../../
datastore/streamdata/stream-input.20100212-105039.queue
Phirehose: Opening new active status stream: ../../datastore/
streamdata/.phirehose-queue.current
Phirehose: Successfully rotated active stream to queue file: ../../
datastore/streamdata/stream-input.20100212-105110.queue
Phirehose: Opening new active status stream: ../../datastore/
streamdata/.phirehose-queue.current
Phirehose: Phirehose connection error occured:
Phirehose: Closing Phirehose connection.
Phirehose: Connecting to twitter stream: http://stream.twitter.com/1/statuses/filter.json
with params: array ( 'delimited' => 'length', 'track' => 'xyzzy',)
No logs are added after that point, so it appears things are failing
inside of connect(). BTW, what does the @ sign before a variable do
(as opposed to the function call)? I haven't seen that before:
@$this->conn = fsockopen($scheme . $urlParts['host'], 80, $errNo,
$errStr, $this->connectTimeout);
J
On Feb 12, 12:21 pm, Adam Green <a...@alertrank.com> wrote:
> First attempt to use new code. I tried replacing the 0.1.0 library
> with the new experimental library with no additional changes to my
> code. Phirehose did get the tweets, and my code stored it into a MySQL
> cache correctly. Unfortunately, my code failed to parse the cached
> tweets, even though it is emitting no errors. I've reverted to 0.1.0.
> I'd rather not take apart my parsing code right now.
>
> Fenn, I'm capturing the status with consume(), but my parsing code is
> failing silently. Has the format of a tweet status resulting from the
> following code changed?
>
> class FilterTrackConsumer extends Phirehose
> {
> public function enqueueStatus($status)
> {
> global $oParse;
> $oParse->insert_cache($status);
> }
>
> }
>
> // Start streaming
> $sc = new FilterTrackConsumer('****', '****', Phirehose::METHOD_FILTER);
> $sc->setTrack($stock_list);
> $sc->consume();
>
>
>
>
>
> On Fri, Feb 12, 2010 at 12:23 PM, Adam Green <a...@alertrank.com> wrote:
> > Old code disconnected at 11:51am ET.
>
> > This is now at the top of my queue, since I can no longer leave the
> > house while it is happening. I'll report back my experiences with the
> > new test code.
>
> >http://alertrank.com
> >http://grazr.com
> >http://topstocktweets.com
>
> --
> Adam Green
> CEO, Grazr Corp
Adam Green
There were no error messages in the error log file.
I have the log file set in php.ini with:
error_log = "[log file name]"
All I'm getting in this file are messages like this:
[12-Feb-2010 14:45:05] Phirehose: Consume rate: 0 status/sec (10
total), avg enqueueStatus(): 0.53ms, avg checkFilterPredicates(): 0ms
(10 total) over 60 seconds.
Let me know if I should be doing anything else to catch the errors.
I hope we get to the bottom of this soon. I can't babysit this all
day, every day.
Misja Hoebe
errors in the log. This is when using the experimental version posted
earlier.
On 12 feb, 21:01, Adam Green <a...@alertrank.com> wrote:
> Connection failure at 2:45pm ET. I am running the experimental code.
> There were no error messages in the error log file.
>
> I have the log file set in php.ini with:
> error_log = "[log file name]"
>
> All I'm getting in this file are messages like this:
> [12-Feb-2010 14:45:05] Phirehose: Consume rate: 0 status/sec (10
> total), avg enqueueStatus(): 0.53ms, avg checkFilterPredicates(): 0ms
> (10 total) over 60 seconds.
>
> Let me know if I should be doing anything else to catch the errors.
>
> I hope we get to the bottom of this soon. I can't babysit this all
> day, every day.
>
> On Fri, Feb 12, 2010 at 2:05 PM, Jason Striegel
>
> ...
>
> meer lezen »
Fenn
Thanks heaps for that - I can't think of anything off the top of my
head that should cause the entire process to hang indefinitely (ie:
all blocking operations have timeouts) but something is definitely
going on.
The @ symbol on the fsockopen line suppresses all error output for
that statement (possibly stupidly), as the error is caught/reported by
the following code. It could be a good debugging exercise to remove
it.
I've not managed to replicate the problem myself which is making it
hard to debug.
The other thing that would be super handy if you're feeling brave and
under linux is attaching strace to the PHP process so we can see what
it's trying to do:
---
strace -p <pid_of_phirehose>
---
I'll keep playing - let me know if you work anything out.
Cheers!
Fenn.
> ...
>
> read more »
Fenn
Thanks heaps Adam and Misja - You've managed to confirm that there's
definitely something going on on Twitter's side (ie: streams can go
"dead"), so we just need to work out how to make Phirehose deal with
it.
Adam - in your example, it looks like you still have a completely
valid/operating stream, it's just that no tweets are coming through -
ie: Phirehose is working, there's just no data being delivered
(assuming you're still getting log messages).
According to the docs, Twitter should send "keep alive" new-lines, so
it should be safe to use the idleReconnectTimeout if implemented
correctly (which it isn't quite yet).
Basically, the experimental version of the library SHOULD reconnect
after 5 minutes of idleness (I'd be interested to see if this is
working for people).
If not, it could be hanging somewhere else, which we should be able to
work out once we can reliably replicate it.
Cheers and good luck!
Fenn.
> ...
>
> read more »
Fenn
I've had my stream running for a couple of days now (using the
experimental version of the library) and so far have been unable to
replicate the idle stream/hanging problem above (from what I can
gather).
Are others still experiencing this?
Cheers/thanks,
Fenn.
> > > >> >>> > > > incoming tweets every 5min. The...
>
> read more »
Misja Hoebe
nothing shows up in the logs, just the last recorded rate info:
2010-02-16T15:12:02+00:00 INFO (6): Consume rate: 0 status/sec (26
total), avg enqueueStatus(): 0.66ms, avg checkFilterPredicates():
0.02ms (10 total) over 60 seconds.
Unfortunately I can't separately test the client, it's fully wrapped
in forking code, and I rely on it in production ...
Misja
> ...
>
> meer lezen »
Adam Green
this problem is killing me. I can't stay online 24/7. I pray that you
find a solution soon.
--
att...@gmail.com
until today 16.11 (GMT+1).
16-02-2010 16:10:27 - Consume rate: 12 status/sec (694 total), avg
enqueueStatus(): 4.98ms, avg checkFilterPredicates(): 356.63ms (12
total) over 60 seconds.
16-02-2010 16:11:27 - Consume rate: 11 status/sec (689 total), avg
enqueueStatus(): 2.1ms, avg checkFilterPredicates(): 349.6ms (12
total) over 60 seconds.
16-02-2010 17:31:34 - Connecting to twitter stream:
http://stream.twitter.com/1/statuses/filter.json with params: array
( 'delimited' => 'length', 'follow' => '12,...)
16-02-2010 17:31:34 - Resolved host stream.twitter.com to
168.143.162.55
16-02-2010 17:31:34 - Connecting to 168.143.162.55
Kim
> ...
>
> plus de détails »
Fenn
The (kinda) bad news is: I too have managed to suffer from the
"everything stops" error - According to my logs, my stream died at
approximately Feb 17 02:12 GMT+11:00, which aligns with.... 15:12 GMT,
EXACTLY the same as you guys.
I have also managed to catch it in the act and done some low level
debugging & network inspection and have managed to work out that the
problem is definitely on Twitter's end (ie: they give you an active
TCP connection that sends/receives NO packets, NO keep-alives, NO
anything - this is against the streaming API spec).
To reiterate - this is a Twitter stream problem, not a problem with
the Phirehose library, per-se.
The good news is, I should be able to make the library detect this
sort of thing and abort/reconnect when it happens. I've refactored
some of the idleness detection code to be further up the stream
processing chain, so it should ALWAYS detect these failures,
regardless of how weirdly the stream is behaving.
I've uploaded another alpha/experimental version here:
http://phirehose.googlecode.com/files/phirehose_0.2.3-alpha-dev.tar.gz
I know it's a pain for some of you to swap in code as we're testing
this, but as this problem is:
- a) Out of spec from the docs (ie: unknown) and,
- b) Intermittent
Then it means I have no more ability to test it than you guys (ie: I
have to wait for the stream to break to see what it's doing), so the
more people testing it, the more it helps me fix it.
If you guys could let me know how you go, that would be great -
Cheers,
Fenn.
On Feb 17, 8:11 am, "att...@gmail.com" <att...@gmail.com> wrote:
> I'm running the experimental version since february 12. It was stable
> until today 16.11 (GMT+1).
>
> 16-02-2010 16:10:27 - Consume rate: 12 status/sec (694 total), avg
> enqueueStatus(): 4.98ms, avg checkFilterPredicates(): 356.63ms (12
> total) over 60 seconds.
> 16-02-2010 16:11:27 - Consume rate: 11 status/sec (689 total), avg
> enqueueStatus(): 2.1ms, avg checkFilterPredicates(): 349.6ms (12
> total) over 60 seconds.
> 16-02-2010 17:31:34 - Connecting to twitter stream:http://stream.twitter.com/1/statuses/filter.jsonwith params: array
> > >> > > > >> >>> DNS...
>
> read more »
Adam Green
code. I'll act as the control. If my copy fails and those who run the
new code stay up, we'll have a good test.
--
John Kalucki
Streaming API. I haven't heard reports of this behavior on other
clients.
I'd like to help you get to the bottom of these issues. If you have a
reproducible error case, I need to know your account name and the
time, in UTC, of the error. I can then examine the server logs and
perhaps diagnose the problem. Please create a new thread for each
client error case on this forum and we can converse there.
I've noticed a larger proportion of duplicate connections and
aggressive reconnections on the Streaming API recently, and I last
night (Feb 16th, around 03:00 UTC) I turned the anti-abuse knob up a
few clicks.
A side question: Does Phirehose repeatedly reconnect as a normal
function of its operation, or does it remain connected until it
receives a TCP RST from twitter.com?
-John Kalucki
http://twitter.com/jkalucki
Infrastructure, Twitter Inc.
> ...
>
> read more »
Fenn
Thanks so much for having a look at this - and I'm impressed at how
much you're keeping an eye on things :)
I'll start another thread (probably tomorrow) with some specific
details of actual connections that were suffing this problem, but in
the meantime, some general comments:
You may be able to identify Phirehose connections by the user agent -
All Phirehose clients will have a UA string that looks something like:
---
'Phirehose/0.2.2 +http://code.google.com/p/phirehose/'
---
(With obvious variation for the specific client version.)
By default, Phirehose stays connected forever and puts a fair bit of
effort to never reconnecting aggressively. TCP and HTTP failures have
a reconnect back off algorithm implemented as per the docs (ie:
doubling up to a reasonable maximum).
Reconnections due to changed filter predicates are limited to an
absolute maximum of once every 2 minutes (unless users are hacking the
client, I guess) and will only reconnect when necessary.
In terms of how it responds to a TCP RST - I have to admit, it's a bit
hard to say for sure. Working in PHP I'm a few layers of abstraction
above such things, but when I was looking at these "dead" connections,
I did notice that linux still had the connection in an ESTABLISHED
state. That said, I haven't looked into this aspect in much detail.
On that note, it's highly likely that PHP (being PHP) is doing
something badly/wrong, which is what's causing this issue - The only
odd bit is that it has only started happening recently, and has been
really quite stable previously (at least since release in sept last
year).
Thanks so much for your time on this, I (and hopefully others) will
get you some more specific details ASAP.
Cheers!
Fenn.
On Feb 17, 3:32 pm, John Kalucki <jkalu...@gmail.com> wrote:
> I'm sorry that you all are having connection problems with the
> Streaming API. I haven't heard reports of this behavior on other
> clients.
>
> I'd like to help you get to the bottom of these issues. If you have a
> reproducible error case, I need to know your account name and the
> time, in UTC, of the error. I can then examine the server logs and
> perhaps diagnose the problem. Please create a new thread for each
> client error case on this forum and we can converse there.
>
> I've noticed a larger proportion of duplicate connections and
> aggressive reconnections on the Streaming API recently, and I last
> night (Feb 16th, around 03:00 UTC) I turned the anti-abuse knob up a
> few clicks.
>
> A side question: Does Phirehose repeatedly reconnect as a normal
> function of its operation, or does it remain connected until it
> receives a TCP RST from twitter.com?
>
> -John Kaluckihttp://twitter.com/jkalucki
> > > > >>> > > > On 11 fév, 01:15, Fenn <fenn.bai...@gmail.com> wrote:...
>
> read more »
Adam Green
John Kalucki
-John
On Feb 17, 9:12 am, Adam Green <a...@alertrank.com> wrote:
> Connection failed at 10:28am ET using version 0.2.2.
>
> ...
>
> read more »
Kim Leloup
Our connection with the account 1prism failed on feb. 17 at 4.28 PM (GMT+1) with the Phirehose version 0.2.3-alpha.
Here our log:
17-02-2010 16:27:48 - Consume rate: 13 status/sec (782 total), avg enqueueStatus(): 1.07ms, avg checkFilterPredicates(): 346.55ms (12 total) over 60 seconds.
17-02-2010 16:28:09 - Phirehose connection error occured:
17-02-2010 16:28:09 - Closing Phirehose connection.
17-02-2010 16:28:09 - Connecting to twitter stream: http://stream.twitter.com/1/statuses/filter.json with params: array ( 'delimited' => 'length', 'follow' => '12,13,...
17-02-2010 16:28:09 - Resolved host stream.twitter.com to 168.143.162.55
17-02-2010 16:28:09 - Connecting to 168.143.162.55
17-02-2010 16:33:17 - Idle timeout: No statuses received for > 300 seconds. Reconnecting.
17-02-2010 16:33:17 - Closing Phirehose connection.
17-02-2010 16:33:17 - Connecting to twitter stream: http://stream.twitter.com/1/statuses/filter.json with params: array ( 'delimited' => 'length', 'follow' => '12,13,...
17-02-2010 16:33:18 - Resolved host stream.twitter.com to 168.143.162.55
17-02-2010 16:33:18 - Connecting to 168.143.162.55
17-02-2010 16:33:18 - Consume rate: 0 status/sec (31 total), avg enqueueStatus(): 4.33ms, avg checkFilterPredicates(): 349.65ms (1 total) over 60 seconds.
17-02-2010 16:34:18 - Consume rate: 12 status/sec (722 total), avg enqueueStatus(): 2.89ms, avg checkFilterPredicates(): 347.89ms (12 total) over 60 seconds.
Thank you for helping us resolving this issue.
Kim
Sergi
with 0.2.3-alpha-dev. I'll let them running and tomorrow I'll take a
look. I keep you up to date.
S.
> > 16-02-2010 17:31:34 - Connecting to twitter stream:http://stream.twitter.com/1/statuses/filter.jsonwithparams: array
apetross
I am hesitant to switch the current version out because we are so
close to production - but I'm experiencing the same issue that others
are reporting here.
> ...
>
> read more »
Adam Green
automatically reconnect, which is an improvement. I'm in production
now, and switching has just been a matter of substituting one file for
the other, no recoding. Fenn is doing a good job of maintaining
compatibility.
--