Feeds disappearing from main feeds table (was Re: Garbage Collection redux)

[Dave Winer]

I have a workaround implemented that should bring back most of the missing feeds if not all of them.

http://worknotes.scripting.com/february2012/22912ByDw/river2FeedsDisappearing/

I changed the subject on this thread, because we don't know what's causing the feeds to disappear. It may or may not be the garbage collection code. I have a feeling it's not that.

Dave

On Wed, Feb 29, 2012 at 4:19 PM, Adam Curry <foob...@gmail.com> wrote:

OK, good. This is a river2 server only, so nothing else runs on it and
indeed, everything is in config.root

Leaving it untouched in that regard.

AC

On Feb 29, 3:17 pm, Dave Winer <dave.wi...@gmail.com> wrote:
> No, it puts the data in config.root unless you do the things you have to do
> (I forget exactly how that works) to make it use river2data.root.
>
> It really doesn't make much difference unless you're using a lot of other
> apps that store stuff in config.root.
>
> And I'm sure that has nothing to do with the problem with feeds getting
> trashed.
>
> Dave
>
>
>
>
>
>
>

> On Wed, Feb 29, 2012 at 4:15 PM, Adam Curry <foobar...@gmail.com> wrote:
> > We can still be daydream believers!
>
> > Interstingly, this install, which I've had for quite some time, does
> > not have a river2data.root!
>
> > Should I just reinstall to get the root?
>
> > AC
>
> > On Feb 29, 3:08 pm, Dave Winer <dave.wi...@gmail.com> wrote:
> > > Actually in my nightmares it's Marty Feldman in the toilet. In his role
> > of
> > > Eye-gor in Young Frankenstein.
>
> > > Something else horrible -- Davy Jones died. I thought for sure he would
> > > outlive me. (Well I never actually thought about it.)
>
> > > First thing I'm doing is doing a save-copy of my river2data.root which
> > had
> > > grown to 1.5GB. I keep forgetting to do that. Lucky I spotted it.
>
> > > I'm going to find out what's happening. It's happening here too, btw.
>
> > > Dave
>
> > > On Wed, Feb 29, 2012 at 4:05 PM, Adam Curry <foobar...@gmail.com> wrote:
> > > > First of all, glad you're feeling better.
>
> > > > re: toilet, biggest laugh of the day here in Austin!
>
> > > > I see that indeed feeds get 'collected' for formatting errors as well
> > > > as non connections (i.e. server not responding or jjst gone)
>
> > > > I have lots of quetions of course, but none pressing. Also, the
> > > > garbage Collection issue is not a show stopper for me, everything is
> > > > working now, but of course I haven't had to unsubscribe from any feeds
> > > > lately :-)
>
> > > > AC
>
> > > > On Feb 29, 3:02 pm, Dave Winer <dave.wi...@gmail.com> wrote:
> > > > > I'm going to take a look at this.
>
> > > > > Just want to point out there are other ways for feeds to get into the
> > > > > trash, not just via garbage collection.
>
> > > > > And for what it's worth I can imagine things that are more
> > horrifying,
> > > > like
> > > > > opening a toilet and finding a person down in there laughing at me.
>
> > > > > Sorry -- I've been fighting a very bad cold and it makes for weird
> > > > dreams.
> > > > > :-)
>
> > > > > Dave
>
> > > > > PS: I'm feeling better.
>
> > > > > On Tue, Feb 28, 2012 at 5:13 PM, Adam Curry <foobar...@gmail.com>
> > wrote:
> > > > > > Looping around to this thread:
>
> >http://groups.google.com/group/river2-users/browse_thread/thread/64e4.
> > > > ..
>
> > > > > > I have 1 server that runs rivers for several users based on opml
> > > > > > subscription lists.
>
> > > > > > One of these if my adamfriends.com river.
>
> > > > > > I started noticing 'slow' news days. But futher inspection showed I
> > > > > > wasn't receiving items from several feeds. There's about 90 feeds
> > that
> > > > > > list alone, so it took a while, possibly weeks for me to catch on
> > that
> > > > > > it was getting worse.
>
> > > > > > I checked the server and to my horror saw that
> > > > > > config.root.river2.trash was filled with subs I indeed wasn't
> > seeing
> > > > > > items from anymore.
>
> > > > > > My roots are up to date, so something must be broken, or perhaps
> > the
> > > > > > fix Dave put in place doesn't solve some other problem that is
> > > > > > collecting these feeds as garbage incorrectly.
>
> > > > > > For now I set config.river2.prefs.flGarbageCollectFeeds to false,
> > > > > > which appears to at least not delete feeds from my list.
>
> > > > > > I tried testing this with a very small list when the issue first
> > came
> > > > > > up and feeds were not deleted immediately, so I think something
> > else
> > > > > > is hapening, but I do not know where/how to look.
>
> > > > > > AC

[Adam Curry]

Self-healing rivers... us Texas folk like that kinda water
management :-)

The fix was applied before I could count feeds on the server, but the
log shows some re-subbing around the time the update came in:

http://blog.curry.com/images/2012/02/29/addedFeeds.jpg

Keeping an eye on the logs and the trash table...

AC

On Feb 29, 5:13 pm, Dave Winer <dave.wi...@gmail.com> wrote:
> I have a workaround implemented that should bring back most of the missing
> feeds if not all of them.
>

> http://worknotes.scripting.com/february2012/22912ByDw/river2FeedsDisa...

[Dave Winer]

How's it working?

[Adam Curry]

So far so good. Nothing trashed that I can detect and still seeing the same amount of feeds.

AC

[Dave Winer]

This morning, everything was still running smooth, I still had 811 feeds in the rotation, nothing had been garbage-collected because I had that turned off.

Then I turned it on, and watched for the next few hours, and so far everything is running fine. I had to increase the time between scans from 10 minutes to 15 minutes because with the extra feeds I'm pushing the limit.

Net-net -- everything looks good over here.

Dave

[Adam Curry]

Here's what I'm seeing:

Some feeds are showing up in the trash table.
The last error in the stats table for these feeds is : Can't process the file because iat doesn't appear to be a feed.

Then apparently the feed *does* appear to be a valid feed and I get resubscribed, resulting in the entire feed showing up in my river. In some cases this is a lot of items.

So the fix works, but may need some additional work if it keeps re-subscribing to feeds and showing the whole feed each time in the river.

I think I need a few more days of observing the patient.

AC

[Dave Winer]

When it does that, includes all the items as new, could you post a link to the river.

Also what feed or feeds is it doing this with?

I need data if we're going to figure this out.

[Adam Curry]

http://adamfriends.com/

Search for

willowgene

That is a twitter feed for user 13732042 as you can see in this screenshot from my trash table:

http://blog.curry.com/images/2012/03/01/trashedFeeds.jpg

AC

[Dave Winer]

So how is this feed getting into the trash?

You seem to have a theory about that -- what is the theory?

Dave

[Adam Curry]

Only thing I can think of is that the feed(s) are producing an error to often and then are queued as 'no good.

I exported my trash table which may help?

http://dl.dropbox.com/u/5538262/tmp/river2.trash.fttb.zip

AC

[Dave Winer]

Hold on. Pretty sure I found the problem.

[Dave Winer]

I just released a utility script.

Update river2.root and run the script. It's at river2suite.utilities.listFeedsWithErrors.

I bet there are a bunch of them. Take a screen shot and let me know.

Dave

[Adam Curry]

List was to big to take a screenshot of!

OPML of the output: http://dl.dropbox.com/u/5538262/tmp/errantFeeds.opml

[Dave Winer]

You were about to lose a bunch of those feeds.

[Dave Winer]

Here's the fix.

http://worknotes.scripting.com/march2012/3112ByDw/river2ResetCtconsecutivereaderrors/

Update river2.root.

Dave

[Adam Curry]

Coolio.

Just ran the utility script and the list is now down to 5. I presume these will zero out next top of the minute

I do believe some of these are indeed broken feeds, checking on that now.

AC