503 errors

[Jeffrey Kishner]

Dave,

I'm getting a lot of varnish cache service errors for my blog.jeffreykishner.com site -- not the home page but many of the the individual pages.

Jeff

[Dave Winer]

We've been emailing about this on the smallpicture-user list. If you're not up on it, I recommend reading the archive.

https://groups.google.com/forum/?fromgroups#!forum/smallpicture-user

Dave

--
You received this message because you are subscribed to the Google Groups "smallpicture-web" group.
To unsubscribe from this group and stop receiving emails from it, send an email to smallpicture-w...@googlegroups.com.
To post to this group, send email to smallpic...@googlegroups.com.
Visit this group at http://groups.google.com/group/smallpicture-web.
For more options, visit https://groups.google.com/groups/opt_out.

[Ken Smith]

I was about to ask about the 503 errors too. Thanks. 

Ken 

574-520-7381

--

[Dave Winer]

Honestly, I don't think the server is coming back.

I can't update Scripting News, and there's a fair amount of my content that is currently inaccessible.

I'm working on going back to my previous CMS. We needed Kyle, and we don't have him. That's just a fact.

Fargo is still great, and OPML is great, all that's solid. Trex -- not.

Dave

[Dave Winer]

BTW, there is something the community can do, if you want to help figure out what changed.

http://smallpicture.com/userDirectory.opml

That's the list of named outlines. 

A reasonable theory is that one or more of these outlines is causing the problem(s) that's keeping the server from functioning as it used to.

See if you find anything unusual. I don't know what that means, sorry. Perhaps something doing a lot of includes of big outlines. That would cause problems.

If we find something, I can just remove it. That part is easy.

Dave

[Frank McPherson]

I've hacked off the biggest to a Trex blog that I was working on when this all kicked up. It's the only change that I have made.

 

It would see the future of Fargo as highly dependent on interfaces to other applications or services. Thinking of the Wordpress integration as an example.

[Betsy Kimak]

I'm making some progress on the Concord WordPress bridge. It's slow going, mostly time constraints. I'll try to work faster! :)

[Andrew Shell]

I wrote a script that checked each of the urls in that outline to test the status code.  Many of those are returning 404 (technically a 302 to a 404) and I'm not sure how Trex is handling that.  After running my check there were 921 outlines vs 1936 originally. 

My version is here:
https://dl.dropboxusercontent.com/s/fv9gmb03smbuj5t/userDirectory2.opml

Maybe Trex is breaking because this file is getting too large?

Cheers,
Andrew

[Ken Smith]

Dave, if you think there's a chance that one or more outlines have introduced or unveiled a bug, would it make sense to back up the outlines that have been most active in the last few weeks, like mine, and then remove them for a day or two and see if anything changes?

Also, another naive idea from an English professor: I remember in the past that someone altering the create date of a post caused some glitch along the way, I don't remember what, but you dealt with it somehow at the time. A couple of times lately I haven't finished my daily writing until after midnight and I have backdated some content by a few hours--could this send the server on a slow wandering "walk" somehow? (Yes, I motivate myself to have an entry in each box of the calendar...sad to say, perhaps.)

Best wishes...

Ken

[Dave Winer]

Maybe someone else can pitch in here. I don't have any information about which outlines were updated when. But the information is in the files (specifically in the headers that the web server returns). It might be possible to prioritize the investigation into which files are doing expensive things for the server (like lots of includes). It's likely an outline that was updated recently.

The other problem you mention didn't have anything with the server thrashing, and I believe the problem was fixed. 

Dave

[Ken Smith]

There sure are some wacky names on that named outline list. I wonder if some of them are malicious in some way, spam or something, or possibly meant to test the capacity somehow. 

How about removing all the named outlines for one day except scripting, to see how the server goes without them? 

I guess that would not be a clear test since it would both remove potential threats and reduce the general load. But some of the names on that list look dubious to me. (What do I know?)

Or: giving some volunteers a list of things to look for and dividing up the named outline list and inspecting them?

Ken

[Dave Winer]

a list of things to look for and dividing up the named outline list and inspecting them?

I did, in a previous email in this thread.

Dave

[Ken Smith]

I started working down the list of named outlines to see how they worked. Over and over again, if the named outline's Dropbox file ended with "untitled.html" then Dropbox said the file didn't exist. If there was a unique file name for the Dropbox file in the named outline list, then I could almost always open that OPML file, in my sample run down a stretch of the list. To me that suggests that all the named outlines ending in untitled.html could be deleted from the list, about 45% of the list of named outlines. Nearly all of these appear to have bogus names or names in some language that doesn't sound at all familiar, such as duolarabbresp. [Google has never heard of duolarabbresp.]

Would it be of any use to delete these possibly spammy named outlines? Dunno. But they look at best like junk to me. Of course the full list could be saved elsewhere just in case.

Ken

On Thu, Oct 3, 2013 at 9:55 AM, Dave Winer <da...@smallpicture.com> wrote:

[Andrew Shell]

I posted about this earlier and didn't get any response so maybe nobody saw it.  I actually wrote a script that removed all the items that returned 404 and here's the result.

https://dl.dropboxusercontent.com/s/fv9gmb03smbuj5t/userDirectory2.opml

1936 to 921 rows.  This does not contain any named outlines since I downloaded it yesterday.  Since things are broken I think we can safely say that any new ones can be discarded.

Cheers,
Andrew

[Dave Winer]

I saw it and I had already done that and the server runs a check every night and keeps track of how many times each one has failed. 

But it's not the problem, and we all have plenty of things we have to deal with, and have to stay focused.

The reason I'm sure it's not the problem is that there were a lot of 404s in the list last week and the server was running pretty well last week.

What we're looking for is OPML files that have been recently updated that do crazy shit with includes. 

Or something like that. 

Dave

[Andrew Shell]

Gotcha. :)

[Dave Winer]

http://www.youtube.com/watch?v=YOMTnLHDWRA