IF Archive improvements
Andrew Plotkin
have noticed that it's a lot less like wading through frozen yogurt.
We have undertaken three improvements. First, we talked to Carnegie
Mellon about using more bandwidth than we were before. Since the
Archive can plausibly claim to be an academic resource, they were cool
with that.
Second, we added a robots.txt file to exclude robots from crawling the
Archive. (They're permitted to crawl the index files -- that's a much
smaller data set, and has, you know, all the index data which we want
Google to see.) By far the largest load on the Archive was robot
crawlers.
Third, we changed the way we were limiting bandwidth. The Archive
server now limits the number of simultaneous downloads from a single
IP address. This should limit the impact of robot crawlers which are
rude enough to ignore robots.txt.
(In case it's not obvious, we *really* would like people to *not
HTTP-crawl the Archive.* Just don't do it. If you want a complete copy
of the Archive, email us about becoming a mirror -- we'll give you
rsync access in exchange for you helping shoulder the load.)
As for http://mirror.ifarchive.org/, it is set up to distribute
requests among these mirrors:
ifarchive.wurb.com ifarchive.smallwhitehouse.org
ifarchive.flavorplex.com ifarchive.heanet.ie
ifarchive.giga.or.at ifarchive.plover.net
As of today, all of those mirrors are available and seem to be
updating properly.
I will try to be better about modifying the mirror list if mirrors go
down. However, mirrors go down. It happens sometimes. I can't do
anything about it except send plaintive email. (And if a mirror dies,
I'm going to send plaintive email first, so it will be on the order of
days before the redirection script is changed.)
Therefore, the best practice for http://mirror.ifarchive.org/ is to
retry on error. If you're writing a script or application that uses
a mirror URL, be prepared to detect HTTP errors and retry.
--Z
--
"And Aholibamah bare Jeush, and Jaalam, and Korah: these were the borogoves..."
*
"Bush has kept America safe from terrorism since 9/11." Too bad his
job was to keep America safe *on* 9/11.
S. John Ross
> If you have triedhttp://ifarchive.org/in the past few days, you will
> have noticed that it's a lot less like wading through frozen yogurt.
Wow. Yeah, I've never seen it so snappy. Kudos and thanks.
Daphne Brinkerhoff
> If you have triedhttp://ifarchive.org/in the past few days, you will
> have noticed that it's a lot less like wading through frozen yogurt.
I have nothing to add, I just wanted to say: holy crap, that's a huge
improvement. Major thanks!
--
Daphne
Conrad
>
> Therefore, the best practice forhttp://mirror.ifarchive.org/is to
> retry on error. If you're writing a script or application that uses
> a mirror URL, be prepared to detect HTTP errors and retry.
Excellent improvements!
Conrad.
Victor Gijsbers
> If you have tried http://ifarchive.org/ in the past few days, you will
> have noticed that it's a lot less like wading through frozen yogurt.
Wow, that's not just better, that's _massively_ better! Great work!
Regards,
Victor
Jim Aikin
> If you have tried http://ifarchive.org/ in the past few days, you will
> have noticed that it's a lot less like wading through frozen yogurt.
Yowza! Smokin'. Thankyew-thankyew-thankyew.
--JA
Otto Grimwald
> If you have tried http://ifarchive.org/ in the past few days, you will
> have noticed that it's a lot less like wading through frozen yogurt.
>
> We have undertaken three improvements.
Great news!
Especially the fix for broken mirrors, and the bandwith improvement.
1/ As for future possible improvements, I know such an archive is not
supposed to be pretty, but will you also add a nicer (and yet simple) css
theme so it will look a bit more attractive?
2/ It could be interesting also to propose on ifarchive and the mirrors to
play the games in the browser, with zmpp for example. (those which are not
zipped at least)
I've made a page which should work with most of the mirrors (except those
which changed the naming convension of ifarchive), but it's not really
conveniant to find a game in this. It's nice for playing a random game
though:
http://ifiction.free.fr/index.php?id=ifarchive
Some foreign games are also at the end of the list.
3/ Will the babel thing, with direct access, play and / or download from
ifwiki or ifdb, be supported soon?
--
-------------------------
http://ifiction.free.fr
Andrew Plotkin
I guess nobody has tried it in the past few days. :) The change
happened Tuesday, I think.
--Z
--
"And Aholibamah bare Jeush, and Jaalam, and Korah: these were the borogoves..."
*
If the Bush administration hasn't thrown you in military prison without trial,
it's for one reason: they don't feel like it. Not because you're an American.
Andrew Plotkin
> Andrew Plotkin wrote:
>
> > If you have tried http://ifarchive.org/ in the past few days, you will
> > have noticed that it's a lot less like wading through frozen yogurt.
> >
> > We have undertaken three improvements.
>
> Great news!
> Especially the fix for broken mirrors
(Cough) I didn't do anything there. The flavorplex mirror came back
online, thus making the list we had correct again.
> 1/ As for future possible improvements, I know such an archive is not
> supposed to be pretty, but will you also add a nicer (and yet simple) css
> theme so it will look a bit more attractive?
We could. Have one in mind? I've done CSS web design before, but I
don't know if anyone has called the result "attractive".
I generally take the tack that a window full of text in your browser's
preferred font and color is never *wrong*...
(I did finally validate the HTML on the front page. The index pages
still have errors but they'll be fixed tomorrow, after they
regenerate.)
> 2/ It could be interesting also to propose on ifarchive and the mirrors to
> play the games in the browser, with zmpp for example. (those which are not
> zipped at least)
I think we want to stick to "storage" as a service goal. As I've said
before, the Archive will never be a great browsing experience, and
nice game presentation without browsing is kind of pointless. This is
the kind of thing that can be done just as well by an external site.
> 3/ Will the babel thing, with direct access, play and / or download from
> ifwiki or ifdb, be supported soon?
Ah... no. No motion in that direction, I'm afraid.
--Z
--
"And Aholibamah bare Jeush, and Jaalam, and Korah: these were the borogoves..."
*
RootShell (IFReviews.org)
What is the "normal" monthly bandwidth values? what is the full size of
the archive?
I have a HUGE space/bandwidth hosting server plan, and wouldn't mind
taking part in having a mirror.
Let me know the details when possible.
Kind regards,
RootShell
Andrew Plotkin
> What would be the specs to become a FTP/HTTP mirror?
Have an HTTP server with a static IP address. Be able to set up a
domain called "ifarchive.***". Be able to run some rsync commands once
per day, at a particular time.
> What is the "normal" monthly bandwidth values?
For mirrors, I don't actually know. Anyone running a mirror want to
chime in?
> what is the full size of the archive?
A bit under 3 gigabytes right now.
Send email to <tech .at. ifarchive .dot. org>, and let us know the
domain name you're going to use and the IP address of the server that
will be rsyncing.
--Z
--
"And Aholibamah bare Jeush, and Jaalam, and Korah: these were the borogoves..."
*
If the Bush administration hasn't shipped you to Syria for interrogation,
it's for one reason: they don't feel like it. Not because you're patriotic.
Ron Newcomb
Obligatory:
Well I for one welcomed our IF-browsing robotic overlords.
And a big capitalized Thank You to you Andrew. (You'll serve the
hivemind well one day.)
-Ron
JDC
> Obligatory:
> Well I for one welcomed our IF-browsing robotic overlords.
In Soviet Russia, IF archives you...
-JDC
Ron Newcomb
> > Obligatory:
> > Well I for one welcomed our IF-browsing robotic overlords.
> In Soviet Russia, IF archives you...
I guess that answers that question: "What does -too- interactive
mean?"
-R
Sabahattin Gucukoglu
I actually *prefer* to browse the archive offline in my own
filesystems with my own tools and have no problems about breathing
second-hand archive (I can appreciate that bandwidth from the primary
site is probably better reserved for mirrors). It isn't that the web
pages are bad or anything, just that I like spontaneous access to
stuff. Consequently, over here in Blighty at least, I'm using rsync
from ifarchive.heanet.ie to keep the network file server updated for
the benefit of my other boxes. Since the archive isn't large at all,
that works great for me.
Just in case anyone else wants it, the path is mirrors/ifarchive.org
at ifarchive.heanet.ie.
Rsync is very cool.
Cheers,
Sabahattin