open-tran.eu server errors?

[Alexander Dupuy]

I am getting:

500 Internal Server Error
Server got itself in trouble

on all open-tran.eu lookups; tested with en.es and en.fr - seems to
affect all lookups. Is there perhaps a problem accessing the database?

@alex

--
mailto:alex....@mac.com

[Jacek Śliwerski]

Alexander Dupuy pisze:

>
> 500 Internal Server Error
> Server got itself in trouble

Thanks for letting me know!

The database locked up. I have no idea why. I have restarted the
server and it is working fine now. I am monitoring the situation.

Jacek

[Joergen]

Hi Jacek

"Server got itself in trouble" again.

[Jacek Śliwerski]

Joergen pisze:

>
> "Server got itself in trouble" again.

The server doesn't handle the load. I browsed through the logs. It
starts with
OSError: [Errno 12] Cannot allocate memory
which is followed by
OperationalError: database is locked
soon thereafter.

Last night, everything came back to normal as soon as the load went
down. I have to say that it is pretty surprising to me, because the
statistics haven't been showing any spike. See Google Analytics stats
here: http://picasaweb.google.com/sliwers/ZrzutyEkranu#5432480730245414754

However, API usage has risen... a lot. In the last four days, open-tran
served:
* 46 GET requests for /compare
* 52 GET requests for /words
* 1686 GET requests for /suggest
* 17651 API calls

I can find only one explanation for this - one of the popular tools has
turned open-tran support as a default option. I don't blame anybody. I
am actually pretty proud that the service gets the momentum. Only, I
wish I had better server for that :)

So here is my plan. As a short-term solution, I will introduce a limit
to the service. Only one API request per client at a time will be
allowed. Concurrent requests from the same client will get an error
response.

But it is not a sustainable long-term strategy. I am afraid that I will
have to cut down the size of the database somehow. I hope to be able to
remove some of the phrases that are simply impossible to ever come up in
a search result. I will analyze the database and see if there are any
ways of reducing the query times and will let you know.

Has anybody any ideas on how to improve the performance? I would be
very glad to hear your ideas.

Thanks!
Jacek

[Alexander Dupuy]

Jacek Śliwerski wrote:

> However, API usage has risen... a lot. In the last four days,
> open-tran served:
> * 46 GET requests for /compare
> * 52 GET requests for /words
> * 1686 GET requests for /suggest
> * 17651 API calls
>
> I can find only one explanation for this - one of the popular tools
> has turned open-tran support as a default option. I don't blame
> anybody. I am actually pretty proud that the service gets the
> momentum. Only, I wish I had better server for that :)

Another free web service that provides a programmatic API
(musicbrainz.org) has an API argument which is required to be the client
application name and version, both to allow tracking this sort of thing,
and to allow for blocking access to a particular application, if it gets
"too popular" for the server - and in fact they used that capability
recently, when the free Pollux application took off and started
hammering their servers.

While it is a bit "too late" to do this with the current OpenTran API,
you might well want to include it in a new API, and then indicate that
the old one will be phased out and dropped at some future point in time.

All other things being equal, though, I suspect that the API hits may be
coming from Virtaal...

@alex

--
mailto:alex....@mac.com

[Jacek Śliwerski]

Alexander Dupuy pisze:

>
> Another free web service that provides a programmatic API
> (musicbrainz.org) has an API argument which is required to be the client
> application name and version, both to allow tracking this sort of thing,
> and to allow for blocking access to a particular application, if it gets
> "too popular" for the server - and in fact they used that capability
> recently, when the free Pollux application took off and started
> hammering their servers.

First thing that occurred to me after reading this paragraph was: I am
sooooo lame. It's such an obvious trick.... once you know it.

> All other things being equal, though, I suspect that the API hits may be
> coming from Virtaal...

I thought that it was Virtaal too. But the newest version has been
pushed neither to Ubuntu, nor to Fedora yet.

Besides, if it were Virtaal, the requests would probably have been
mostly for English phrases. And I am seeing lots of other phrases -
Greek, Russian, Spanish, Italian.... strange.

Jacek