Downloads Stats Problems

[Nick Quaranto]

So, our download counts are off. There's something wrong with the cron job and it won't log *any* output as to why it's failing. Currently we have:

[rubycentral@255155-db1 ~]$ crontab -l# Gemcutter cron jobs

# http://cronwtf.github.com/

# Daily download rotation

0 19 * * * cd /var/www/rubycentral/gemcutter.org/current && /usr/bin/env RAILS_ENV=production HOME=/home/rubycentral RAILS_ENV=production bundle exec rake daily_cron --trace >> /var/www/rubycentral/gemcutter.org/current/log/daily_cron.log 2>&1

# Weekly legacy index rotation - running it at 7 AM in case something goes wrong

7 19 * * 0 cd /var/www/rubycentral/gemcutter.org/current && /usr/bin/env RAILS_ENV=production HOME=/home/rubycentral RAILS_ENV=production bundle exec rake weekly_cron --trace  >> /var/www/rubycentral/gemcutter.org/current/log/weekly_cron.log 2>&1

And neither spit out anything in the log. 

Right now the error I'm getting is:

[rubycentral@255155-db1 ~]$ cd /var/www/rubycentral/gemcutter.org/current && /usr/bin/env RAILS_ENV=production HOME=/home/rubycentral RAILS_ENV=production bundle exec rake daily_cron --trace

Using /usr/local/rvm/gems/ruby-1.9.3-p0

** Invoke daily_cron (first_time)

** Invoke gemcutter:downloads:rollover (first_time)

** Invoke environment (first_time)

** Execute environment

** Execute gemcutter:downloads:rollover

rake aborted!

stack level too deep

I vote we do a few things:

1) I have no idea why this is happening yet. If you have some time, a recent Redis dump is here:

http://static.rubycentral.org/dump.rdb.gz

Basically, just try to set up the app and run the rake task, see why this is happening:

https://github.com/rubygems/rubygems.org/wiki/Development-Setup

2) After that, move to http://github.com/javan/whenever or something else that makes this easier to manage.

3) Finally, move the Redis keys to something more robust so this doesn't happen anymore. Right now there's only 2 keys: "yesterday" and "today". We should just do daily keys, or maybe look into a different system of processing that won't fail terribly like this.

Basically these are going to take several hours, and I'm wiped out right now. I can probably tackle it tomorrow but no guarantees (and there's support issues that need urgent attention too). 

-Nick

[Nick Quaranto]

Ok, so the cron job was failing due to a Ruby bug!

http://redmine.ruby-lang.org/issues/5719

Worked around it and deployed, so tomorrow our counts should roll over ok. I'm moving the download counts for yesterday over, but right now we've lost the older data, unless Tom can restore backups from Rackspace (not sure if this is possible or not).

So we still need to:

1) Move to a better system so this wont happen again. Basically, we need to just use HINCRBY when the download happens for the daily counts (it'll be O(1) as well so impact should be minimal: http://redis.io/commands/hincrby), and the cron job can update the Versions table on a nightly basis (or heck, hourly, whatever) so searches can work still.

2) Possibly move off cron to whenever? Not sure what is best here.

-Nick

[Kieran P]

Quick comment: "Possibly move off cron to whenever?" sounds like you think their different. Whenever is just a Ruby DSL which compiles to crontab compatible lines. You'd be able to commit the file to gemcutter repo and automatically update your crontabs at deploy time (which is helpful). I recommend it, but it's not different from cron.

[Travis Reeder]

How about using IronWorker scheduler to replace Cron?  http://docs.iron.io/worker/ruby/scheduling

Cloud based so you'd never have to worry about it again. I'd be happy to help make this happen.

Disclaimer: Co-founder of Iron.io

Cheers,

Travis

[Nick Quaranto]

Apparently the cronjob is still not working. Not sure why yet, haven't had a chance to look into it. Ugh.

[Nick Quaranto]

Ok, here's my plan going forward. Given the following:

1) Cronjobs and more server maintenance sucks.

2) All gemcutter really cares about is real time data, and how many downloads exist for a given gem/version.

3) The current graphs (on profiles and on stats/show) are really unpleasant and not too useful.

So, I'm going to remove historical downloads on the site.

In place of this, we'll run a publicly facing Redis instance that will use Pub/Sub and allow anyone out there to build out their own downloads stuff with our data. Redis allows for disabling/renaming of commands so I think it will be safe to publicly expose this.

Basically, it'll work something like:

   telnet downloads.rubygems.org 6380

   SUBSCRIBE downloads:v1

and messages could come in as JSON, like so:

   {rubygem: "rails", version: "3.0.0"}

I'm not sure what other information would be useful here, I'm all ears though. Any thoughts?

-Nick

[Luis Lavena]

On Dec 15, 12:29 am, Nick Quaranto <n...@quaran.to> wrote:
>
> Basically, it'll work something like:
>
>    telnet downloads.rubygems.org 6380
>    SUBSCRIBE downloads:v1
>
> and messages could come in as JSON, like so:
>
>    {rubygem: "rails", version: "3.0.0"}
>
> I'm not sure what other information would be useful here, I'm all ears
> though. Any thoughts?
>

From which version of Ruby/Platform/RubyGems the version was
downloaded?

This is part of the user agent set by the remote fetcher.

Getting that info too will be great.

--
Luis Lavena

[Nick Quaranto]

Ooh, good idea. We can definitely publish that too.

[7rans]

That seems a rather unfortunate way to go.

Why can't each gem record store a hash of `date=>count`?

[7rans]

I was thinking, over time a lot of versions of gems are going to build-up in the data stores that are no longer useful. It occurs to me that some resources might be freed up if gemcutter supported a command, kind of like `yank`, but meaning `no longer supported`.

[wbr]

I hate to be a nag about this, but can we at least get some sort of
short-term resolution? Since the data is inaccurate, I would suggest
removing all statistical reports from the web site.

[Nick Quaranto]

I've been really busy with the holidays. Also, there's a massive support queue of issues that I cleared this weekend. There's an open pull request that I haven't had a chance to look at yet. Not sure what else to say here other than I'm strapped for time, and I'll get to it when I can.

Also, after talking to one of the Redis maintainers they said the public Redis API is a bad idea, basically Redis assumes clients will read off the socket, and in the Real World ™ that doesn't always happen.

[rubiii]

hey guys,

any news on this topic? i just look at some stats via the rubygems api and the latest daily downloads are from 2013-03-08.

not sure why there's only 30 days of data, but i would really like to work with this stuff. did i miss something or can i do anything to help you out?

cheers,

daniel

[Evan Phoenix]

The download stats need to be entirely reworked. Until thats done, they'll only be kept in small windows.

-- 

Evan Phoenix // ev...@phx.io

--
You received this message because you are subscribed to the Google Groups "rubygems.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rubygems-org...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[rubiii]

well, as i said, let me know if i can do anything to help.

cheers,

daniel