Group renamed and October pow-wow

[Nick Quaranto]

I've renamed the group! Huzzah! Not sure how I forgot about this before.

We're going to have a Google Hangout on Friday (10/19) to discuss current issues at 9PM EST. The link will be posted on this thread once we've kicked it off, and in the #rubygems IRC channel.

Issues to cover:

* Current status of why the server is unhappy

* Progress and update on moving infrastructure

* Fixing our stats pages and counting downloading

* Improving the Bundler API's performance

[Nick Quaranto]

Side note: I'll try to hook up YouTube streaming for those looking to
just watch. It would be great if someone could take notes and push
them here or to the GitHub wiki, as well.

[Evan Phoenix]

Just put the site in maint mode by doing the following:

cd /etc/httpd/conf

vim httpd.conf

comment out "SetEnv RAILS_ENV production"

uncomment "SetEnv RAILS_ENV maintenance"

run "sudo /usr/sbin/apachectl restart"

-- 
Evan Phoenix // ev...@phx.io

On Wednesday, October 17, 2012 at 10:59 AM, Erik Hollensbe wrote:

Already discussed in IRC, but consolidating the convo, these details before the hangout would really help:

- Database configuration (Redis, Postgres, etc)

- Network map (just a description will work)

- Machine configurations (cpu/ram/disk are crucial)

- Details of any custom software (save the rubygems.org source) critical to the infra (descriptions and/or code)

- Listing of known pain points

  - Redis disk space has already been detailed by evan (AOF growing out of control)

Monitoring and charting info would also be nice if you have that easily available, but if not it's not critical for this conversation I don't think.

Be well.

-Erik

[Nick Quaranto]

I've gotten no less than 3 responses saying this is a bad time. Perhaps we aught to move it! Vote for when you can make it, please:

http://doodle.com/hd7h8ep6psrz7bsp

-Nick

On Wednesday, October 17, 2012 at 3:08 PM, de wrote:

You're going to have a discussion about related issues on Friday at 9pm... what? I'm happy to pick up the torch too, but I'd suggest a different day - time.

[Trotter Cashion]

Those times are PT or ET?

- Trotter

[Nick Quaranto]

EST. I have noted this in ALL CAPS now. The Doodle poll is also time-zone aware, or so it told me.

[Trotter Cashion]

So it is, that's fancy.

[Erik Hollensbe]

Hey guys, not to be a pest, but any chance we can get some follow up on these questions? I know everyone's busy, a novel isn't required but some basic description of the systems that are in play would be really useful for saving on repeating ourselves during the hangout.

-- 

Erik Hollensbe

Sent with Sparrow

[Evan Phoenix]

See below inline.

-- 
Evan Phoenix // ev...@phx.io

On Friday, October 19, 2012 at 9:53 AM, Erik Hollensbe wrote:

- Database configuration (Redis, Postgres, etc)

Redis 2.4 and Postgres 8.4 on the same machine as the main app. 

- Network map (just a description will work)

lb1 (containing f1 and f2) -> b1 (where the app is)

f1 and f2 server the redirects and proxy to b1 for the app. 

- Machine configurations (cpu/ram/disk are crucial)

lb1 is a quad-core 1.9Ghz AMD, 4G of ram.

f1 and f2 both are 2 cores and 2G of ram.

b1 is a quad-core 2.3Ghz Xeon, 16G of ram. 

- Details of any custom software (save the rubygems.org source) critical to the infra (descriptions and/or code)

Nginx is used for the frontends, passenger currently handles the app on b1. stat-update handles incrementing the download counters. 

- Listing of known pain points
  - Redis disk space has already been detailed by evan (AOF growing out of control)

Download stats go straight into redis (and are basically constantly being hit) cause a persistent write load due to the AOF file. This also inflates the AOF file overtime.

Daily download stats are in redis and a huge hashes, eating up heap space for rarely seen stats.

The bundler-api requires building a giant ruby object tree and Marshaling it, basically making it CPU bound. It's common to see 500k responses and this only gets worse over time because the API always returns all versions of the requested gems.

Monitoring and charting info would also be nice if you have that easily available, but if not it's not critical for this conversation I don't think.

We use pingdom, stillalive, and newrelic.

[Erik Hollensbe]

Awesome. Thanks Evan.

-- 

Erik Hollensbe

Sent with Sparrow

[Nick Quaranto]

I'm going to go ahead and say the meeting will still be at 9PM EST
tonight. 13 have +1'd tonight. Next time (when it's not such an
emergency setting) we'll follow up with the Doodle link first.

[Erik Hollensbe]

13 people will be too many for a single hangout. Not quite sure how to handle this, but I figured you should be aware of it.

-- 

Erik Hollensbe

Sent with Sparrow

[Kristopher Murata]

Try to broadcast it using youtube: http://www.google.com/+/learnmore/hangouts/onair.html

I don't want to participate, but will be nice to listen to the discussion and share a few words over IRC or even Google Hangout chat.

[Trotter Cashion]

Yea, I'm happy to just listen in at this point. I don't feel that I
_really_ have enough knowledge to contribute yet.

[Nick Quaranto]

We're going to hook that up! If you'd just like to listen in you're more than welcome to.

[Kristopher Murata]

Thanks Nick! I'm on the same situation as Trotter, I want to help, but I don't think I'm ready to contribute yet. I just checked out rubygems.org codebase yesterday and was giving it a try.

[Christopher Meiklejohn]

My $0.02:

There are plenty of open pull requests and issues that need a tiny bit of extra work to get merged in; while not all directly related to the performance issues, it's a good place to start to learn how the codebase works.

I'm more than happy to Skype/Hangout or whatever with anyone who is interested and provide any context or help.

- Chris

-- 

Christopher Meiklejohn

[Karel Minařík]

Nick, Evan -- could you also discuss what to do about the elasticsearch search implementation? [1]

Thanks!,

Karel

[1] https://github.com/rubygems/rubygems.org/pull/455

[Nick Quaranto]

Karel,

I think the focus of this meeting is going to be getting the infrastructure stuff figured out, and making sure our server(s) aren't on fire constantly. Once we handle that, we can look into getting bigger features like this one merged in.

-Nick

[Karel Minařík]

Hi Nick,

I think the focus of this meeting is going to be getting the infrastructure stuff figured out, and making sure our server(s) aren't on fire constantly. Once we handle that, we can look into getting bigger features like this one merged in.

I understand; the only reason I pinged about that was that the search feature is part of of the infrastructure discussion -- how the search server(s) will be deployed, managed, monitored, etc.

Thanks!,

Karel

[Nick Quaranto]

I agree that the search infrastructure should be part of the consideration going forward…we should keep it in mind for tonight's discussion. Thanks for keeping us in check, Karel.

-Nick

[Christopher Meiklejohn]

I reviewed that pull and tested the branch; looks good, but I held back on pushing it forward until we had successfully completed the server migration.

Sent from my iPhone

[Karel Minařík]

I reviewed that pull and tested the branch; looks good, but I held back on pushing it forward until we had successfully completed the server migration.

Absolutely -- we need to figure out how to properly set it up for production. I'm at hand at all times to help with that.

Karel 

[Christopher Meiklejohn]

Sounds good && thanks.

Sent from my iPhone

[Evan Phoenix]

Hi Karel,

I absolutely want to talk with you about this and I'm sorry I haven't gotten back to you. I had planned to get this going as soon as we migrated to rackspace cloud, but that ended up blowing up in my face. I promise we'll get back to it soon.

-- 
Evan Phoenix // ev...@phx.io

[Karel Minařík]

Hi Evan,

 

I absolutely want to talk with you about this and I'm sorry I haven't gotten back to you. I had planned to get this going as soon as we migrated to rackspace cloud, but that ended up blowing up in my face. I promise we'll get back to it soon.

no worries! I understand it's hard to hit all those things at once. Once it settles a bit, let's get back to it, I'll be here to help with everything.

Best!,

Karel

[Joseph Caudle]

Is the video (or IRC transcript) going to be available after the Hangout? I'd like to be able to help out, but won't be around this evening. 

Joseph

--

Joseph Caudle

Sedes Sapientiae, ora pro nobis.

[Chris Kelly]

I was going to join to see what resources (people or otherwise) New Relic could provide, but a flight delay has me landing about when you're getting started. 

Can the recording be posted after the hangout is over? I'll get online as soon as I can. 

[Nick Quaranto]

Here's the YouTube link!

http://youtu.be/z73uiWKdJhw

If you want to be in the hangout, come into #rubygems on Freenode and
I can invite you.

On Oct 17, 1:58 pm, Nick Quaranto <n...@quaran.to> wrote:
> Side note: I'll try to hook up YouTube streaming for those looking to
> just watch. It would be great if someone could take notes and push
> them here or to the GitHub wiki, as well.

[Nick Quaranto]

Here's the YouTube video of the hangout (I thought I had posted this
as we were getting set up...sorry if I excluded anyone):

http://www.youtube.com/watch?v=z73uiWKdJhw

And a transcript, thanks to Wes Garrison!

https://gist.github.com/3921560

Gotta say, we need to do these more often. G+ is awesome.

-Nick