show and tell?

54 views
Skip to first unread message

Matthew Lyon

unread,
Jan 8, 2013, 1:51:07 PM1/8/13
to pdx-visu...@googlegroups.com
Hi group, long-time lurker first-time poster.

As part of my work at AppFog, I've built out a graphing system that I think is pretty cool. It provides an aggregated view of the characteristics of certain types of events, and lets you drill down by facets of the events.

For example, this is from a report of our API, showing incoming HTTP requests and below that, the "dispatched" requests to the datacenters we provide hosting in around the world (I'm on an 11" MacBook, can't make larger screenshots):


The top half of each graph shows three things: in blue, the "onset" or number of new events in a 15 second period, in grey, a histogram of the number "concurrent" events over the interval that are open but haven't completed, for any given set of unique facets (useful for finding resource leaks or unclosed connections), and in red the number of events that completed with an error of some kind.

The bottom half of the graph shows a histogram of the time taken to complete events over the interval given. These graphs use a 15-second-per-pixel interval.

Since I see that around 8:50 the dispatch was slow, I can drill down by "target" and see that the problem lies with the AWS Singapore region:


I've thrown this type of measurement on everything I can think of: calls between different parts of our SOA, database / memcache, workers for various parts of our system, and have started using it for error reporting as well. It's proven invaluable in hunting down problems with our code, with our network operation and configuration, and has given some interesting insight into what happens during an AWS outage.

The data is gathered by a very simple nodejs app, stored in postgresql, and the graphs are rendered client-side using the canvas API from the results of a SQL query. Currently we're trimming the data to the past three hours, but are working to make it more archival.

I plan to open source this system soon, but would love to show it to an interested audience and get some feedback first. Is this something that would interest the group for a future meetup?

Thanks

Matthew Lyon
@mattly / 503-927-3357
Sr. Software Architect, AppFog

Jesse Cooke

unread,
Jan 8, 2013, 1:59:56 PM1/8/13
to pdx-visu...@googlegroups.com
I would love to hear more about this!


--------------------------------------------
Jesse Cooke :: N-tier Engineer
jc00ke.com / @jc00ke


--
You received this message because you are subscribed to the Google Groups "pdx-visualization" group.
To post to this group, send email to pdx-visu...@googlegroups.com.
To unsubscribe from this group, send email to pdx-visualizat...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pdx-visualization?hl=en.

Ryan Miller

unread,
Jan 8, 2013, 2:01:37 PM1/8/13
to pdx-visu...@googlegroups.com
me too.  

I might also have something to show off, depending on the meeting date.

Amber Case

unread,
Jan 8, 2013, 2:19:14 PM1/8/13
to pdx-visu...@googlegroups.com
Awesome! If we can lock in one more speaker, it'd be great to have another meetup.

Kim Rees

unread,
Jan 8, 2013, 2:43:13 PM1/8/13
to pdx-visu...@googlegroups.com

Hi all,

 

Coincidentally, we are in the process of securing a meetup space. We’re hoping to get this on the docket for the end of the month.

 

Amber, were there other potential presenters who contacted you personally?

 

Kim

 

Periscopic :: Do good with data

www.periscopic.com

503-295-7946

@krees, @periscopic

Amber Case

unread,
Jan 8, 2013, 6:08:07 PM1/8/13
to pdx-visu...@googlegroups.com
Hi, Kim! None to my knowledge, although I just sent out a tweet. Feel free to organize the event/set the date, and I'll let you know if anyone else pops up!

Ryan Miller

unread,
Jan 8, 2013, 7:20:35 PM1/8/13
to pdx-visu...@googlegroups.com
Kim,  Will we see the latest from Periscopic?  It doesn't seem like a real PDX Data Viz meetup without some new work from you guys :-)

Amber Case

unread,
Jan 8, 2013, 7:47:49 PM1/8/13
to pdx-visu...@googlegroups.com, pdx-visu...@googlegroups.com
Agreed! If so, that's 3 presenters! 


Best, 
Amber 

--
Amber Case, Director
Esri R&D Center, Portland
920 SW 3rd Ave. #400
---
--

M. Edward (Ed) Borasky

unread,
Jan 17, 2013, 4:04:09 AM1/17/13
to pdx-visu...@googlegroups.com
Did we get a data/venue yet? A couple of people asked me about this at
the #ONAPDX meeting tonight.
Twitter: http://twitter.com/znmeb; Computational Journalism Publishers
Workbench: http://znmeb.github.com/Computational-Journalism-Publishers-Workbench/

How the Hell can the lion sleep with all those people singing "A weem
oh way!" at the top of their lungs?

Kim Rees

unread,
Jan 17, 2013, 4:39:55 AM1/17/13
to pdx-visu...@googlegroups.com
We're are still settling on a venue. As for dates, what day the week of Feb 4th looks good for people?

Kim

Mike Lonergan

unread,
Jan 17, 2013, 12:00:58 PM1/17/13
to pdx-visu...@googlegroups.com, pdx-visu...@googlegroups.com
Monday or Tuesday would be great - Agile Open Northwest conference is Wed-Fri that week.

Mike

Ryan Miller

unread,
Jan 17, 2013, 12:23:51 PM1/17/13
to pdx-visu...@googlegroups.com
I concur on Monday - Tuesday.  PDX-UX is having Noah Pepper of Lucky Sort present on the 7th.  Lots of folks here should be interested in that.

Amber Case

unread,
Jan 17, 2013, 12:49:44 PM1/17/13
to pdx-visu...@googlegroups.com
Agreed.

Ryan, do you know if Thetus might be up for hosting? Geoloqi no longer has room as our team has grown to fill in a lot of the empty floor space, and we no longer have access to chairs.

Urban Airship might be another option. Barb Stark is the contact there.

Christopher Rivard

unread,
Jan 17, 2013, 1:34:55 PM1/17/13
to pdx-visu...@googlegroups.com
Jive is willing to host again - although the acoustics in the 5th floor room are not stellar and the training room we were in last time we hosted was a little small. Let me know.

What about Elemental?

Amber Case

unread,
Jan 17, 2013, 1:40:57 PM1/17/13
to pdx-visu...@googlegroups.com
Chris,

Last time I was at Jive it was 3 years ago. Is there still a main room that can hold about 70 people? Is there a way to get a PA system in there?

Has anyone been to an event at the new Elemental HQ? Sounds like a great place, and Bailey's is below, which is good for drinks after.

Christopher Rivard

unread,
Jan 17, 2013, 2:01:39 PM1/17/13
to pdx-visu...@googlegroups.com
I attended a Refresh Portland event (w/ Sheetal Dube) at Jive before I worked here (about 3 years ago). That was held on the 3rd floor - I think that's the space you recall. That space is now filled with desks. 

The large rooms we have available are a training room (we held the meeting in that room when we hosted) and the 5th floor cafeteria area. We did drinks/networking on 5 and then went down to the training room on 4. 

You're happy to stop by and take a look if you want. 

I'll ping Tom Quick at Elemental and ask if they are interested and have space.

Jesse Cooke

unread,
Jan 17, 2013, 2:06:17 PM1/17/13
to pdx-visu...@googlegroups.com
On Thu, Jan 17, 2013 at 10:40 AM, Amber Case <caseo...@gmail.com> wrote:
Chris,

Last time I was at Jive it was 3 years ago. Is there still a main room that can hold about 70 people? Is there a way to get a PA system in there?

Has anyone been to an event at the new Elemental HQ? Sounds like a great place, and Bailey's is below, which is good for drinks after.
I went to a Ruby meeting at Elemental last year. It was a bit small for the unexpectedly large number of attendees we had that night, but it worked out really well.
Plus I think they live-streamed it ;-)

Ryan Miller

unread,
Jan 17, 2013, 2:08:53 PM1/17/13
to pdx-visu...@googlegroups.com
The training room at Jive was pretty crowded and the cafeteria space didn't seem like it would be the right shape for a presentation.  Urban Airship has a great space (or did last time I was there about 5 months ago).

I'll be at Thetus tonight for PDX-UX and I'll check on that.  I don't work there anymore so I need to find someone who does willing to stick around for the whole meeting.

Davy Stevenson

unread,
Jan 17, 2013, 2:15:19 PM1/17/13
to pdx-visu...@googlegroups.com
Hey Everyone, Davy Stevenson from Elemental piping in. We would definitely be interested in hosting a meeting for you guys. Our largest conference room isn't the biggest around, but it can hold a decent amount of people (I think the count for the Ruby Brigade meeting was ~70 and that was doable) and we have a secondary monitor to help make sure everyone can see the presentation. Our space is also amplified, and we have a sweet recording system set up with capabilities to live-stream, or at least to record the presentations for archiving and later distribution.

M. Edward (Ed) Borasky

unread,
Jan 17, 2013, 2:53:35 PM1/17/13
to pdx-visu...@googlegroups.com
On Thu, Jan 17, 2013 at 11:08 AM, Ryan Miller <ninjas...@gmail.com> wrote:
> The training room at Jive was pretty crowded and the cafeteria space didn't
> seem like it would be the right shape for a presentation. Urban Airship has
> a great space (or did last time I was there about 5 months ago).

Yes - and the Data Science group meets there, so there are some
overlaps in people. I don't know who the visualization pros at Urban
Airship are, however.

Amber Case

unread,
Jan 17, 2013, 3:16:03 PM1/17/13
to pdx-visu...@googlegroups.com
Just asked Barb Stark at UA. Will let you know if we can get it! If not, Thetus would be a great backup if we can get it. 

Kim Rees

unread,
Jan 17, 2013, 3:50:05 PM1/17/13
to pdx-visu...@googlegroups.com

Let’s do it!  I like the idea of LiveStreaming. I know Amber did that for one of the events and it was very popular.

 

Does the evening of Feb 5th work for you all at Elemental? Once we set a date, I’ll put out a Meetup notice.

 

Kim

Amber Case

unread,
Jan 17, 2013, 3:55:22 PM1/17/13
to pdx-visu...@googlegroups.com
Awesome! Sounds fantastic!

Davy Stevenson

unread,
Jan 17, 2013, 4:05:34 PM1/17/13
to pdx-visu...@googlegroups.com
Yes, Feb 5th will work for us. I'll unfortunately miss things due to being on vacation, but there are plenty of other people here who would be interested in being host for this event. One question for the group -- Elemental could provide some refreshments if that is a standard thing people would like. Beer would be easy to provide.

Davy

Davy Stevenson

unread,
Jan 17, 2013, 4:57:58 PM1/17/13
to pdx-visu...@googlegroups.com
Yes, Feb 5th definitely works for Elemental. Unfortunately I'll be gone on vacation on that date, but there will be no problem finding an interested host here. I could also plan to have Elemental provide refreshments (beer?) if that's something the group is interested in.

Davy

* Trying again with my other email -- hopefully I don't spam the group twice!


On Thursday, January 17, 2013 12:55:22 PM UTC-8, caseorganic wrote:

Jesse Cooke

unread,
Jan 17, 2013, 6:05:37 PM1/17/13
to pdx-visu...@googlegroups.com
Looks like Davy isn't able to join this group, or her posts are getting flagged or something.
Can someone with SUPER COW POWERS approve her if necessary?

Matthew Lyon

unread,
Jan 17, 2013, 4:00:56 PM1/17/13
to pdx-visu...@googlegroups.com
Ack, I can't do Tuesday nights. I'd be happy to do something on my event graphs another time, but Tuesday nights are out for me until March.

--
Matthew Lyon
503.927.3357

Kim Rees

unread,
Jan 18, 2013, 4:32:37 AM1/18/13
to pdx-visu...@googlegroups.com

How about the following week? Does Feb 13th work?

 

Davy, what other dates work for you and Elemental?

Davy Stevenson

unread,
Jan 18, 2013, 2:12:32 PM1/18/13
to pdx-visu...@googlegroups.com
Feb 13th would work for us as well. And I'd be available that day too.

Ryan Miller

unread,
Jan 18, 2013, 2:43:18 PM1/18/13
to pdx-visu...@googlegroups.com
I would love some extra time to prep my stuff.  Matthew gave me a preview of his work last night at PDX-UX.  We don't want to miss it!  good stuff

Davy Stevenson

unread,
Jan 18, 2013, 8:28:40 PM1/18/13
to pdx-visu...@googlegroups.com
I'm so sorry guys, Feb 13th actually doesn't work for Elemental, we are hosting the PDX Cloud group that night. It's on Calagator but wasn't placed on our company calendar yet. We could handle Feb 11th or 12th (or pretty much any other night).

Davy

Amber Case

unread,
Jan 18, 2013, 8:49:46 PM1/18/13
to pdx-visu...@googlegroups.com, pdx-visu...@googlegroups.com
Urban Airship can host whenever. Let me know the date and Barb can put it on the calendar! 

---
Amber Case
Director, Esri R&D Center, Portland 920 SW 3rd Ave Suite 400 | Portland, OR 97204
---
e: ac...@esri.com    |   m: 1-503-729-8632  |   c: 1-971-275-6757
w: http://caseorganic.com/blog tw: @caseorganic
---

M. Edward (Ed) Borasky

unread,
Jan 18, 2013, 10:34:26 PM1/18/13
to pdx-visu...@googlegroups.com
Excellent!

Matthew Lyon

unread,
Jan 19, 2013, 10:28:50 AM1/19/13
to pdx-visu...@googlegroups.com
The 11th or the 13th work for me.

--
Matthew Lyon
503.927.3357

Reply all
Reply to author
Forward
0 new messages