How to get started with data visualization?

[Bryan Berry]

As a sysadmin and quasi-developer, I have always worked in offices where solving performance issues could not be solved by plucking "low-hanging fruit." However, I hope in the future that I will be able to be better generate and analyze performance data to find the non-obvious causes of slowdown and their solutions. 

Can anyone recommend a general guide to generating and visualizing time series data? On the web I find scattered tutorials but nothing comprehensive. I actually did take 2 statistics courses 16 years ago but that knowledge has rusted away. 

Cheers,

BryanWB

[ranjib dey]

What kind of performance improvements you are looking for? I generally follow this:

1) Have monitoring , and generate enough metrics that capture your app, service integration points and infrastructure. Model any information which you think can be use full as a metric. And monitor it.

2) Feed those metrics to graphite (or if your monitoring system has in built charting system use that, like pnp4nagios in case of nagios)

3) Collate these charts against central log management system and CI system to get a holistic view (like which commit cost how much response time, or what error in app caused a cpu spike etc)

4) Use R for anything that requires most sophisticated or thorough analysis beyond this

regards

ranjib

[Elliot James Murphy]

The book Visualize This is an excellent intro to practical data viz, it shows how to get started with many different tools. I've seen several sys admins and developers do nice work with analysis based on what they learned from the book.

http://book.flowingdata.com/

--
elliot murphy

[Guess Who?]

I suggest  you pick up the book by Edward Tufte: The Visual Display of Quantitative Information.  One of the best, if not the best, on the subject.

Chuck Wegrzyn

[Scott McCarty]

Agreed, great book. Actually, there are at least three. I borrowed them from a friend once. Attempting to buy them on Amazon right since this reminded me :-)

Best Regards
Scott M

[Guess Who?]

The three are worth the money. He used to teach a one day class on the
subject, and it too was a great introduction.

C.

[Spike Morelli]

They are fantastic books indeed, and not just for their content, but also for their making if you appreciate that kind of thing. That said in my experience they are not as effective as 'visualize this' when it comes down to getting stuff into a graph. I'd still recommend to pick them up, but I'd use something like 'visualise this' or 'Information Dashboard Design: The Effective Visual Communication of Data' as more practical companion books.

hope that helps,

Spike

--

http://www.spikelab.org/

http://twitter.com/spikelab

http://uk.linkedin.com/in/spikemorelli

[Bryan Berry]

wow! this is exactly the chain of responses I was hoping for. I will take a look at Tufte's book and Visualize this. Thanks again!

[Peco Karayanev]

For Tufte fans (self included) check out it his web site on when / where he is teaching seminars. They are relatively inexpensive and you get the four books he published for free as part of the course (you can even get an autograph from the man himself). If you are a data geek it is a must-attend event!

 

Cheers

[Guess Who?]

haha, I will second that sentiment as well! I didn't get my autographed; damn. I might just have to take it again.

C.

[Anthony J Burton]

It occurred to me early on in this thread that a really cool meet up for after one of Tufte's workshops would be a more extensive workshop on tools and techniques specific to devops. Tufte usually has a workshop in Arlington VA, but I don't see one scheduled right now…otherwise I'd think about organizing this for #devopsdc.

--ant

[Mason Turner]

+1 for a coordinated @devopsdc meetup, when an Arlington course is scheduled. Let me know how I can help.

-- Mason Turner (mobile)

[Anthony J Burton]

I just heard back from the Tufte people, and they say that the next class in Arlington will be sometime in May (dates to be posted to their website soon). I've received a number of responses showing interest in a follow on workshop with visualization topics specific to DevOps.

I think it makes sense to try to organize something that would highlight topics from the Tufte class and work with specific tools to get answers from different kind of monitoring or performance data. I'm thinking of an unconference sortofa thing; any other ideas about what might work?

Hopefully, if we can get a good group, I can recommend that Mr Tufte specifically address monitoring and/or performance data. At the very least we'd be there to ask these types of specific questions.

If we get 10 people who sign up, there's a 20% (on $380) discount to the Tufte event. And, you can get your books signed, which is of course priceless.

Perhaps this is something I should move off the list, and bring into the DevOpsDC mailing list? I wanted to post this here at this point to again gauge/spark interest.

--ant

[Chas.]

I can't make it to Arlington, so I am taking the class in Boston. It would be nice if he is up to the idea of having his class televised (for the same fee).  I'd love to hear his thoughts on displaying monitoring/performance data visualization. Someone down there (in Arlington) want to ask?

If you guys down there do decide to get together it would be cool to pull in others of us on the fringes...lol

Peace,

Chuck Wegrzyn

[Scott Sanders]

We would send at least three people to this event, possibly more. In addition, we'd be happy to sponsor pizza/beer/etc. Thanks for doing the legwork, Anthony!

-Scott Sanders

[Jessvin Thomas]

I would be in for Boston. I could probably get some others too.

Sent from my Windows Phone

---

From: Chas.
Sent: 2/3/2012 1:26 PM
To: devops-t...@googlegroups.com

Subject: Re: How to get started with data visualization?

[Guess Who?]

Ant, are you in communication with Edward already? I don't want to start a lot of overlapping conversations about focusing some of his class on monitoring and performance visualization. If you aren't, I'd be more than happy to talk to him about it.

Regards,
Chuck Wegrzyn

On 2/5/2012 2:05 PM, Jessvin Thomas wrote:

I would be in for Boston. I could probably get some others too.

Sent from my Windows Phone

---

From: Chas.
Sent: 2/3/2012 1:26 PM
To: devops-t...@googlegroups.com
Subject: Re: How to get started with data visualization?

I can't make it to Arlington, so I am taking the class in Boston. It would be nice if he is up to the idea of having his class televised (for the same fee). �I'd love to hear his thoughts on displaying monitoring/performance data visualization. Someone down there (in Arlington) want to ask?

If you guys down there do decide to get together it would be cool to pull in others of us on the fringes...lol

Peace,

Chuck Wegrzyn

On Fri, Feb 3, 2012 at 1:15 PM, Anthony J Burton <anfern...@gmail.com> wrote:

I just heard back from the Tufte people, and they say that the next class in Arlington will be sometime in May (dates to be posted to their website soon). I've received a number of responses showing interest in a follow on workshop with visualization topics specific to DevOps.

I think it makes sense to try to organize something that would highlight topics from the Tufte class and work with specific tools to get answers from different kind of monitoring or performance data. I'm thinking of an unconference sortofa thing; any other ideas about what might work?

Hopefully, if we can get a good group, I can recommend that Mr Tufte specifically address monitoring and/or performance data. At the very least we'd be there to ask these types of specific questions.

If we get 10 people who sign up, there's a 20% (on $380) discount to the Tufte event. And, you can get your books signed, which is of course priceless.

Perhaps this is something I should move off the list, and bring into the DevOpsDC mailing list? I wanted to post this here at this point to again gauge/spark interest.

--ant

On Thursday, January 26, 2012 at 11:49 AM, Anthony J Burton wrote:

It occurred to me early on in this thread that a really cool meet up for after one of Tufte's workshops would be a more extensive workshop on tools and techniques specific to devops. Tufte usually has a workshop in Arlington VA, but I don't see one scheduled right now�otherwise I'd think about organizing this for #devopsdc.

--ant

On Thursday, January 26, 2012 at 11:07 AM, Guess Who? wrote:

haha, I will second that sentiment as well! I didn't get my autographed; damn. I might just have to take it again.

C.

On 1/26/2012 11:04 AM, Peco Karayanev wrote:

For Tufte fans (self included) check out it his web site on when / where he is teaching seminars. They are relatively inexpensive and you get the four books he published for free as part of the course (you can even get an autograph from the man himself). If you are a data geek it is a must-attend event!

�

Cheers

[Damon Edwards]

If the class was focused on monitoring and performance (or at least a class filled with DevOps minded folks) I would be interested. 

If it was recorded and I could watch remotely (even at similar fee) it would be a no-brainer.

-Damon

On Feb 5, 2012, at 11:15 AM, Guess Who? <eprpa...@gmail.com> wrote:

Ant, are you in communication with Edward already? I don't want to start a lot of overlapping conversations about focusing some of his class on monitoring and performance visualization. If you aren't, I'd be more than happy to talk to him about it.

Regards,
Chuck Wegrzyn

On 2/5/2012 2:05 PM, Jessvin Thomas wrote:

I would be in for Boston. I could probably get some others too.

Sent from my Windows Phone

---

From: Chas.
Sent: 2/3/2012 1:26 PM
To: devops-t...@googlegroups.com
Subject: Re: How to get started with data visualization?

I can't make it to Arlington, so I am taking the class in Boston. It would be nice if he is up to the idea of having his class televised (for the same fee).  I'd love to hear his thoughts on displaying monitoring/performance data visualization. Someone down there (in Arlington) want to ask?

If you guys down there do decide to get together it would be cool to pull in others of us on the fringes...lol

Peace,

Chuck Wegrzyn

On Fri, Feb 3, 2012 at 1:15 PM, Anthony J Burton <anfern...@gmail.com> wrote:

I just heard back from the Tufte people, and they say that the next class in Arlington will be sometime in May (dates to be posted to their website soon). I've received a number of responses showing interest in a follow on workshop with visualization topics specific to DevOps.

I think it makes sense to try to organize something that would highlight topics from the Tufte class and work with specific tools to get answers from different kind of monitoring or performance data. I'm thinking of an unconference sortofa thing; any other ideas about what might work?

Hopefully, if we can get a good group, I can recommend that Mr Tufte specifically address monitoring and/or performance data. At the very least we'd be there to ask these types of specific questions.

If we get 10 people who sign up, there's a 20% (on $380) discount to the Tufte event. And, you can get your books signed, which is of course priceless.

Perhaps this is something I should move off the list, and bring into the DevOpsDC mailing list? I wanted to post this here at this point to again gauge/spark interest.

--ant

On Thursday, January 26, 2012 at 11:49 AM, Anthony J Burton wrote:

It occurred to me early on in this thread that a really cool meet up for after one of Tufte's workshops would be a more extensive workshop on tools and techniques specific to devops. Tufte usually has a workshop in Arlington VA, but I don't see one scheduled right now…otherwise I'd think about organizing this for #devopsdc.

--ant

On Thursday, January 26, 2012 at 11:07 AM, Guess Who? wrote:

haha, I will second that sentiment as well! I didn't get my autographed; damn. I might just have to take it again.

C.

On 1/26/2012 11:04 AM, Peco Karayanev wrote:

For Tufte fans (self included) check out it his web site on when / where he is teaching seminars. They are relatively inexpensive and you get the four books he published for free as part of the course (you can even get an autograph from the man himself). If you are a data geek it is a must-attend event!

 

Cheers

[Guess Who?]

Once I know if "Ant" is handling it or not, one of us will talk to Tufte and see what is possible.

Chuck Wegrzyn

[Anthony J Burton]

I'm not in contact with Edward directly, I just sent a general message and heard back from someone within the organization. I would only really be able to organize something for the DC area, although it is compelling to consider potentially taking the class in Boston and helping to organize a follow on meet up, there, and then using that as a basis for something in DC. Maybe that's just magic thinking tho. Any excuse to go to Boston...

I don't think filming the session is something they're going to agree to, but perhaps it's worth asking.

--ant

[Jessvin Thomas]

I'm in Boston and could help.
random coincidence: I grew up in the same town as the checks for the class go to :)

Is there anyone beside chuck who might be interested in the Boston meetup so  I can gauge interest level?

Jessvin Thomas

Sent from my Windows Phone

---

From: Anthony J Burton
Sent: 2/6/2012 12:14 PM

[Chas.]

I've already been in touch with Tufte. So back off Jessvin....

Chuck

[Sebastian Otaegui]

Hello all,

Can you guys post the full info about the class in Arlington 

Thanks

--
Those who do not understand Unix are condemned to reinvent it, poorly.
Any sufficiently recent Microsoft OS contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Unix.

[Chas.]

For the classes you can go to http://edwardtufte.com . I am trying to co-ordinate a bit more related to devops and in particular visualization ideas for performance and operations. I'll let everyone how it goes.

Chuck Wegrzyn

[Anthony J Burton]

They said that the class(es) in Arlington would be in May, and that they would be posting the specific dates "soon".

I definitely would like to know how receptive they are to suggesting performance and operations. 

By this point they have to be saying, "WTF is up with these DevOps people…"

--ant

[Jacob Rosenberg]

On Feb 6, 2012, at 3:48 PM, Anthony J Burton wrote:
> By this point they have to be saying, "WTF is up with these DevOps people…"

I believe the expression you're looking for is "Woohoo"

[Khyron]

Anthony:

I would be interested in the Arlington class.  We can take specifics off-list, if that helps.  I should be on 

the DevOpsDC list as well but in case...  I'll see about generating some more interest from internal to my 

group with my employer.

Claude Johnson

--

"May the bridges I burn light my way forward." - Unknown

"You can choose your friends, you can choose the deals." - Equity Private

| www.linkedin.com/in/claudeljohnson | www.whatderass.com | @khyron4eva |

[Chas.]

I took his class a few years back and he was always receptive to answering specific questions (I was involved in healthcare and needed to display some information for which I couldn't find anything good). So I am hopeful he will be willing to spend some cycles on it.

It would almost be useful to have him teach at a devops meeting since there would be a bigger selection of us present.

Chuck

[Sebastian Otaegui]

+1 to that.

[Anthony J Burton]

I asked about that specifically and this was the response:

"ET is the only one who deals with the information from the course, and his schedule is always tight whenever he is on location, so he unfortunately would not be available for the workshop."

--ant

[Jessvin Thomas]

No worries. I missed your email looking for you or ant to coordinate coordinates apologize for that. I should have said anyway i was wondering if there was anyone else in Boston in addition to us, not excluding you anyway.

Let me know if I can by lending space or food or something.

jessvin

Sent from my Windows Phone

---

From: Chas.
Sent: 2/6/2012 2:20 PM
To: devops-t...@googlegroups.com
Cc: Anthony J Burton