Re: Network Analyzer

[Nathan Toone]

I thought I'd copy this email to the neta-discuss list as well - that way maybe others can join in on suggestions...previous emails in this thread are appended at the end.  I went through all that trouble, and didn't send it... ;)  Here it is now.

On Apr 23, 2007, at 10:55 PM, Casey Marshall wrote:

That is a good idea (though, Network Analyzer uses libpcap, not tcpdump itself), and I considered doing that. Working out an existing API is not very fun, though, and isn't particularly easy.

A benefit of rolling my own plugin interface is that you can write dissectors in Objective-C, and use Cocoa.

That is true - maybe I can spend some time looking through how Wireshark does it, and get some ideas - maybe there is some shared code that we can use (I know that there is a library that gets built with wireshark - I'm not sure what it all contains though.)

I'm thinking that the toolbar "filter" field could have different modes, like the search field in Xcode's documentation browser: one mode does an API search for the query, the other a full documentation search. It integrates a pop-up in the text field itself. One mode accepts random text strings, the other hex strings.

That is what I was envisioning (I had forgotten that the exact thing I was looking for was in Xcode...)  Another option (beyond text and hex) would probably be the ability to enter in a "wireshark/libpcap-style" query directly...many people are used to using that syntax as well.

Another option with this would be to give "small" per-field filters - similar to what Mail.app does when you search.  When you search, it gives you options to search in "To", "From", "Subject", and "Entire Message" - it might make sense to restrict the toolbar search similarly to certain areas (packet headers, body, etc)...it also might not make sense...  I don't know.  :)

The current capture filter (which constructs a filter that libpcap itself uses) does something like this. The "find" panel can be much more expressive than that filter, however, so the UI may necessarily be more complex. I'd also like to have some good way to construct AND or OR relations between predicates, because now many find/smart folder interfaces make you choose "match all" or "match any." I haven't come up with a good way to present something more complicated, though.

The way you can do "And" and "Or" using this type of filter is to allow a filter rule of "Matches xxx filter"...especially if set it up to save "smart captures".  You could then do, for example, a filter called "My Traffic" (Match any of source=me, dest=me) and a filter called "My HTTP Traffic" (match all of protocol=http, "My Traffic"), to effectively get "(source=me || dest=me) && protocol=http"  It's not the "easiest", but if you allow for the ability to match other rules, then the UI remains simple and elegant.  Once you try to add in "And" or "Or" between elements, it starts to get really complicated - especially when you want to use parenthesis for order of operations.  Nesting "smart captures" within each other will, in effect, do that for you.

That's a great idea. I even thought of a pretty good interface for that: add a "blue sidebar" to the left of the three main panels, which holds saved searches ("smart captures?" hmm), and clicking on them filters the current capture in the window with that search. Lots of programs are adopting this interface (like Mail, iTunes, iPhoto, Colloquy, and even Thunderbird) and it looks nice.

The sidebar could even be a simple capture file manager -- you also get a list of captures you've made, and look at them, or even look at multiple captures at once.

There's still a lot more here. Like, finding a way to erase the "packet" delineation when you are looking at a protocol -- you don't care about what packet arrived when, just the traffic at the protocol level.

The "new" and "cool" color is that grey in iTunes... ;)  Blue is *sooo* 10.4.... :)

I agree - and was thinking about it a bit yesterday.  Maybe an interface laid out similar to Mail - or any other of a number of applications - where each packet is a "message" and clicking it brings up the details.  But there are folders to manage it.

Another really cool item which I think could be use would be if there were a drawer that would automatically "Follow TCP Stream".  From watching people here in our office use Wireshark, the first thing they do when they find the packet they want is to choose "Follow TCP Stream".  Why make them do it separately, and in a different window?  Why not just open a "TCP Stream" drawer which would automatically follow it for them?  

Agreed. It always feels like "work" when I'm trying to get some bit of information out of Wireshark.

Argh! If only I had the time to hack on this ;-)

I know how you feel - I'm pretty pressed for time as well - but as I mentioned before, I'm very willing to help out.  I have checked out the code and can look through it.  Let me know of anywhere where I can help out!

-Nathan

======================================

Previous messages in thread (for posterity... :) :

======================================

-----------------------

On Apr 21, 2007, at 1:45 AM, Nathan Toone wrote:

Hey -

I just stumbled across your Network Analyzer program and I think it's looking really great and holding quite a bit of promise.  I've also been thinking about writing a cocoa frontend to Wireshark for a while - and every time I start I get discouraged.  I am very disappointed with the state of capturing software on the Mac.

I wanted to let you know that it's looking really good and offer any help in UI design or testing that you may need.  I'm not an expert Cocoa programmer (yet) - I've mainly worked in C and writing libraries instead (Linux and Mac, mainly).  However, I'm willing to help out in whatever way you think would be helpful.

Good luck!

-Nathan

-----------------------

On Apr 22, 2007, at 1:33 AM, Casey Marshall wrote:

Thanks for the feedback and encouragement!

What I'm looking to do next with Network Analyzer is a couple of things: one, write more "dissector" plugins that decode protocols. Wireshark has a ton of these, that can decode nearly any network protocol active today, and I'd need some, too. Part of this is evaluating/changing the plugin API, which may have some rough spots. I am at a disadvantage, because I chose to write my own dissector plugin API instead of adopting Wireshark's, but maybe that one can be adapted.

The other thing I want to do is present an "insanely great" find/filter interface, so a user can filter and search a trace as easily as possible. I'm thinking now of a two-level interface: one, a "filter" bar in the toolbar, where you type random strings, and this filters the current window of packets that contain that byte sequence. Two, have a "find" panel where a user can structure queries more precisely (based on key-value matching, like, "match HTTP field foo: with value XXX," etc.). This selects packets that match the query. Even this seems pretty limited about what's possible, in the landscape of visual packet search/filtering. But anyway, I'd appreciate any ideas on how to go about searching captured data.

Cheers.

-----------------------

On Apr 22, 2007, at 8:49 AM, Nathan Toone wrote:

I have a couple of opinions/suggestions here (and they are only suggestions)

For the dissectors, I wonder if it would be possible to build Network Analyzer using tshark (the command-line version of wireshark) rather than based off of tcpdump.  You may even be able to use tcpdump for capturing and tshark for dissecting.  I don't know how many dissectors are available for tshark - but it might get you a bit of a jump.  tshark compiles just fine for  Mac OS (no need for X11)

I think the idea for the find/filter interface is the way you want to go.  I am a big proponent of the "Apple" way of doing things.  A toolbar filter which just gives you the ability to type in strings (you may also want to find data) would be very good.  Have you ever seen the toolbar filter for 0xed?  I think that it would be a really good thing to implement something similar.  That is, give them the search bar and allow them to enter in either strings or hex data - and they choose by clicking the dropdown by the looking glass whether they want to search for strings or hex.  That filter would just display any packets that contain the string (or hex data) that they enter in.

The "find" panel, in my opinion should be structured like all the other "find" panels in OSX (ie. like smart folders in iTunes, rules in Mail.app, etc)  You pick a field, a comparator (is, is not, contains, starts with, is less than, is before, etc) - depending on the field that they select, and a value which can either be a dropdown, or a text box, or something else - again, depending on the field and comparator that they have selected.

I am not very familiar with cocoa, but I would guess that there are prebuilt UI items that do these things.

Also - I just thought (while writing this to you) that it might be cool to have some kind of "saved folder"-type thing...that would be the real killer.  For example, imagine the ability to write a filter and save it for "HTTP traffic originating from my computer" or some other common filters that people may use.  You could capture traffic (or load a capture) and open up one of your "saved folders" to see stuff you commonly want to see.

Anyway - those are just a couple of ideas/opinions.  I think find/filtering needs to be awesome...that's one of the things that I absolutely *HATE* about wireshark...their find functions are horrible!

-Nathan

-----------------------

On Apr 23, 2007, at 7:49 AM, Casey Marshall wrote:

For the dissectors, I wonder if it would be possible to build Network Analyzer using tshark (the command-line version of wireshark) rather than based off of tcpdump.  You may even be able to use tcpdump for capturing and tshark for dissecting.  I don't know how many dissectors are available for tshark - but it might get you a bit of a jump.  tshark compiles just fine for  Mac OS (no need for X11)

That is a good idea (though, Network Analyzer uses libpcap, not tcpdump itself), and I considered doing that. Working out an existing API is not very fun, though, and isn't particularly easy.

A benefit of rolling my own plugin interface is that you can write dissectors in Objective-C, and use Cocoa.

I think the idea for the find/filter interface is the way you want to go.  I am a big proponent of the "Apple" way of doing things.  A toolbar filter which just gives you the ability to type in strings (you may also want to find data) would be very good.  Have you ever seen the toolbar filter for 0xed?  I think that it would be a really good thing to implement something similar.  That is, give them the search bar and allow them to enter in either strings or hex data - and they choose by clicking the dropdown by the looking glass whether they want to search for strings or hex.  That filter would just display any packets that contain the string (or hex data) that they enter in.

I'm thinking that the toolbar "filter" field could have different modes, like the search field in Xcode's documentation browser: one mode does an API search for the query, the other a full documentation search. It integrates a pop-up in the text field itself. One mode accepts random text strings, the other hex strings.

The "find" panel, in my opinion should be structured like all the other "find" panels in OSX (ie. like smart folders in iTunes, rules in Mail.app, etc)  You pick a field, a comparator (is, is not, contains, starts with, is less than, is before, etc) - depending on the field that they select, and a value which can either be a dropdown, or a text box, or something else - again, depending on the field and comparator that they have selected.

I am not very familiar with cocoa, but I would guess that there are prebuilt UI items that do these things.

The current capture filter (which constructs a filter that libpcap itself uses) does something like this. The "find" panel can be much more expressive than that filter, however, so the UI may necessarily be more complex. I'd also like to have some good way to construct AND or OR relations between predicates, because now many find/smart folder interfaces make you choose "match all" or "match any." I haven't come up with a good way to present something more complicated, though.

Also - I just thought (while writing this to you) that it might be cool to have some kind of "saved folder"-type thing...that would be the real killer.  For example, imagine the ability to write a filter and save it for "HTTP traffic originating from my computer" or some other common filters that people may use.  You could capture traffic (or load a capture) and open up one of your "saved folders" to see stuff you commonly want to see.

That's a great idea. I even thought of a pretty good interface for that: add a "blue sidebar" to the left of the three main panels, which holds saved searches ("smart captures?" hmm), and clicking on them filters the current capture in the window with that search. Lots of programs are adopting this interface (like Mail, iTunes, iPhoto, Colloquy, and even Thunderbird) and it looks nice.

The sidebar could even be a simple capture file manager -- you also get a list of captures you've made, and look at them, or even look at multiple captures at once.

There's still a lot more here. Like, finding a way to erase the "packet" delineation when you are looking at a protocol -- you don't care about what packet arrived when, just the traffic at the protocol level.

Anyway - those are just a couple of ideas/opinions.  I think find/filtering needs to be awesome...that's one of the things that I absolutely *HATE* about wireshark...their find functions are horrible!

Agreed. It always feels like "work" when I'm trying to get some bit of information out of Wireshark.

Argh! If only I had the time to hack on this ;-)

Thanks.

[Casey Marshall]

On Apr 24, 2007, at 11:24 AM, Nathan Toone wrote:

> On Apr 23, 2007, at 10:55 PM, Casey Marshall wrote:
>>
>> That is a good idea (though, Network Analyzer uses libpcap, not
>> tcpdump itself), and I considered doing that. Working out an
>> existing API is not very fun, though, and isn't particularly easy.
>>
>> A benefit of rolling my own plugin interface is that you can write
>> dissectors in Objective-C, and use Cocoa.
>>
>
> That is true - maybe I can spend some time looking through how
> Wireshark does it, and get some ideas - maybe there is some shared
> code that we can use (I know that there is a library that gets
> built with wireshark - I'm not sure what it all contains though.)
>
>> I'm thinking that the toolbar "filter" field could have different
>> modes, like the search field in Xcode's documentation browser: one
>> mode does an API search for the query, the other a full
>> documentation search. It integrates a pop-up in the text field
>> itself. One mode accepts random text strings, the other hex strings.
>>
>
> That is what I was envisioning (I had forgotten that the exact
> thing I was looking for was in Xcode...) Another option (beyond
> text and hex) would probably be the ability to enter in a
> "wireshark/libpcap-style" query directly...many people are used to
> using that syntax as well.
>

+1

> Another option with this would be to give "small" per-field filters
> - similar to what Mail.app does when you search. When you search,
> it gives you options to search in "To", "From", "Subject", and
> "Entire Message" - it might make sense to restrict the toolbar
> search similarly to certain areas (packet headers, body, etc)...it
> also might not make sense... I don't know. :)
>

That's a good idea. The only issue is that network traffic is
necessarily more than one protocol (the only common protocol is
usually Ethernet), so it's hard to distill a small set of fields to
search from a variety of "header" and "body" concepts.

Options, though could include matching "Address", or "Port", since
*most* traffic uses these two concepts.

>>
>> The current capture filter (which constructs a filter that libpcap
>> itself uses) does something like this. The "find" panel can be
>> much more expressive than that filter, however, so the UI may
>> necessarily be more complex. I'd also like to have some good way
>> to construct AND or OR relations between predicates, because now
>> many find/smart folder interfaces make you choose "match all" or
>> "match any." I haven't come up with a good way to present
>> something more complicated, though.
>>
>
> The way you can do "And" and "Or" using this type of filter is to
> allow a filter rule of "Matches xxx filter"...especially if set it
> up to save "smart captures". You could then do, for example, a
> filter called "My Traffic" (Match any of source=me, dest=me) and a
> filter called "My HTTP Traffic" (match all of protocol=http, "My
> Traffic"), to effectively get "(source=me || dest=me) &&
> protocol=http" It's not the "easiest", but if you allow for the
> ability to match other rules, then the UI remains simple and
> elegant. Once you try to add in "And" or "Or" between elements, it
> starts to get really complicated - especially when you want to use
> parenthesis for order of operations. Nesting "smart captures"
> within each other will, in effect, do that for you.
>

That's a nice way to handle it. The only gotcha is that you have to
forbid circular references.

I was thinking of a way to handle this, where you have a "two level"
search filter: you have a set of filter groups, each with a "match
all" or "match any" choice, and containing that, you have another
"match all" or "match any" rule.

Bogus ascii-art depiction:

Match all/any of the following: +

> Filter "foo"; match all/any of the following -
filter foo predicate 1 - +
filter foo predicate 2 - +

> Filter "bar"; match all/any of the following -
filter bar predicate 1 - +
filter bar predicate 2 - +

So each sub-filter has a disclosure triangle, to expand it, and has a
removal button (the -) to remove that whole sub-filter. Each sub-
filter has a smart-folder-like list of predicates, with add/remove
buttons. Finally, the top-level has an add button, which adds a new
sub-filter.

That at least lets you make a sentence in conjunctive normal form, so
you can express any filter possible. It just isn't pretty, though.

A drawer would work for that, though the real estate in drawers is
usually small. Another nice way to do it is to make a "bezel"
interface, similar to the translucent utility windows we see in some
apps. The content of the bezel could be the TCP stream you're
following (color-coded depending on which peer was sending it, maybe).

>> Agreed. It always feels like "work" when I'm trying to get some
>> bit of information out of Wireshark.
>>
>> Argh! If only I had the time to hack on this ;-)
>
> I know how you feel - I'm pretty pressed for time as well - but as
> I mentioned before, I'm very willing to help out. I have checked
> out the code and can look through it. Let me know of anywhere
> where I can help out!
>

This discussion is a good start :-)

Thanks.