Palo Alto Network Traffic Parsing

[James Espinosa]

Hi,

I'm not sure if I'm mistaken or not, but I notice that my PAN traffic is not being parsed correctly by ELSA. I can see that palo_alto_traffic exists in the patterndb.xml file but I'm not sure why it is not getting parsed correctly. I can only see the entire message being indexed in ELSA but not in its broken down form (I can only query program, class, or complete keyword search). Hopefully this makes sense.

Thanks

[Rich Graves]

In your search results, are they coming across as class=NONE?

If so, you need to do the default-commented-out inserts in /usr/local/elsa/node/conf/schema.sql.

You might be the second user of these patterns. Improvements would be most welcome. I haven't had time, and Martin doesn't have such a firewall.

[James Espinosa]

Thanks for the reply Rich. Yes, search results come in as class=NONE and program=unknown. Ill take a look at the schema.sql and post an update when I find out. 

[Martin Holste]

Lucky for you, one of our customers just got a PAN device, so I too
have access to these logs and am experiencing the same parsing
problem. It's on my list today to get corrected.

[James Espinosa]

That is great to hear! Looking forward to an update when you get around to it then.

Thanks guys!

[Rich Graves]

Always nice to have more users.

PAN reserves the right to add or change fields for different software versions. I'd wanted to avoid it, but it is probably safest in the long run to give up, go to https://firewall/#device::shared::device/server-profile/syslog and create a Custom Log Format. This could then be exported as XML and added to ELSA documentation.

[Rich Graves]

FWIW, I have *no* problem parsing traffic logs from a 5020 running PANOS 4.1.6. I've made no changes to stock patterndb.

I do have two types of class=NONE. First is SYSTEM events. I don't have any problem with leaving these unparsed.

17:06,00****000***,SYSTEM,userid,0,2012/09/04 10:17:06,,connect-agent-failure,,0,0,general,high,User-ID-Agent blah(vsys1): Error: Failed to connect to User-ID-Agent at **.**.**.**(**.**.**.**):****,104552,0x0

Second is that a few (less than 5%) URL logs aren't parsed. Maybe it's the curly braces or some other characters in the URL?

17:10,002501000259,THREAT,url,0,2012/09/04 10:17:09,**.**.**.**,69.171.224.37,0.0.0.0,0.0.0.0,Blag-Network-Allow-Blah,,,facebook-base,vsys1,External,External,vlan.111,ethernet1/1,forward-syslog-to-elsa,2012/09/04 10:17:09,836957,1,53280,80,0,0,0x8000,tcp,alert,"www.facebook.com/ajax/pagelet/generic.php/MoreStoriesPagelet?ajaxpipe=1&ajaxpipe_token=***********&no_script_path=1&data={""filter"":""h_nor"",""oldest"":1346771373,""oldestMR"":1346765753,""last_seen_time"":1346771796,""scroll_count"":1,""scroll_position"":7,""last_viewstate_id"":""-********************"",""cached_scroll_qid"":""**********************"",""delay_load_count"":12}&__user=************&__a=1&__adt=2",(9999),All,informational,client-to-server,167280881,0x0,United States,United States,0,text/html

[Martin Holste]

Yes, it definitely looks like you can't count on the log format all
the time, but I did just commit some additional alternate patterns
which will pickup the pseudo-WELF format I'm seeing. Here's the
shortcut for inserting the new fields:

INSERT INTO classes (id, class, parent_id) VALUES(24, "PALO_ALTO_URL", 0);
INSERT INTO classes (id, class, parent_id) VALUES(25, "PALO_ALTO_TRAFFIC", 0);
INSERT INTO fields (field, field_type, pattern_type) VALUES
("src_zone", "string", "QSTRING");
INSERT INTO fields (field, field_type, pattern_type) VALUES
("dst_zone", "string", "QSTRING");
INSERT INTO fields (field, field_type, pattern_type) VALUES ("rule",
"string", "QSTRING");
INSERT INTO fields (field, field_type, pattern_type) VALUES
("country", "string", "QSTRING");
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_URL"), (SELECT
id FROM fields WHERE field="srcip"), 5);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_URL"), (SELECT
id FROM fields WHERE field="dstip"), 6);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_URL"), (SELECT
id FROM fields WHERE field="content_length"), 7);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_URL"), (SELECT
id FROM fields WHERE field="rule"), 11);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_URL"), (SELECT
id FROM fields WHERE field="user"), 12);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_URL"), (SELECT
id FROM fields WHERE field="category"), 13);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_URL"), (SELECT
id FROM fields WHERE field="site"), 14);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_URL"), (SELECT
id FROM fields WHERE field="uri"), 15);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_URL"), (SELECT
id FROM fields WHERE field="country"), 16);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_TRAFFIC"),
(SELECT id FROM fields WHERE field="srcip"), 5);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_TRAFFIC"),
(SELECT id FROM fields WHERE field="dstip"), 6);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_TRAFFIC"),
(SELECT id FROM fields WHERE field="srcport"), 7);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_TRAFFIC"),
(SELECT id FROM fields WHERE field="dstport"), 8);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_TRAFFIC"),
(SELECT id FROM fields WHERE field="proto"), 9);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_TRAFFIC"),
(SELECT id FROM fields WHERE field="conn_bytes"), 10);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_TRAFFIC"),
(SELECT id FROM fields WHERE field="src_zone"), 11);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_TRAFFIC"),
(SELECT id FROM fields WHERE field="dst_zone"), 12);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_TRAFFIC"),
(SELECT id FROM fields WHERE field="i_int"), 13);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_TRAFFIC"),
(SELECT id FROM fields WHERE field="o_int"), 14);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_TRAFFIC"),
(SELECT id FROM fields WHERE field="country"), 15);
INSERT INTO fields_classes_map (class_id, field_id, field_order)
VALUES ((SELECT id FROM classes WHERE class="PALO_ALTO_TRAFFIC"),
(SELECT id FROM fields WHERE field="category"), 16);

Then just copy the patterndb.xml to /usr/local/elsa/node/conf/ and
your instance should pick it up immediately.

[Rich Graves]

> I noticed that my pan traffic example fails on the default pattern that is included with ELSA because of the second field

That number is your serial number (hopefully it's not terribly sensitive). I'm guessing the "C" means a cluster or an RMA or something.

Yes, I think it should be ESTRING, but I'm no expert either.

[James Espinosa]

Rich: Good call, here's a repost in case anyone else comes up with anything. Thanks!

Hi Martin,

I've updated to the latest ELSA release, and have been playing around with the palo_alto_traffic pattern but have had absolutely no luck in getting it to work properly. Below is a sample message that I received (IP addresses and names have been changed for this example), and the results from my pattern output. I'm not sure what I'm doing wrong. Any additional help is much appreciated.

Thanks,

PAN TRAFFIC EXAMPLE

14:02,000?C123456,TRAFFIC,start,1,2012/10/03 06:14:02,192.168.0.2,74.125.228.34,0.0.0.0,0.0.0.0,US,,,dns,vsys1,US Traffic,US Traffic,ethernet1/3,ethernet1/3,ELSA-TR,2012/10/03 06:14:01,21757,1,59403,53,0,0,0x0,udp,allow,94,94,0,1,2012/10/03 06:14:02,0,any,0,3644123456,0x0,US,United Kingdom,0,1,0

MODIFIED PATTERN EXAMPLE

<pattern>@NUMBER::@:@NUMBER::@,@ESTRING::,@TRAFFIC,@ESTRING:s5:,@@ESTRING::,@@ESTRING::,@@IPv4:i0:@,@IPv4:i1:@,@IPv4::@,@IPv4::@,@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING:s0:,@@ESTRING:s1:,@@ESTRING:s2:,@ESTRING:s3:,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING:i2:,@@ESTRING:i3:,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING:i4:,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING::,@@ESTRING:s4:,@@ESTRING::,@@ESTRING::,@@ESTRING::,@</pattern>

PDBTOOL TEST

/usr/local/syslog-ng/bin/pdbtool match -p /usr/local/elsa/node/conf/patterndb.xml -P palo_alto_traffic -M "14:02,000?C123456,TRAFFIC,start,1,2012/10/03 06:14:02,192.168.0.2,74.125.228.34,0.0.0.0,0.0.0.0,US,,,dns,vsys1,US Traffic,US Traffic,ethernet1/3,ethernet1/3,ELSA-TR,2012/10/03 06:14:01,21757,1,59403,53,0,0,0x0,udp,allow,94,94,0,1,2012/10/03 06:14:02,0,any,0,3644123456,0x0,US,United Kingdom,0,1,0"

PDBTOOL RESULTS

Key contains '@' without escaping; key='@', value='25'

MESSAGE=14:02,000?C123456,TRAFFIC,start,1,2012/10/03 06:14:02,192.168.0.2,74.125.228.34,0.0.0.0,0.0.0.0,US,,,dns,vsys1,US Traffic,US Traffic,ethernet1/3,ethernet1/3,ELSA-TR,2012/10/03 06:14:01,21757,1,59403,53,0,0,0x0,udp,allow,94,94,0,1,2012/10/03 06:14:02,0,any,0,3644123456,0x0,US,United Kingdom,0,1,0

PROGRAM=palo_alto_traffic

.classifier.class=unknown

PATTERN NOTICE

I noticed that my pan traffic example fails on the default pattern that is included with ELSA because of the second field (MESSAGE=14:02,000?C123456,TRAFFIC). The default pattern matches on @NUMBER, and as you can see it contains a "C", which throws it off. It should probably be @ESTRING - but I'm not an expert and I am just starting to get familiar with this stuff :)

[Martin Holste]

Yep, exactly right!  I've updated the pattern to use ESTRING instead of NUMBER.  I was unfortunate in that the test message I built the pattern on was a hexadecimal number with no letters in it, so I didn't know it was hex.  You can grab the latest patterndb.xml and overwrite the existing one and things should work immediately.

[James Espinosa]

Awesome! Thanks, I can now confirm that it's parsing correctly :) thanks!

[Eric G]

On Thursday, December 27, 2012 11:24:23 AM UTC-5, James Espinosa wrote:

Awesome! Thanks, I can now confirm that it's parsing correctly :) thanks!

I have been pointing people at ELSA on the Palo Alto forums every chance I get, so I'm glad to hear it's working. I can throw up an ELSA box and try to see if ELSA's parsing the latest PA PANOS revision (5.0.7) if I get some spare time. 

[Gianni K]

Does any one know where  to check for the User-ID-Agent xxxxxxxxx:Error: Failed To Connect to User-ID-Agent ?

[dd...@g-cpartners.com]

Hello all,

I'm new to Security Onion and I'm attempting to get syslog-ng to recognize the pattern from PanOS 8.0.8. I've noticed a difference in patterndb.xml from the syslog format that PanOS is spitting out. Has anyone updated the traffic patterns for PanOS 8? 

Sorry for dumb questions, 

Thanks