how to read isoblue public data

[Agenis Marc]

Hi, 

Im interested in Isoblue project, and I would like to start learning by downloading the public data logs and reading them. I did not quite understood the page "How to Read Data", i mean i don't understand if this is to read data from the isoblue or from the zip files. 

Where can I find simple instructions to open and read the zip files? (and like, maybe, transfering them into more exchangeable support, like json or bigquery table).  I'm not familiar with kafka

THanks,

Marc

[Yang Wang]

Hello Marc,

i mean i don't understand if this is to read data from the isoblue or from the zip files. 

The "How to Read Data"  section specifies the steps to read data from our backed-up public Kafka logs files.

Specifications of the data files are available here. All the files are in .tar.gz format.

Where can I find simple instructions to open and read the zip files? (and like, maybe, transfering them into more exchangeable support, like json or bigquery table).  I'm not familiar with kafka

A simple tar command like `tar xvf yourfile.tar.gz` will extract all the contents of a tar archive.

You don't necessarily need to know every nitty-gritty of Kafka to get the CAN, GPS data out of our published Kafka logs. The only thing you need to do is setting up the right Java environment and using the tools we provided in the "How to Read Data" section to read data.

For transfering existing data for more exchangeable formats, there are very new tools like ksqlDB. Or you can take a look at something we wrote for piping Kafka data into a MySQL database in Python.

Hope this helps.

Thanks,

Yang

[c.la...@gmail.com]

I was in the same situation. I wrote a stack overflow question on this. The Python2 scripts kept breaking for me, so I tried translating to Python3. https://stackoverflow.com/questions/62833949/quick-way-to-view-and-manipulate-kafka-stored-data-isoblue-isobus

I now suspect I am not "Consuming" all the data even though I am using the offset setting of "earliest". I thought it would take longer to go though 1GB+ of data.

I am not recognising PGNs. I can seen some Address Claim PGNs and Propitiatory Messages (0x00EA00 and 0x00EF00) in "case-8010-wheat-ault-072018", but the data isn't as I expected.

replaying: "case-sprayer-wheat-ault-062017" I was expecting to see Process data PGN 0x00CB00, but didn't. I really want to see an object pool transfer sequence 0x00E700 and TP messages 0x00EC00 0x00EB00 or ETP messages

{'message': '14ff1307faf6401af3130', 'pdu_pf': 255, 'payload_bytes': ['fa', 'f6', '40', '1a', 'f3', '13'], 'timestamp': 1492805003.304419, 'priority': 5, 'header': '14ff1307', 'pgn': 65299, 'source': 7, 'payload_int': 21934838445818}

{'message': '14ff140700c030ff3cecec', 'pdu_pf': 255, 'payload_bytes': ['00', 'c0', '30', 'ff', '3c', 'ec', 'ec'], 'timestamp': 1492805003.304966, 'priority': 5, 'header': '14ff1407', 'pgn': 65300, 'source': 7, 'payload_int': 66687841227292672}

{'message': '14ff27070fcffffffffffff', 'pdu_pf': 255, 'payload_bytes': ['0f', 'cf', 'ff', 'ff', 'ff', 'ff', 'ff'], 'timestamp': 1492805003.305582, 'priority': 5, 'header': '14ff2707', 'pgn': 65319, 'source': 7, 'payload_int': 72057594037915407}

{'message': '14ff16057d7d7d7d6d3bb01', 'pdu_pf': 255, 'payload_bytes': ['7d', '7d', '7d', '7d', '6d', '3b', 'b0'], 'timestamp': 1492805003.307868, 'priority': 5, 'header': '14ff1605', 'pgn': 65302, 'source': 5, 'payload_int': 49604937343925629}

{'message': '14ff25030ffffffffffffff', 'pdu_pf': 255, 'payload_bytes': ['0f', 'ff', 'ff', 'ff', 'ff', 'ff', 'ff'], 'timestamp': 1492805003.310929, 'priority': 5, 'header': '14ff2503', 'pgn': 65317, 'source': 3, 'payload_int': 72057594037927695}

{'message': '14ff030367bf0ff6819fcff', 'pdu_pf': 255, 'payload_bytes': ['67', 'bf', '0f', 'f6', '81', '9f', 'cf'], 'timestamp': 1492805003.311499, 'priority': 5, 'header': '14ff0303', 'pgn': 65283, 'source': 3, 'payload_int': 58440700706930535}

{'message': '14ff2b03000000000', 'pdu_pf': 255, 'payload_bytes': ['00', '00', '00', '00'], 'timestamp': 1492805003.312077, 'priority': 5, 'header': '14ff2b03', 'pgn': 65323, 'source': 3, 'payload_int': 0}

{'message': '14ff0f070f438a00000', 'pdu_pf': 255, 'payload_bytes': ['0f', '43', '8a', '00', '00'], 'timestamp': 1492805003.313689, 'priority': 5, 'header': '14ff0f07', 'pgn': 65295, 'source': 7, 'payload_int': 9061135}

{'message': '0cff1c00ec1b08a00000', 'pdu_pf': 255, 'payload_bytes': ['ec', '1b', '08', 'a0', '00', '00'], 'timestamp': 1492805003.314317, 'priority': 3, 'header': '0cff1c00', 'pgn': 65308, 'source': 0, 'payload_int': 2684885996}

{'message': '14ff28077d7d7d7d7d111101', 'pdu_pf': 255, 'payload_bytes': ['7d', '7d', '7d', '7d', '7d', '11', '11', '01'], 'timestamp': 1492805003.314824, 'priority': 5, 'header': '14ff2807', 'pgn': 65320, 'source': 7, 'payload_int': 76861899315969405}

{'message': '14ff10173b1c0000d400', 'pdu_pf': 255, 'payload_bytes': ['3b', '1c', '00', '00', 'd4', '00'], 'timestamp': 1492805003.324881, 'priority': 5, 'header': '14ff1017', 'pgn': 65296, 'source': 23, 'payload_int': 910533073979}

{'message': '14ff11170003000000', 'pdu_pf': 255, 'payload_bytes': ['00', '03', '00', '00', '00'], 'timestamp': 1492805003.325457, 'priority': 5, 'header': '14ff1117', 'pgn': 65297, 'source': 23, 'payload_int': 768}

{'message': '14ff0728040c11a1414', 'pdu_pf': 255, 'payload_bytes': ['04', '0c', '11', 'a1', '41'], 'timestamp': 1492805003.326027, 'priority': 5, 'header': '14ff0728', 'pgn': 65287, 'source': 40, 'payload_int': 281875123204}

{'message': '18fef000ffffff00ffffff', 'pdu_pf': 254, 'payload_bytes': ['ff', 'ff', 'ff', '00', 'ff', 'ff', 'ff'], 'timestamp': 1492805003.332962, 'priority': 6, 'header': '18fef000', 'pgn': 65264, 'source': 0, 'payload_int': 72057589759737855}

{'message': '0cff1c00ee1b08900000', 'pdu_pf': 255, 'payload_bytes': ['ee', '1b', '08', '90', '00', '00'], 'timestamp': 1492805003.333543, 'priority': 3, 'header': '0cff1c00', 'pgn': 65308, 'source': 0, 'payload_int': 2416450542}

{'message': '14fef017ffffffffffff0ff', 'pdu_pf': 254, 'payload_bytes': ['ff', 'ff', 'ff', 'ff', 'ff', 'ff', '0f'], 'timestamp': 1492805003.334812, 'priority': 5, 'header': '14fef017', 'pgn': 65264, 'source': 23, 'payload_int': 4503599627370495}

{'message': '14ff292800081ab1414', 'pdu_pf': 255, 'payload_bytes': ['00', '08', '1a', 'b1', '41'], 'timestamp': 1492805003.335385, 'priority': 5, 'header': '14ff2928', 'pgn': 65321, 'source': 40, 'payload_int': 282144147456}

{'message': '14ff1505304410c3fcffff', 'pdu_pf': 255, 'payload_bytes': ['30', '44', '10', 'c3', 'fc', 'ff', 'ff'], 'timestamp': 1492805003.347655, 'priority': 5, 'header': '14ff1505', 'pgn': 65301, 'source': 5, 'payload_int': 72057580130681904}

{'message': '14000003c0aaaafaffffffff', 'pdu_pf': 0, 'payload_bytes': ['c0', 'aa', 'aa', 'fa', 'ff', 'ff', 'ff', 'ff'], 'timestamp': 1492805003.350726, 'priority': 5, 'header': '14000003', 'pgn': 0, 'source': 3, 'payload_int': 18446744073620073152}

{'message': '0cff1c00e71b08900000', 'pdu_pf': 255, 'payload_bytes': ['e7', '1b', '08', '90', '00', '00'], 'timestamp': 1492805003.353588, 'priority': 3, 'header': '0cff1c00', 'pgn': 65308, 'source': 0, 'payload_int': 2416450535}

{'message': '14ff16057d7d7d7d6d3bb01', 'pdu_pf': 255, 'payload_bytes': ['7d', '7d', '7d', '7d', '6d', '3b', 'b0'], 'timestamp': 1492805003.357728, 'priority': 5, 'header': '14ff1605', 'pgn': 65302, 'source': 5, 'payload_int': 49604937343925629}

{'message': '14ff25030ffffffffffffff', 'pdu_pf': 255, 'payload_bytes': ['0f', 'ff', 'ff', 'ff', 'ff', 'ff', 'ff'], 'timestamp': 1492805003.360753, 'priority': 5, 'header': '14ff2503', 'pgn': 65317, 'source': 3, 'payload_int': 72057594037927695}

{'message': '14ff1705f0150251409e', 'pdu_pf': 255, 'payload_bytes': ['f0', '15', '02', '51', '40', '9e'], 'timestamp': 1492805003.367778, 'priority': 5, 'header': '14ff1705', 'pgn': 65303, 'source': 5, 'payload_int': 173999074186736}

{'message': '0cff1c00ee1b08900000', 'pdu_pf': 255, 'payload_bytes': ['ee', '1b', '08', '90', '00', '00'], 'timestamp': 1492805003.373471, 'priority': 3, 'header': '0cff1c00', 'pgn': 65308, 'source': 0, 'payload_int': 2416450542}

{'message': '14ff1017281c0000d400', 'pdu_pf': 255, 'payload_bytes': ['28', '1c', '00', '00', 'd4', '00'], 'timestamp': 1492805003.374843, 'priority': 5, 'header': '14ff1017', 'pgn': 65296, 'source': 23, 'payload_int': 910533073960}

{'message': '14ff11170003000000', 'pdu_pf': 255, 'payload_bytes': ['00', '03', '00', '00', '00'], 'timestamp': 1492805003.375422, 'priority': 5, 'header': '14ff1117', 'pgn': 65297, 'source': 23, 'payload_int': 768}

{'message': '14ff18056c88e83fcffffff', 'pdu_pf': 255, 'payload_bytes': ['6c', '88', 'e8', '3f', 'cf', 'ff', 'ff'], 'timestamp': 1492805003.377739, 'priority': 5, 'header': '14ff1805', 'pgn': 65304, 'source': 5, 'payload_int': 72057384656734316}

{'message': '0cff1c00d21b08900000', 'pdu_pf': 255, 'payload_bytes': ['d2', '1b', '08', '90', '00', '00'], 'timestamp': 1492805003.393459, 'priority': 3, 'header': '0cff1c00', 'pgn': 65308, 'source': 0, 'payload_int': 2416450514}

{'message': '14ff1505304410c3fcffff', 'pdu_pf': 255, 'payload_bytes': ['30', '44', '10', 'c3', 'fc', 'ff', 'ff'], 'timestamp': 1492805003.397692, 'priority': 5, 'header': '14ff1505', 'pgn': 65301, 'source': 5, 'payload_int': 72057580130681904}

{'message': '14000003c0aaaafaffffffff', 'pdu_pf': 0, 'payload_bytes': ['c0', 'aa', 'aa', 'fa', 'ff', 'ff', 'ff', 'ff'], 'timestamp': 1492805003.400851, 'priority': 5, 'header': '14000003', 'pgn': 0, 'source': 3, 'payload_int': 18446744073620073152}

{'message': '14ff0c031f718294047d', 'pdu_pf': 255, 'payload_bytes': ['1f', '71', '82', '94', '04', '7d'], 'timestamp': 1492805003.401424, 'priority': 5, 'header': '14ff0c03', 'pgn': 65292, 'source': 3, 'payload_int': 137458624917791}

{'message': '14ff2303ffffffffffffffff', 'pdu_pf': 255, 'payload_bytes': ['ff', 'ff', 'ff', 'ff', 'ff', 'ff', 'ff', 'ff'], 'timestamp': 1492805003.402011, 'priority': 5, 'header': '14ff2303', 'pgn': 65315, 'source': 3, 'payload_int': 18446744073709551615}

{'message': '14ff0e07000003f0ff', 'pdu_pf': 255, 'payload_bytes': ['00', '00', '03', 'f0', 'ff'], 'timestamp': 1492805003.403836, 'priority': 5, 'header': '14ff0e07', 'pgn': 65294, 'source': 7, 'payload_int': 1099243388928}

{'message': '14ff1307faf6401af3130', 'pdu_pf': 255, 'payload_bytes': ['fa', 'f6', '40', '1a', 'f3', '13'], 'timestamp': 1492805003.404396, 'priority': 5, 'header': '14ff1307', 'pgn': 65299, 'source': 7, 'payload_int': 21934838445818}

{'message': '14ff140700c030ff3cecec', 'pdu_pf': 255, 'payload_bytes': ['00', 'c0', '30', 'ff', '3c', 'ec', 'ec'], 'timestamp': 1492805003.40496, 'priority': 5, 'header': '14ff1407', 'pgn': 65300, 'source': 7, 'payload_int': 66687841227292672}

{'message': '14ff27070fcffffffffffff', 'pdu_pf': 255, 'payload_bytes': ['0f', 'cf', 'ff', 'ff', 'ff', 'ff', 'ff'], 'timestamp': 1492805003.40554, 'priority': 5, 'header': '14ff2707', 'pgn': 65319, 'source': 7, 'payload_int': 72057594037915407}

{'message': '14ff16057d7d7d7d6d3bb01', 'pdu_pf': 255, 'payload_bytes': ['7d', '7d', '7d', '7d', '6d', '3b', 'b0'], 'timestamp': 1492805003.407855, 'priority': 5, 'header': '14ff1605', 'pgn': 65302, 'source': 5, 'payload_int': 49604937343925629}

{'message': '14ff25030ffffffffffffff', 'pdu_pf': 255, 'payload_bytes': ['0f', 'ff', 'ff', 'ff', 'ff', 'ff', 'ff'], 'timestamp': 1492805003.410913, 'priority': 5, 'header': '14ff2503', 'pgn': 65317, 'source': 3, 'payload_int': 72057594037927695}

{'message': '14ff030367bf0ff6819fcff', 'pdu_pf': 255, 'payload_bytes': ['67', 'bf', '0f', 'f6', '81', '9f', 'cf'], 'timestamp': 1492805003.411471, 'priority': 5, 'header': '14ff0303', 'pgn': 65283, 'source': 3, 'payload_int': 58440700706930535}

{'message': '14ff2b03000000000', 'pdu_pf': 255, 'payload_bytes': ['00', '00', '00', '00'], 'timestamp': 1492805003.412068, 'priority': 5, 'header': '14ff2b03', 'pgn': 65323, 'source': 3, 'payload_int': 0}

{'message': '14ff0f070f433a00000', 'pdu_pf': 255, 'payload_bytes': ['0f', '43', '3a', '00', '00'], 'timestamp': 1492805003.41368, 'priority': 5, 'header': '14ff0f07', 'pgn': 65295, 'source': 7, 'payload_int': 3818255}

{'message': '14ff28077d7d7d7d7d111101', 'pdu_pf': 255, 'payload_bytes': ['7d', '7d', '7d', '7d', '7d', '11', '11', '01'], 'timestamp': 1492805003.414241, 'priority': 5, 'header': '14ff2807', 'pgn': 65320, 'source': 7, 'payload_int': 76861899315969405}

{'message': '0cff1c00df1b08900000', 'pdu_pf': 255, 'payload_bytes': ['df', '1b', '08', '90', '00', '00'], 'timestamp': 1492805003.41484, 'priority': 3, 'header': '0cff1c00', 'pgn': 65308, 'source': 0, 'payload_int': 2416450527}

[Yang Wang]

Hello,

The Python2 scripts kept breaking for me, so I tried translating to Python3. https://stackoverflow.com/questions/62833949/quick-way-to-view-and-manipulate-kafka-stored-data-isoblue-isobus

 

I now suspect I am not "Consuming" all the data even though I am using the offset setting of "earliest". I thought it would take longer to go though 1GB+ of data. 

It looks like you are consuming some data from the Kafka logs so that's good. It means that your Kafka broker is up and running along with your Zookeeper. Otherwise, your consumer script won't give you anything.

One thing that might cause your problem: in the Stack Overflow post, you set `group_id='can-test'` in configuring your KafkaConsumer. Once a group_id is given, a Kafka consumer will auto-commit its offset. Hence, even if you specify your `auto_offset_reset` to be `earliest`, your consumer might not be consuming from the earliest offset.

I am not recognising PGNs. I can seen some Address Claim PGNs and Propitiatory Messages (0x00EA00 and 0x00EF00) in "case-8010-wheat-ault-072018", but the data isn't as I expected.

 

replaying: "case-sprayer-wheat-ault-062017" I was expecting to see Process data PGN 0x00CB00, but didn't. I really want to see an object pool transfer sequence 0x00E700 and TP messages 0x00EC00 0x00EB00 or ETP messages

What PGNs are you looking for?  For object pool transfer and TP messages, these might be within the `imp` (implement bus) Kafka topic. From your Stack Overflow post, it looks like you are consume from the `tra` (tractor bus). Would you mind trying consuming from the `imp` topic?

Hope this helps.

Yang