Attempting to display gtfs realtime feeds on an Esp8266 mcu

[fixingthingsguy]

So I got to the point where I can receive data from GTFS NYCT feeds

__________________________________________________________________________________
#include "C:\Users\xx\Documents\Arduino\src\gtfs-realtime.pb.h"
#include "C:\Users\xx\Documents\Arduino\src\gtfs-realtime.pb.c"
#include "pb_common.h"
#include "pb_decode.h"
#include "pb.h"
#include <Arduino.h>
#include <Arduino_JSON.h>
#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>

#include <ESP8266HTTPClient.h>
#include <WiFiClientSecureBearSSL.h>
ESP8266WiFiMulti WiFiMulti;
 

void setup() {

  Serial.begin(115200);
  // Serial.setDebugOutput(true);

  Serial.println();
  Serial.println();
  Serial.println();

  for (uint8_t t = 4; t > 0; t--) {
    Serial.printf("[SETUP] WAIT %d...\n", t);
    Serial.flush();
    delay(1000);
  }

  WiFi.mode(WIFI_STA);
  WiFiMulti.addAP("xxxxx", "xxxx");
}

void loop() {
  size_t message_length;
  bool status;
  // wait for WiFi connection
  if ((WiFiMulti.run() == WL_CONNECTED)) {

    std::unique_ptr<BearSSL::WiFiClientSecure> client(new BearSSL::WiFiClientSecure);

    // bool mfln = client->probeMaxFragmentLength("tls.mbed.org", 443, 1024);
    // Serial.printf("\nConnecting to https://tls.mbed.org\n");
    // Serial.printf("Maximum fragment Length negotiation supported: %s\n", mfln ? "yes" : "no");
    //if (mfln) { client->setBufferSizes(1024, 1024); }

    Serial.print("[HTTPS] begin...\n");

    // configure server and url
    const uint8_t fingerprint[20] = { 0x15, 0x77, 0xdc, 0x04, 0x7c, 0x00, 0xf8, 0x70, 0x09, 0x34, 0x24, 0xf4, 0xd3, 0xa1, 0x7a, 0x6c, 0x1e, 0xa3, 0xe0, 0x2a };

    // client->setFingerprint(fingerprint);
    client->setInsecure();
    HTTPClient https;

    if (https.begin(*client, "https://api-endpoint.mta.info/Dataservice/mtagtfsfeeds/nyct%2Fgtfs-nqrw")) {  // HTTPS
      https.addHeader("x-api-key", "xxxxxxxxxxxxxxxxxxxxxxxxxx");
      https.addHeader("Content-Type", "application/json\r\n");
      Serial.print("[HTTPS] GET...\n");
      // start connection and send HTTP header
      // start connection and send HTTP header
      int httpCode = https.GET();

      if (httpCode == HTTP_CODE_OK) {
        Serial.println("Response body:");
        int len = https.getSize();
        Serial.print("Content length: ");
        Serial.println(len);
        char buffer[128];
        String gtfs_buffer = "";
        while (https.connected() && (len > 0 || len == -1)) {
          // Read data in chunks
          int size = std::min((int)sizeof(buffer) - 1, len);
          int bytesRead = client->readBytes(buffer, size);
          if (bytesRead == 0) {
            break;
          }
           buffer[bytesRead] = '\0';
          gtfs_buffer += buffer;
          if (len > 0) {
            len -= bytesRead;
          }
 
}


          Serial.println(gtfs_buffer);

        }
      }
    }
  }

}

The serial output looks like this 

X D
G2 G2 ! ⸮⸮ɟ ⸮⸮ɟ " R11N⸮>
G2 G2 ! ⸮⸮ɟ ⸮⸮ɟ " R60N⸮>
G2 G2 ! ⸮⸮ɟ ⸮⸮ɟ " R09N⸮>
G2 G2 ! ⸮⸮ɟ ⸮⸮ɟ " R08N⸮>
G2 G2 ! ⸮⸮ɟ ⸮⸮ɟ " R06N⸮>
G2 G2 ! ⸮⸮ɟ ⸮⸮ɟ " R05N⸮>
G2 G2 ! ⸮⸮ɟ ⸮⸮ɟ " R04N⸮>
G2 G2 ! ⸮⸮ɟ ⸮⸮ɟ " R03N⸮>
G2 G2 ! ⸮⸮ɟ ⸮⸮ɟ " R01N⸮>
G2 G2 X
000012N"M
;
074016_N..N 12:20:10 20230219* N⸮>

__________________________________________________________________________________

So, I am wondering if I am receiving what I should. There is no way I know to tell.

Note, that I ran the gtfs...proto file though nanopb and extracted the various

required as indicated in the includes.

So I assumed all that was correctly received (http =200, etc). And  I attemped a decode 

as below   after I got done receive the full data from my request

  // Initialize the message structure
transit_realtime_FeedMessage message = transit_realtime_FeedMessage_init_zero;

// Initialize the stream from the response buffer
pb_istream_t stream = pb_istream_from_buffer(buffer, bytesRead);

// Decode the stream in a loop
while (stream.bytes_left > 0) {
  // Decode the next message from the stream
  bool status = pb_decode(&stream, transit_realtime_FeedMessage_fields, &message);
  if (!status) {
    Serial.println("Error: Failed to decode GTFS-Realtime message");
    break;
  }

  // Process the decoded message
  // ...

  // Clear the message structure for the next message
  message = transit_realtime_FeedMessage_init_zero;
}

The result was a reset at the ESP8266. The most likely expl. is the ESP can't handle the size of the message ~20kbytes

Any suggestions to move forward appreciated. Including if there is a better way to present this code. 

Regards and thanks for your patience 

        

_______________________________________________________________________________

[Petteri Aimonen]

Hi,

> So, I am wondering if I am receiving what I should. There is no way I know
> to tell.

Modify your code to print a hex dump of the data, and you can use something
like this to check the data:
https://protobuf-decoder.netlify.app/

Or you can also run your nanopb decoding code on PC on that data to test.

> The result was a reset at the ESP8266. The most likely expl. is the ESP
> can't handle the size of the message ~20kbytes

Could be stack overflow. You can try putting "static" on the larger
allocations, such as the buffer and message struct.

Nanopb can also decode the message incrementally (storing only part at time in RAM),
but that requires using the callback API.

--
Petteri

[Sudhir Bijanki]

Thanks for the suggestions

> So, I am wondering if I am receiving what I should. There is no way I know
> to tell.

Modify your code to print a hex dump of the data, and you can use something
like this to check the data:
https://protobuf-decoder.netlify.app/

Or you can also run your nanopb decoding code on PC on that data to test.

I was able to verify that my junky looking output is binary data and manually ran

the data on my pc and it was correct. But I did not use nanopb as you suggested

(didn' read your comment carefully). However, it at least verifies I'm getting "good" data based

on the parsefromstring output. 

Nanopb can also decode the message incrementally (storing only part at time in RAM),
but that requires using the callback API.

Doesn't look that I have enough RAM based on observation of failure of the pb_decode response being False

So I will now attempt to use the callback API.

Thanks

--
You received this message because you are subscribed to the Google Groups "nanopb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nanopb+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nanopb/20230219202713.5hoaucqfqia5g42c%40lakka.kapsi.fi.