Help fixing cereal code

[Gene Massion]

I've been using Cereal since mid-last year quite nicely.  I have 2 C++ structs that are generated on a PC in my lab and transmitted via the Iridium satellite network to a float that profiles up and down in the ocean acquiring data about the health of the ocean.  I wrote a C++ program using Visual Studio to cerealize the C++ structs files in my lab. It also decerealizes the code to check itself.  I have C++ code running on the STM32H7A3 microcontroller in my ocean-going float that decerealizes the files back into C++ structs. It's all been working fine until I did something recently that broke the decerealization code on the float for one of the sctructs and I haven't been able to figure out why.  The other struct still decerealizes fine. I don't get an error message of any kind, the float code just jumps to the hardErrorHandler when it gets to the 

parseResult_ = reader.template Parse<parseFlags>(is, *this);

in this function.

    //! Parse JSON text from an input stream (with Encoding conversion)
    /*! \tparam parseFlags Combination of \ref ParseFlag.
        \tparam SourceEncoding Encoding of input stream
        \tparam InputStream Type of input stream, implementing Stream concept
        \param is Input stream to be parsed.
        \return The document itself for fluent API.
    */
    template <unsigned parseFlags, typename SourceEncoding, typename InputStream>
    GenericDocument& ParseStream(InputStream& is) {
        GenericReader<SourceEncoding, Encoding, StackAllocator> reader(
            stack_.HasAllocator() ? &stack_.GetAllocator() : 0);
        ClearStackOnExit scope(*this);
        parseResult_ = reader.template Parse<parseFlags>(is, *this);
        if (parseResult_) {
            CEREAL_RAPIDJSON_ASSERT(stack_.GetSize() == sizeof(ValueType)); // Got one and only one root object
            ValueType::operator=(*stack_.template Pop<ValueType>(1));// Move value from stack to document
        }
        return *this;
    }

Here's what I've checked so far.  

1. If I move the cereailzed files from the target to the PC, the PC code decerealizes it just fine.

2. The decerealizing code running on the float is the same for both structs.  One works for a simple struct but the more complicated struct with arrays in it has broken

3. The structs defined in the float code have exactly the same names and in the same order as the structs defined in the PC code.  Given that it looks like the cerealized files do not have variable names in them, does the name even matter?

What else can I check for?  I've attached the PC  CerealConsoleApplication and the files for decerializing running on the float and the call stack if anyone wants to look at the details.  

I recognize this is a detailed, complicated question.  Any help I can get will be appreciated.  Thanks

Thanks

[Rand Voorhies]

Hi Gene,

  I'm trying to understand the issue and reproduce on my machine.  It seems to work just fine on my laptop and produce reasonable results without crashing.  I'm assuming you're transferring the JSON serialized data over the wire, right?  If so, can you print out the serialized data from the float ARM microconotroller after it's transmitted to it?  That will help diagnose that at least it's getting transferred over the wire correctly.  Have you tried hardcoding e.g. the MissionDataJSON.crl into a string on the float code, and trying to deserialize it directly without dealing with transmitting it?

By the way, if you want to have properly named fields in your JSON serialized data, you need to use either the CEREAL_NVP macro or the cereal::make_nvp function (https://uscilab.github.io/cereal/serialization_archives.html), e.g.

struct ConfigData
{
std::string SerialNum;
std::string SWVerNum;
float recoveryBelPos;
float surfaceOpsBelPos;

template <class Archive>
void serialize(Archive& configFileArchive) {
configFileArchive(
CEREAL_NVP(SerialNum),
CEREAL_NVP(SWVerNum),
CEREAL_NVP(recoveryBelPos),
CEREAL_NVP(surfaceOpsBelPos)
);
}

};

struct DepthTable
{
float pressure_dBar;
float platformVel_dBarPerSec;
float parktime_hours;
uint16_t sampleList;

template <class Archive>
void serialize(Archive& missionFileArchive)
{
missionFileArchive(
CEREAL_NVP(pressure_dBar),
CEREAL_NVP(platformVel_dBarPerSec),
CEREAL_NVP(parktime_hours),
CEREAL_NVP(sampleList)
);
}
};

struct MissionDataStruct
{
uint16_t parkSamplePeriod_minutes;
float maxMissionDuration_hours;
uint16_t maxProfiles;
float maxPressure_dBar;
std::vector<DepthTable> descendTable;
std::vector<DepthTable> ascendTable;

template <class Archive>
void serialize(Archive& missionFileArchive)
{
missionFileArchive(
CEREAL_NVP(parkSamplePeriod_minutes),
CEREAL_NVP(maxMissionDuration_hours),
CEREAL_NVP(maxProfiles),
CEREAL_NVP(maxPressure_dBar),
CEREAL_NVP(descendTable),
CEREAL_NVP(ascendTable)
);
}
};

-- Rand

--
You received this message because you are subscribed to the Google Groups "cereal serialization library" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cerealcpp+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cerealcpp/305fa6c9-6d01-43c7-9551-3e57fb59928bn%40googlegroups.com.

[Gene Massion]

Hi Rand

 

Thanks for the quick response.  For this set of tests, I copied the JSON file directly from the PC hard drive to the float uSD card, nothing was transmitted over the wire.  The MissionDataJSONtest.crl file I attached to my help request is the file stored on the float uSD card and processed by the float code.  I’ve inspected it several times and diff’ed it and it appears identical to the file generated and tested on the PC.

 

Given that it works on your laptop, my next step is to modify the PC code and float code so the struct being cerealized is just one scalar variable.  If that works, I’ll add one variable at a time and eventually the arrays until something breaks.

 

And thanks for the note about _NVP.  I’ll take a look at that.

 

Thanks again

 

Gene Massion

he/him/his
Ocean Engineer
P (831) 775-1922   F (831) 775-1620

Monterey Bay Aquarium Research Institute 7700 Sandholdt Road, Moss Landing CA 95039 www.mbari.org Advancing marine science and engineering to understand our changing ocean.

[Randolph Voorhies]

Yeah that’s a great idea. Also, are you running the latest version on both the uC and your machine?

Sent from my iPhone

On Mar 18, 2024, at 9:19 AM, Gene Massion <mag...@mbari.org> wrote:



Hi Rand

 

Thanks for the quick response.  For this set of tests, I copied the JSON file directly from the PC hard drive to the float uSD card, nothing was transmitted over the wire.  The MissionDataJSONtest.crl file I attached to my help request is the file stored on the float uSD card and processed by the float code.  I’ve inspected it several times and diff’ed it and it appears identical to the file generated and tested on the PC.

 

Given that it works on your laptop, my next step is to modify the PC code and float code so the struct being cerealized is just one scalar variable.  If that works, I’ll add one variable at a time and eventually the arrays until something breaks.

 

And thanks for the note about _NVP.  I’ll take a look at that.

 

Thanks again

 

Gene Massion

he/him/his
Ocean Engineer
P (831) 775-1922   F (831) 775-1620

<image001.gif>

<image002.jpg>

[Gene Massion]

I’ll check about the versions but the fact one .crl file works and it’s just the more complicated one that doesn’t suggests the libraries are compatible.

 

Cheers

 

Gene Massion

he/him/his
Ocean Engineer
P (831) 775-1922   F (831) 775-1620

Monterey Bay Aquarium Research Institute

[Gene Massion]

New clue: I think I mentioned I have 2 different structs that I decerealize.  One is pretty simple, the other is just a little more complicated in that it has 2 std::vectors in it.  Whichever stuct I decerealize first, works and the second one fails.   Seems like I’m not disposing of something properly but I sure can’t find it.

 

I’ve pasted the code here.  There’s a lot of it but 90% of it is just printing out the decerealized stucts.  There’s actually just a few lines of cereal code.  Here’s the function that gets called to decerealize the MissionDataStruct.  The decerealization is done at the top with MissionData::readMissionData().  Everything below that just prints out the values in the struct.

 

InitPlatformState::Actions InitPlatformState::doReadMissionFile(const StateEvent event)

{

      std::string comment;

      char temp[64];

 

      {

           MissionData::readMissionData();

      }

 

      comment.clear();

      comment.append("Park sample period (minutes): ");

      snprintf(temp, sizeof(temp), "%6d", MissionData::getParkSamplePeriod_minutes());

      comment.append(temp);

      smLogEngLine(comment);

     

      comment.clear();

      comment.append("Max mission duration (hours): ");

      snprintf(temp, sizeof(temp), "%8.4f", MissionData::getMaxMissionDuration_hours());

      comment.append(temp);

      smLogEngLine(comment);

     

      comment.clear();

      comment.append("Max num of profiles: ");

      snprintf(temp, sizeof(temp), "%4d", MissionData::getMaxProfiles());

      comment.append(temp);

      smLogEngLine(comment);

     

      comment.clear();

      comment.append("Max Pressure (dBar): ");

      snprintf(temp, sizeof(temp), "%7.1f", MissionData::getMaxPressure_dBar());

      comment.append(temp);

      smLogEngLine(comment);

     

      comment.clear();

      comment.append("Line Num \tpressure (dBar)\tplatform velocity (dBar/sec)\tparktime (hours)\tsampleList");

      smLogEngLine(comment);

 

      MissionData::DepthTable tableEntry;

      std::string direction;

 

      uint16_t numEntrys  = MissionData::getSizeOfDescendTable();

      for (int j = 0; j < 2; j++)

      {

           for (int i = 0; i < numEntrys; i++)

           {

                 if (j == 0)

                 {

                      tableEntry = MissionData::getDescendTableEntry(i);

                      direction = "Descend ";

                 }

                 else

                 {

                      tableEntry = MissionData::getAscendTableEntry(i);

                      direction = "Ascend ";

                 }               

                 comment.clear();

                 snprintf(temp,

                      sizeof(temp),

                      "%s%d\t%7.1f\t%6.3f\t%8.4f\t%#4X",

                      direction.c_str(),

                      i,

                       tableEntry.pressure_dBar,

                       tableEntry.platformVel_dBarPerSec,

                       tableEntry.parktime_hours,

                      tableEntry.sampleList);

                 comment.append(temp);

                 smLogEngLine(comment);

           }

           numEntrys = MissionData::getSizeOfAscendTable();

      }    

     

      return initCTD1;

}

 

Here’s readMissionData().

 

void MissionData::readMissionData()

{

      // TODO: what to do if file doesn't exist?

      // should this fcn return a bool to indicate success?

      std::stringstream missionFileStream;

     

      missionFileStream = FileSys::readFile("/MissionDataJSON.crl");

 

      cereal::JSONInputArchive iarchive(missionFileStream);

      iarchive(cpfMissionData);

}

 

Here’s the function that gets called to decerealize the ConfigDataStruct.  The decerealization is done at the top with ConfigData::readConfigData. Again, everything below ConfigData::readConfigData() is just printing out the struct.

 

InitPlatformState::Actions InitPlatformState::doReadConfigFile(const StateEvent event)

{

      std::string comment;

      char temp[8];

 

      {

           ConfigData::readConfigData();

      }

 

      comment.clear();

      comment.append("Config file SN: ");

      comment.append(ConfigData::getSN());

      Logger::logEngLine(true, { stmchEngLogID, RTClock::getDateTimestamp().data(), stateLabel, comment });

 

      comment.clear();

      comment.append("Config file SW Ver Num: ");

      comment.append(ConfigData::getSWVerNum());

      Logger::logEngLine(true, { stmchEngLogID, RTClock::getDateTimestamp().data(), stateLabel, comment });

 

      comment.clear();

      comment.append("Config file Surf Ops Bel Pos (pctFull): ");

      snprintf(temp, sizeof(temp), "%5.2f", ConfigData::getSurfaceOpsBelPos_pctFull());

      comment.append(temp);

      Logger::logEngLine(true, { stmchEngLogID, RTClock::getDateTimestamp().data(), stateLabel, comment });

 

      comment.clear();

      comment.append("Config file Recovery Bel Pos (pctFull): ");

      snprintf(temp, sizeof(temp), "%5.2f", ConfigData::getRecoveryBelPos_pctFull());

      comment.append(temp);

      Logger::logEngLine(true, { stmchEngLogID, RTClock::getDateTimestamp().data(), stateLabel, comment });

 

      return readMissionFile;

}

 

Here’s readConfigData()

 

void ConfigData::readConfigData()

{    

      // TODO: need a cpfAssert or recovery or default to safe configData file?

      std::stringstream configFileStream;

     

      configFileStream = FileSys::readFile("/ConfigDataJSON.crl");

 

      cereal::JSONInputArchive iarchive(configFileStream);

      iarchive(cpfConfigData);

     

      std::cout << "ConfigData.SerialNum: " << getSN() << std::endl;

      std::cout << "ConfigData.SWVerNum: " << getSWVerNum() << std::endl;

      std::cout << "ConfigData.recoveryBelPos_ptcFull: " << getRecoveryBelPos_pctFull() << std::endl;

      std::cout << "ConfigData.SurfaceOpsBelPos_pctFull: " << getSurfaceOpsBelPos_pctFull() << std::endl;

}

 

Thanks again for all your help.

 

Gene Massion

From: Gene Massion [mailto:mag...@mbari.org]
Sent: Monday, March 18, 2024 9:31 AM
To: Randolph Voorhies <rand.v...@gmail.com>
Cc: cereal serialization library <cere...@googlegroups.com>

Subject: RE: [cerealcpp] Help fixing cereal code

 

I’ll check about the versions but the fact one .crl file works and it’s just the more complicated one that doesn’t suggests the libraries are compatible.

 

Cheers

 

Gene Massion

he/him/his
Ocean Engineer
P (831) 775-1922   F (831) 775-1620

Monterey Bay Aquarium Research Institute

7700 Sandholdt Road, Moss Landing CA 95039 www.mbari.org Advancing marine science and engineering to understand our changing ocean.

 

 

 

 

 

From: Randolph Voorhies <rand.v...@gmail.com>

Sent: Monday, March 18, 2024 9:26 AM

[Rand Voorhies]

Hey Gene,

> Whichever stuct I decerealize first, works and the second one fails

This sounds like the biggest clue.  Can you try to reduce your code to a single .cpp file that crashes? Ideally, just hardcode the serialized JSON as const strings at the top so it's all self-contained.  If you send that, then I can help by running it through a gdb / valgrind / whatever to see what's up.

-- Rand