Premature EOF error with List of List of struct implementation under VS2017/x64

[yojiro....@gmail.com]

Hello

I have been learning this wonderful software.

So I have been looking for an opportunity to use capnproto for my project,

and now I have it but I need to use it with MSVC 2017(recently updated to latest version 15.9.33), x64.

I only need serialization and no need for RPC or async.

This is my very first time posting a message to this community, please let me know if I'm not doing right here.

I have a schema whose basic construct is List of List of struct ( x,y points in 4d expressed by two level lists). Second level will have a few hundreds of entries and first level will have lots of entries.

And I am now stuck with a issue. If I put many entries in either of lists (more than a few hundreds) and write that to a file, then when I read the message, I get exception like this:

Any help would be very appreciated.

printSparseDir4D: std::exception: kj\io.c++:53: failed: expected n >= minBytes; Premature EOF

stack: 7ff672faac46 7ff672fa3fd6 7ff672fa2de5 7ff672fa1308 7ff672fb15d3 7ffc1aa97c23 7ffc1b18d720

Now I found that writing 4 entries in first level, two entries in second level (two x,y points each in 4 frames), I get the same error.

I'm putting my schema and my read/write code below. I'm using these codes with capnproto0.8.0. But actually this schema is a little striped version than the one I want to use. When I used that a little more complex schema, I faced similar issue with capnproto 0.7.0 and 0.8.0. 

I built capnproto following the steps in the website, had to skip heavy-tests project and kj-async (0.8.0), hoping the build was ok. 

Is there anything missing or done wrong? 

It would be very much appreciated for any help.

Thank you.

sparse4d.capnp :

# unique file ID, generated by `capnp id`

@0xae7dda29a38d355f;

struct Sparse4DHeader {

    tSize @0 :Int32;

    zSize @1 :Int32;

    ySize @2 :Int32;

    xSize @3 :Int32;

}

struct Dir4D {

    header @0 :Sparse4DHeader;

    planes @1 :List(SparsePlane);

    struct SparsePlane {

        t @0 :UInt16;

        z @1 :UInt16;

        points @2 :List(Point);

    }

}

struct Point {

    x @0 :UInt16;

    y @1 :UInt16;

    intensity @2 :Float32;

}

And read/write code:

/**

 * Copyright (c) 2021 ASTOM R&D

 * Author: Yojiro Nobukuni

 */

//#include "pch.h"

#include "../sparse4d.capnp.h"

#include <capnp/message.h>

#include <capnp/serialize-packed.h>

#include <iostream>

#include <ios>

#include <sys/types.h>

#include <sys/stat.h>

#include <fcntl.h>

//#include <unistd.h>

 /**

  * g++ writer.cpp photodeco_kernel.capnp.c++ -o writer -lcapnp -lkj

  * ./writer > test.message

  */

/**

 * Write PhotodecoKernel message to fd

 * @param int fd fd to read message in

 */

void writeSparse4D(int fd, unsigned int nofPlanes, unsigned int nofPoints, bool packed)

{

::capnp::MallocMessageBuilder message;

auto dir4d = message.initRoot<Dir4D>();

auto header = dir4d.initHeader();

header.setTSize(7);

header.setZSize(7);

header.setXSize(7);

header.setYSize(7);

auto planes = dir4d.initPlanes(nofPlanes);

for (auto plane : planes) {

plane.setT(9);

plane.setZ(8);

auto points = plane.initPoints(nofPoints);

#if 0

                // This one or the other one in #else block, same behavior.

for (unsigned int i = 0; i < nofPoints; i++) {

points[i].setX(1);

points[i].setY(2);

points[i].setIntensity(3);

}

#else

for (auto p : points) {

p.setX(1);

p.setY(2);

p.setIntensity(3);

}

#endif

}

try {

if (packed)

writePackedMessageToFd(fd, message);

else

writeMessageToFd(fd, message);

}

catch (std::exception& e) {

std::cout << "writeSparse4D: std::exception: " << e.what() << std::endl;

return;

}

}

/**

 * Print PhotodecoKernerl kernel in text

 * @param PhotodecoKernel::Reader kernel

 */

void printDir4D(Dir4D::Reader dir4d, bool printPoints)

{

auto header = dir4d.getHeader();

std::cout << "  sizes[T,Z,Y,X] = "

<< header.getTSize() << ", " << header.getZSize() << ", "  << header.getYSize() << ", " << header.getXSize() << ", "

<< std::endl;

//std::cout << "  nofPoints = " << header.getNofPoints() << std::endl;

if (!printPoints) return;

size_t nofPoints = 0;

auto planes = dir4d.getPlanes();

for (auto plane : planes) {

const auto t = plane.getT();

const auto z = plane.getZ();

std::cout << "plane(t,z): " << t << ", " << z << std::endl;

for (auto p : plane.getPoints()) {

std::cout << t << ", " << z << ", " << p.getY() << ", " << p.getX() << ", " << p.getIntensity() << std::endl;

nofPoints++;

}

}

std::cout << "Total nof KernelPoints = " << nofPoints << std::endl;

}

/**

 * Read PhotodecoKernerl message from fd and print it in text

 * @param int fd

 * @param bool packed

 */

void printSparseDir4D(int fd, bool printPoints, bool packed)

{

std::cout << "printSparseDir4D" << std::endl;

// readerOptions for reading message.

capnp::ReaderOptions readerOptions;

readerOptions.traversalLimitInWords = 1000 * 1000 * 1000 * readerOptions.traversalLimitInWords;

//readerOptions.nestingLimit;

::capnp::MessageReader *message;

try {

if (packed)

message = new ::capnp::PackedFdMessageReader(fd, readerOptions);

else

message = new ::capnp::StreamFdMessageReader(fd, readerOptions);

auto dir4d = message->getRoot<Dir4D>();

printDir4D(dir4d, printPoints);

}

catch (std::exception& e) {

std::cout << "printSparseDir4D: std::exception: " << e.what() << std::endl;

return;

}

delete message;

}

 

[yojiro....@gmail.com]

Hi

So, trying to understand what is happening.

using capnp.exe (32bit) in capnproto-tools-win32-0.8.0, message can be read correctly.

capnp.exe decode sparse4d.capnp Dir4D < myoutput.bin

By looking at hex dump of the message file, it looks like a proper message format.

So, writing new message is okay but reading it is not working properly.

Using capnproto-0.8.0 built with VS2017/x64 , and create a new message, save it to a file, and then reading it by my program fails.

Since all happens when I do message.getRoot<Dir4D>(),  this "Premature EOF" situation happens inside capnp library.

Is there any way to fix this? or to narrow down what is wrong?

Thank you.

2021年2月16日火曜日 4:56:44 UTC+9 yojiro....@gmail.com:

[yojiro....@gmail.com]

Hi

I could do a little more things.

Interestingly, capnp.exe built under x64 can do decode message.

I bet it does different processing than MessageReader's do and simply follow message format structure.

I made a test code using MapViewOfFile  and FlatArrayMessageReader,.

It can read messages that I couldnt by using StreamFdMessageReader.

But easily fails with a little large data, say 32 entries in first level List and 1 entry in the second level List,

getting following exception:

Exception: capnp\serialize.c++:51: failed: expected array.size() >= offset + segmentSize; Message ends prematurely in first segment.

stack: 7ff6a2af4b35 7ff6a2af3577 7ff6a2af14b6 7ff6a2b02323 7ffc1aa97c23 7ffc1b18d720

So I think I'm stuck. Any advice on how to proceed will be greatly appreciated.

Thank you in advance.

2021年2月16日火曜日 16:23:38 UTC+9 yojiro....@gmail.com:

[Kenton Varda]

Hi,

Sorry for the slow reply.

Just going to throw out a guess here, but seeing that you're using windows, did you make sure to use O_BINARY when opening the file descriptor? If you didn't, then the C library will silently corrupt your data by converting line endings.

If that's not it, then could you provide a complete self-contained program (with main() function and all parameters filled in) that demonstrates the problem?

-Kenton

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/2d1ae382-e739-4ad0-9e74-cbdcc0ab1a0fn%40googlegroups.com.

[yojiro....@gmail.com]

Hi  Kenton,

Glad to see your reply.

Yes I am aware of open mode. I do, for writing,

return _sopen_s(pfd, fpath.c_str(), (_O_BINARY | _O_CREAT | _O_RDWR | _O_TRUNC), _SH_DENYNO, (_S_IREAD | _S_IWRITE));

and for reading,

return _sopen_s(pfd, fpath.c_str(), (_O_BINARY | _O_RDWR), (_SH_DENYNO), (_S_IREAD | _S_IWRITE));

Using _O_WRONLY for writing, _O_RDONLY for reading did not change the situation.

For shared flag,  I have tried SH_DENYNO or SH_DENYRW but made no difference, as I had expected.

I can provide my code. I will send it to you when its ready, have to clean it a little I guess.

Thank you very much for the message.

2021年2月20日土曜日 1:55:08 UTC+9 ken...@cloudflare.com:

[yojiro....@gmail.com]

Hi Kenton

I prepared my source for sending to you.

Can I send it to your email address? or reply to this group with it as attachment?

Thank you.

2021年2月21日日曜日 20:09:41 UTC+9 yojiro....@gmail.com:

[Kenton Varda]

Hi Yojiro,

It would be best if you could reduce the problem to a minimal self-contained test case. If your program is multiple files then it is probably too big for us to look at. All I really need is for you to add a main() function to the code you sent before, so I can see exactly the code that demonstrates the problem.

-Kenton

To view this discussion on the web visit https://groups.google.com/d/msgid/capnproto/854ef4ca-17b6-4776-bd4d-03340c75dbbcn%40googlegroups.com.