Byte length in streamsql files

[mbe...@windhoverlabs.com]

I'm trying to use the CFDP plugin.  This is the streamsql file I was recommended to use:

create stream cfdp_in as select substring(packet, 6) as pdu from tm_realtime where extract_short(packet, 0) = 6141
create stream cfdp_out (gentime TIMESTAMP, entityId long, seqNum int, pdu  binary)
insert into tc_realtime select gentime, 'cfdp-service' as origin, seqNum, '/yamcs/cfdp/upload' as cmdName, unhex('17FDC0000000') + pdu as binary from cfdp_out

However, this is making several assumptions that may be application prohibitive, and I would like to try to submit the fix myself.

1)  This is assuming the outgoing stream is CCSDS, and this specifies that there is no secondary header.  In my case, my remote entity requires a secondary header, but that shouldn't be a problem.  My secondary command header is fixed anyway, so that's an easy one to include in this sql query.

2)  The primary header includes message length, but this sets that field to 0.  This would still work if the remote entity ignores the length field.  What I would like to do is basically:

unhex('17FDC000') + size(pdu) + pdu     <-- Basically generate a 4 byte padded size field

I don't know how to get the size of the binary type.  Does the streamsql API have a method to do this, or do I need to add an expression?

Furthermore, is this really the best solution anyway?  Or should I create a plugin that takes messages from one stream, sticks a primary and secondary header on it, and writes it to the realtime stream?  Basically do this in code, rather than in a streamsql.  What's the best solution?

[mbe...@windhoverlabs.com]

Or should I just update my TC provider to correct it?

[Nicolae Mihalache]

The streamsql file you see in the cfdp example is mission specific. You can change it as you wish for your particular mission or not use it at all.

In the example I think there is a post-processor which will insert the correct packet length and also fill the sequence count (same for the normal TCs because the XTCE command description also does not generate a packet length or sequence count).

Of course it's possible to write some java code to do the same thing but that will also be mission specific. You can have a data link which reads/writes directly to/from the CFDP streams, I guess one particular mission could embed CFDP PDUs directly into CCSDS frames somehow.

If you want, you can improve the documentation to make it more clear.

--
You received this message because you are subscribed to the Google Groups "yamcs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to yamcs+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/yamcs/8c4269b7-008c-4c95-aa45-bbf49e646653n%40googlegroups.com.

[mbe...@windhoverlabs.com]

I figured it was mission specific.  Can you point me to the post-processor configuration and code? 

[mbe...@windhoverlabs.com]

Is it "IssCommandPostprocessor"?

[Nicolae Mihalache]

the one that is used in that cfdp example is here:

https://github.com/yamcs/yamcs/blob/master/examples/cfdp/src/main/yamcs/etc/yamcs.cfdp.yaml#L74

two bits about post-processors in general are here:

https://docs.yamcs.org/yamcs-server-manual/links/command-post-processor/

any documentation (or other) improvement is always welcome.

To view this discussion on the web visit https://groups.google.com/d/msgid/yamcs/d49bd940-84aa-4371-bd4f-88a1c1ac6f0dn%40googlegroups.com.

[mbe...@windhoverlabs.com]

That makes sense now.  I added the sequence counts in my TC provider, but it makes a lot more sense to do that in a post processor.  How does the command get to the post processor and how does the processed command get to the TC provider? 

[mbe...@windhoverlabs.com]

I think I see now.  Is the post processor between the stream and the TC provider and the pre processor between the stream and the TM provider?  Basically, do commands get published to the stream, processed through the post processor and then sent to the TC provider?

[Nicolae Mihalache]

sorry, what do you call the TC provider? 

A TC data link?

please have a look at this diagram:

https://docs.yamcs.org/yamcs-server-manual/general/architecture/

To view this discussion on the web visit https://groups.google.com/d/msgid/yamcs/118b89db-9d64-4fef-a20b-81abf31fca7en%40googlegroups.com.

[Nicolae Mihalache]

Something like that. We have been turning these things around for some years trying to make something re-usable in various scenarios but I think we settled now for this schema.

[mbe...@windhoverlabs.com]

I like it.  I noticed you already have a CFS post processor.  I added that in and it works just fine.  Thanks.  There's a bug with the post processor, but I'll submit a fix and a merge request.  The primary header has always been defined as big endian, but the secondary header has changed a bit throughout the years.  The checksum and command code were flipped.  I'll make those configurable parameters to remove dependency on the CFS version.

[mbe...@windhoverlabs.com]

Correction.  There is not a bug.  I didn't notice the "swapChecksumFc" argument.  That fixes it.