Running YAMCS with Space DataLink Frames

[paul madle]

Hi All,

I am very new to YAMCS and wanted to use it as a basis for some on board software that I need to write.

The Getting Started example has proven very helpful and I've evaluated that the command and telemetry frames that are generated by ./simulator.py are both in a format specified by the Space Packet Protocol (CCSDS 133.0-B-2). This is great but I need to be dealing with TM/TC frames as specified in TM/TC Space Data Link Protocol (CCSDS CCSDS 132.0-B-2 & CCSDS 232.0-B-3). 

Looking through the documentation, it seems that this is all possible so I started by adding the following to the yamcs.myproject.yaml file (under "datalinks:") to start YAMCS recognising these frames:

  - name: UDP_FRAME_IN
    class: org.yamcs.tctm.ccsds.UdpTmFrameLink
    port: 10017
    frameType: "TM"
    spacecraftId: 0xAB
    frameLength: 512
    frameHeaderErrorControlPresent: true
    insertZoneLength: 0
    errorDetection: CRC16
    clcwStream: clcw
    goodFrameStream: tm_realtime
    badFrameStream: tm_realtime
    virtualChannels:
      - vcId: 0
        ocfPresent: true
        service: "PACKET"
        maxPacketLength: 2048
        stripEncapsulationHeader: true
        packetPreprocessorClassName: org.yamcs.tctm.GenericPacketPreprocessor
        packetPreprocessorArgs:
          timestampOffset: 10
          seqCountOffset: 20  
          errorDetection:
            type: CRC-16-CCIIT
          byteOrder: BIG_ENDIAN
          timeEncoding:
            type: CUC
            epoch: CUSTOM
            epochUTC: "2010-09-01T00:00:00Z"
            timeIncludesLeapSeconds: true
    stream: "tm_realtime"

I then hoped that the connection to the stream for good and bad frames ("tm_realtime") would result in data being passed to this server stream. 

I then modified the "simulator.py" files so that the telemetry frame is sent on both port 10015 (the default for myproject) and 10017 (the new port I have used for the TM frame above). 

I had hoped that I'd then be able to see on the YAMCS client some indication of good (or even bad) packets arriving and being interpreted. Sadly, the 10017 UDP packets coming from simulator.py seem to be being totally ignored by YAMCS (at least, nothing obvious on the web client). I can def see the 10017 UDP packets using wireshark. 

Am I following the right steps? Is there something missing?

Many thanks,

Paul Madle

Senior Software Engineer, KISPE Space Systems, UK.  

[Nicolae Mihalache]

(sorry I missed your message, it  went to spam like some other mails from this list. very weird...)

You can see here a working example of ccsds frames:
https://github.com/yamcs/yamcs/tree/master/examples/ccsds-frames

 It uses AOS frames, not TM.

You can enable debugging by adding this line in logging.properties (in the etc directory):

org.yamcs.tctm.ccsds.level = ALL

That should make it print all incoming frames.

Otherwise, did you actually look at the tuple counts on the good and bad frames streams?

You can see that in the admin area -> databases ->streams  or directly at the URL http://localhost:8090/admin/databases/myproject/streams

Note that the simulator.py sends packets.

When switching to a frame protocol you still need the packets but embedded in frames.  The theory is that the frames are fixed size (required for optimizing the downlink) whereas the packets are variable size (as generated by the spacecraft/payload). Decoding variable size packets out of fixed size frames is what those frame links do in Yamcs. 

My guess is that if you did not pay attention to proper packet encoding in frames, all the decoding will fail.

--
You received this message because you are subscribed to the Google Groups "yamcs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to yamcs+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/yamcs/fb2fcd2b-da1f-4323-9742-2d71df146e1cn%40googlegroups.com.

[paul madle]

Thanks very much Nicolae,

That is helpful.... I will take a look at the tuple count.

My investigatory work took me to the ccsds-frames example on your main yamcs repo (not the getting started repo).

I am having problems attempting to get this example going. I have followed the following steps:

clone repo

install JDK8, maven & npm

Built Java jars

Built web interface

now running command 

sudo ./run-example.sh ccsds-frames

It is here that I am getting some errors:

[WARNING] The POM for org.yamcs:simulator:jar:5.5.6-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.yamcs:yamcs-core:jar:5.5.6-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.yamcs:yamcs-web:jar:5.5.6-SNAPSHOT is missing, no dependency information available
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.260 s
[INFO] Finished at: 2022-01-17T14:05:51Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project ccsds-frames: Could not resolve dependencies for project org.yamcs.examples:ccsds-frames:jar:5.5.6-SNAPSHOT: The following artifacts could not be resolved: org.yamcs:simulator:jar:5.5.6-SNAPSHOT, org.yamcs:yamcs-core:jar:5.5.6-SNAPSHOT, org.yamcs:yamcs-web:jar:5.5.6-SNAPSHOT: Could not find artifact org.yamcs:simulator:jar:5.5.6-SNAPSHOT

Can you help? 

Many thanks,

Paul Madle

[Nicolae Mihalache]

Don't use sudo to run it because that will look for the jars in the ~root/.m2 directory whereas you have installed them in the ~user/m2.

Just use as a user ./run-example ccsds-frames.

Yamcs listens by default to port 8090 so no special privileges required.

nicolae

To view this discussion on the web visit https://groups.google.com/d/msgid/yamcs/ff6a3622-b11b-4883-9b56-f8e134acf023n%40googlegroups.com.

[paul madle]

Hi Nicholae,

Thanks again for the swift feedback. 

I tried to run the command using normal privledges: 

./run-example ccsds-frames

I am now getting a different error:

14:47:13.747 _global [1] YamcsServer Loading service HttpServer
14:47:15.113 _global [1] RdbStorageEngine Creating or loading tablespace _global
14:47:15.159 _global [1] stdout java.io.IOException: org.rocksdb.RocksDBException: while open a file for lock: /storage/yamcs-data/_global.rdb//LOCK: Permission denied
14:47:15.163 _global [1] stdout         at org.yamcs.yarch.rocksdb.RDBFactory.rdb(RDBFactory.java:106)
14:47:15.163 _global [1] stdout         at org.yamcs.yarch.rocksdb.RDBFactory.getRdb(RDBFactory.java:63)
14:47:15.163 _global [1] stdout         at org.yamcs.yarch.rocksdb.Tablespace.loadDb(Tablespace.java:143)
14:47:15.164 _global [1] stdout         at org.yamcs.yarch.rocksdb.RdbStorageEngine.createTablespace(RdbStorageEngine.java:184)
14:47:15.165 _global [1] stdout         at org.yamcs.yarch.rocksdb.RdbStorageEngine.getTablespace(RdbStorageEngine.java:157)
14:47:15.165 _global [1] stdout         at org.yamcs.yarch.rocksdb.RdbStorageEngine.loadTables(RdbStorageEngine.java:289)
14:47:15.165 _global [1] stdout         at org.yamcs.yarch.YarchDatabaseInstance.loadTables(YarchDatabaseInstance.java:182)
14:47:15.166 _global [1] stdout         at org.yamcs.yarch.YarchDatabaseInstance.<init>(YarchDatabaseInstance.java:97)
14:47:15.166 _global [1] stdout         at org.yamcs.yarch.YarchDatabase.getInstance(YarchDatabase.java:95)
14:47:15.166 _global [1] stdout         at org.yamcs.security.Directory.<init>(Directory.java:49)
14:47:15.166 _global [1] stdout         at org.yamcs.security.SecurityStore.<init>(SecurityStore.java:77)
14:47:15.180 _global [1] stdout         at org.yamcs.YamcsServer.addGlobalServicesAndInstances(YamcsServer.java:1281)
14:47:15.181 _global [1] stdout         at org.yamcs.YamcsServer.prepareStart(YamcsServer.java:1126)
14:47:15.181 _global [1] stdout         at org.yamcs.YamcsServer.main(YamcsServer.java:928)
14:47:15.182 _global [1] stdout Caused by: org.rocksdb.RocksDBException: while open a file for lock: /storage/yamcs-data/_global.rdb//LOCK: Permission denied
14:47:15.182 _global [1] stdout         at org.rocksdb.RocksDB.open(Native Method)
14:47:15.182 _global [1] stdout         at org.rocksdb.RocksDB.open(RocksDB.java:286)
14:47:15.182 _global [1] stdout         at org.yamcs.yarch.rocksdb.YRDB.<init>(YRDB.java:99)
14:47:15.182 _global [1] stdout         at org.yamcs.yarch.rocksdb.RDBFactory.rdb(RDBFactory.java:103)
14:47:15.183 _global [1] stdout         ... 13 more
14:47:15.185 _global [1] RdbStorageEngine Cannot load tablespace _global: java.io.IOException: org.rocksdb.RocksDBException: while open a file for lock: /storage/yamcs-data/_global.rdb//LOCK: Permission denied

I attempted to remove the LOCK file at location /storage/yamcs-data/_global.rdb// but no joy... I get the same error.

Any suggestions are welcome :)

Many thanks,

Paul.

[Nicolae Mihalache]

That's maybe because you have the /storage directory owned by root. 

You change owner with:

sudo chown -R $USER /storage 

To view this discussion on the web visit https://groups.google.com/d/msgid/yamcs/c0e6b883-e58a-481a-85af-2933910fc3efn%40googlegroups.com.

[paul madle]

Thats fantastic and has resolved this issue... Brilliant! the server now appears to be running. However, I am getting the 404 error on my browser when I navigate to the page linked to the local server on my machine... I think this is because the web interface is not building correctly... 

I am getting this:

> @yamcs/web...@1.0.0 build
> ng build

Node.js version v17.3.1 detected.
Odd numbered Node.js versions will not enter LTS status and should not be used for production. For more information, please see https://nodejs.org/en/about/releases/.
⠴ Generating browser application bundles (phase: building)...node:internal/crypto/hash:67
  this[kHandle] = new _Hash(algorithm, xofLen);
                  ^

Error: error:0308010C:digital envelope routines::unsupported
    at new Hash (node:internal/crypto/hash:67:19)
    at Object.createHash (node:crypto:130:10)
    at BulkUpdateDecorator.hashFactory (/home/paul/Documents/yamcs/full_yamcs/yamcs-web/src/main/webapp/node_modules/webpack/lib/util/createHash.js:145:18)
    at BulkUpdateDecorator.update (/home/paul/Documents/yamcs/full_yamcs/yamcs-web/src/main/webapp/node_modules/webpack/lib/util/createHash.js:46:50)
    at RawSource.updateHash (/home/paul/Documents/yamcs/full_yamcs/yamcs-web/src/main/webapp/node_modules/webpack/node_modules/webpack-sources/lib/RawSource.js:77:8)
    at NormalModule._initBuildHash (/home/paul/Documents/yamcs/full_yamcs/yamcs-web/src/main/webapp/node_modules/webpack/lib/NormalModule.js:880:17)
    at handleParseResult (/home/paul/Documents/yamcs/full_yamcs/yamcs-web/src/main/webapp/node_modules/webpack/lib/NormalModule.js:946:10)
    at /home/paul/Documents/yamcs/full_yamcs/yamcs-web/src/main/webapp/node_modules/webpack/lib/NormalModule.js:1040:4
    at processResult (/home/paul/Documents/yamcs/full_yamcs/yamcs-web/src/main/webapp/node_modules/webpack/lib/NormalModule.js:755:11)
    at /home/paul/Documents/yamcs/full_yamcs/yamcs-web/src/main/webapp/node_modules/webpack/lib/NormalModule.js:819:5 {
  opensslErrorStack: [ 'error:03000086:digital envelope routines::initialization error' ],
  library: 'digital envelope routines',
  reason: 'unsupported',
  code: 'ERR_OSSL_EVP_UNSUPPORTED'

Do I need a specific version of Nodejs?

Many thanks,

Paul.

[fabia...@spaceapplications.com]

Hi, yes this error seems related to node 17.

If you can, please use node 14 or 16. Both are LTS versions. I myself use 14.

[paul madle]

Thanks Fabia & Nicolae,

I now have the ccsds-frames example up and running.

[paul madle]

Ok, 

2 questions for you:

1. How is the simulated telemetry generated? Is there a python script somewhere? I can see the class org.yamcs.simulator.SimulatorCommander in the yamcs.ccsds-frames.yaml file... I am guessing this has something to do with it?

2. I now need to switch from AOS to TM frames... Is this an easy transition?... I am starting by changing the instances of "AOS" to "TM" in the yamcs.ccsds-frames.yaml code... I am guessing I'll then get some errors at build time that I'll attempt to fix. Do you have any wisdom that may help?

Many thanks,

Paul. 

[paul madle]

Ok, I have now done this very simple change and I can see TM frames now flying around... Harrah... When I look at the UDP traffic using wireshark, I can decode the TM frames in the UDP packets... very happy... I now have something to base on board code on.

[Nicolae Mihalache]

1. the simulator is implemented in  java,  you can see the code here:

https://github.com/yamcs/yamcs/blob/master/simulator/src/main/java/org/yamcs/simulator/UdpTmFrameLink.java

2. AOS and TM frames have different headers but in the configuration there isn't much difference, practically you can just change the frameType property from AOS to TM like you did. There are some restrictions with the number of virtual channels (AOS allows more than TM) but not fundamental differences.

You can read about it here:

https://docs.yamcs.org/yamcs-server-manual/links/ccsds-frame-processing/

That page is generated out of this:

https://github.com/yamcs/yamcs/blob/master/docs/server-manual/links/ccsds-frame-processing.rst

You can make a pull request if you have some suggestions (or find typos, etc).

To view this discussion on the web visit https://groups.google.com/d/msgid/yamcs/7d855590-249c-4a39-a45b-555878db4e14n%40googlegroups.com.

[paul madle]

Fantastic,

I am now ensuring that telecommand frames look as I expect too... I have captured the UDP traffic coming from the yamcs server when I send a telecommand and am managing to decode what looks like TC frames from yamcs (this is how I confirmed the TM frame format)...

Comparing the data I am getting in the UDP data field to the spec (CCSDS 232.0-B-4), I can see an extra 2 bytes prior to the CCSDS TC frame (values 0xEB 0x90). I can't really figure out what these extra bytes are for? Am I misunderstanding something?

Many thanks,

Paul.

[Nicolae Mihalache]

Maybe because you are looking at CLTUs not frames. Try commenting out this line:

#cltuEncoding: BCH

To view this discussion on the web visit https://groups.google.com/d/msgid/yamcs/bbf2157a-8adc-4237-9fac-2131ad461e62n%40googlegroups.com.

[paul madle]

Hi Nicolae, 

As always, you are correct... Commenting this out gets rid of the first 2 bytes (From looking at it, this BCH encoding forms part of the CCSDS 231.0-B-3 "TC Synchronization and Channel Coding."), this is below the TC Space Data Link protocol and provides forwards error correction.

Magic.

Paul.

[Marcin Wilk]

Hello,

I am trying to follow Paul steps. I managed to run  ccsds-frames example. Also

- I changed in file yamcs.ccsds-frames.yaml frame type from AOS to TM

- Comment out cltuEncoding: BCH

Now when I am sending TC from web server I can see on console frame that I was expecting. There is 5 octets of "Transfer Frame Primary Header" + "Transfer Frame Data Field". What I can not see is "Frame Error Control Field". This field is optional but it would help my testes a lot if I could also have error control.

Is there any setting I could change to see also 2 bytes of error control in the end of the frame?

BR,

Marcin

[Nicolae Mihalache]

Did you try specifying:

  errorDetection: CRC16

?

To view this discussion on the web visit https://groups.google.com/d/msgid/yamcs/780e5ff8-8351-42f2-90a1-48211730b236n%40googlegroups.com.