Running EW on industrial radios

[Victor Preatoni]

Hi group,

Just wanted to share an application note we wrote a few weeks ago, about compiling and running EW on embedded ARM Linux UHF radios like XetaWave EDGE or ZumLink ZumIQ.

Publication is in Spanish, sorry, but google translator will do a good job.

File attached to this message. Otherwise, you can access online at:

https://repositorio.segemar.gov.ar/handle/308849217/4270

If translation does not work well, I can rewrite document in English.

Thanks,

Victor

[Paul Friberg]

Very nicely done Victor. Thanks for sharing your paper. I see you are using the systemctl to start/stop earthworm, Bravo!

Are you happy with the Xeta Wave radios for this application?

We build similar concentrators, but use industrial computers with SSD drives to store the data locally in a winston and forward it to the front-line EW processor, just like you do using Chad's ringserver, and all run from within Earthworm as you did. For most of our applications we have been using cellular modems and have only had to rely on point to point radios for remote sites like you are doing.

Paul

--
--
You received this message because you are subscribed to the Google
Groups "Earthworm Community Forum" group.
 
To post to this group, send an email to earthwo...@googlegroups.com
 
To unsubscribe from this group, send an email to
earthworm_for...@googlegroups.com
 
For more options, visit this group at
http://groups.google.com/group/earthworm_forum?hl=en

---
You received this message because you are subscribed to the Google Groups "Earthworm Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to earthworm_for...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/earthworm_forum/e29eb307-a79b-41e4-ad71-2539910fe16en%40googlegroups.com.

--

===================================
Paul Friberg   p.fr...@isti.com
CEO/Seismologist
ISTI==Instrumental Software Technologies, Inc.
Phone +1.518.602.0001  

[Victor Preatoni]

Hi Paul,

Software installation went straight-forward. I've been reading the group messages and I saw some issues when compiling on RaspberryPi, but was not the case on XetaEdge.

CPU usage is really low!. I've not tested any processing modules, but the main idea was to use XetaEdge as a SeedLink buffer.

Regarding reliability, time will tell.... I expect the embedded Debian is doing some write balancing to spread Flash wear across the full 4GB capacity, but who knows... A solution could be mounting file system as read-only, and create a RAM disk to store logs and Seedlink buffer, with the risk of loosing data on power outages. Maybe that will be next step if we detect a premature Flash failure.

Victor

[Renate Hartog]

Hi Victor,

That is very interesting. Mind if I ask a bit about how you use this in the field? I assume the purpose would be to send picks rather than waveforms? Or do you have another use case?

Thanks! Renate

To view this discussion on the web visit https://groups.google.com/d/msgid/earthworm_forum/CALsXm6UkD4i8KdsHTHkkH24_gqh%3DsUUQFBc0ttWRnSyzwpEAnA%40mail.gmail.com.

[Victor Preatoni]

Not really for picks. It acts as a SeedLink cache.

On volcano monitoring networks, there usually are quite a few seismometers distributed along the volcano, interconnected using low bandwidth UHF radios , and then reporting 24/7 across a satellite link to our datacenter. The telemetry network from sensors to satellite antena is really slow. So, when the satellite link comes down, and up again after a few hours, days, or even months, slink2ew module has the ability to request old data. That is a great feature, but requesting old data from each station saturates the telemetry bandwidth. So, having a local cache that will be able to continue gathering data even when the sat link is down, will save us that precious telemetry bandwidth, since when the link comes up, slink2ew will ask for older data to the local seedlink server instead of asking each of the seismometers.

Hope I was clear enough :-P

Victor

[Larry Baker]

Victor and others,

I read your application note for the installation of Earthworm in the XetaEdge9 edge computer.  I wanted to pass along my experience using ARM SoC and SBC embedded systems for over ten years, including for Earthworm.

I am the one who added ARM support to Earthworm in 2018 for the 7.10 release.  There is an email thread in this mailing list from October 11, 2017, named "Raspberry Pi 3" where I described what I had done at that time to run Earthworm on ARM processors.  Earthworm 7.10 was not released until almost two years after that, on August 20, 2019.

I tried to find out exactly what processor and motherboard you are using, but the XetaWave web site has not much useful technical information.  All I could learn was that the XetaWave9 has a 1GHz ARM Cortex-A8 processor, 1 GB RAM, and 4 GB flash.  (I recommend finding other vendors to evaluate for your next project that are not so opaque.). For my earthworm systems, I used a BeagleBone Black (BBB), which has a Sitara 1 GHz ARM Cortex-A8 processor, 512 MB RAM, and 4 GB eMMC flash.  This seems virtually identical to the SBC in your system.

I have found that ARM processors have no trouble running Earthworm, even the very early ones without hardware floating-point.  I have never had a need to consider 64-bit ARM processors.  Where I have seen bottlenecks are I/O: writing to flash (the old, slow flash memory, NOT eMMC flash or SD cards) and Winston ImportEW consuming as much as 90% of the CPU (36 channels, 200 samples/s, 32-bit samples, or 28.8 KB/s).  I benchmarked I/O to the internal eMMC flash at 6 MB/s, which seems more than adequate.  Java seems to be the bottleneck.

Here's some issues I dealt with to enhance the reliability of the systems which, like yours, were in remote field locations:

1. U-Boot hangs - A lot of embedded SoCs/SBCs use U-Boot to configure the hardware and load the OS, like the BIOS on a PC.  When a BBB boots, by default U-Boot interrupts the boot process when any character is entered to the console serial port.  U-Boot prompts for user input, waiting forever; that is, without an input timeout.  When nothing is connected to the BBB console serial port, the pins are left floating.  The BBB UART is susceptible to noise during a power-up or reboot sequence, which causes spurious data to be detected on the console serial port.  About one time in twenty, the BBB would fail to boot because U-Boot was waiting for a command to proceed.  I modified U-Boot to require a specific character sequence ("+++") to interrupt the boot process that was virtually impossible to be generated by noise, as well as a 30-second input timeout.

2. Flash storage is precious - You described how you eliminated building the Earthworm modules you did not need to save space.  For my systems, I pruned everything I did not need from the OS as well.  For example, the Debian distribution for BBB was for a desktop.  I disabled the HDMI audio and video hardware and removed the desktop components.  Then, I removed all the packages that I did not need.  (This also reduces the attack surface from packages that are not necessarily secure.)

3. Avoid unnecessary writes to flash - As you know, flash memory wears out.  To increase the flash lifetime, disable unnecessary writes, such as access time stamp updates when files and directories are accessed (the noatime flag in /etc/fstab).  I used a BBB system to run a Winston waveserver, collecting data continuously from several buildings: 36 channels, sampled 200/s, 32-bit data.  I used it for a few years and had no eMMC flash (2 GB eMMC) or MicroSD card (32 GB) failures.

4. Mount the rootfs read-only - When the USGS Northern California Seismic Network deployed their NetWorm SBC systems, there were failures caused by corrupted file systems.  It turned out that they were unplugged by firemen in the fire stations when they hosed down the stations without being properly shut down.  For the Earthworm systems I build, I load the OS and applications on the rootfs and use a separate file system (on the BBB MicroSD card) for data.  I also make the rootfs read-only; it could not be corrupted, at least by Linux.  (I had to slightly reconfigure Debian for that to work.)  If your systems do not include the optional MicroSD card slot (mentioned in the XetaEdge9 data sheet as a Memory Capacity option), try to partition your flash storage into a rootfs and a data file system.  I use BTRFS for my data file system.  It has more reliability features than ext2/3/4.  I don't know what Debian uses these days.  If you are able, try using BTRFS with compression.  (I have never tried that.)  As you said, the CPU has plenty of available cycles.  If the rootfs can also be a BTRFS file system, try compressing it too.

5. Use a watchdog timer - systemd has watchdog timer support.  Unfortunately, it required a more recent version of systemd than the one in Debian Linux for the BBB I had.  I used the Linux watchdog daemon in the Debian watchdog package.  I configured it to watch systemd, in addition to its normal operation to trip when the system hangs or crashes.  Your hardware may have a hardware watchdog timer.  I use those on Earthworm PC servers.

6. Power - You have undoubtedly already considered what to do about power outages and your power budget.  I found that with the combination of a read-only rootfs and BTRFS for data, my BBB always came back up after a power outage.  A BBB only draws about 2W.  I am sure most of your power is used by the radio.  On Earthworm PC servers, I use UPS power and the Linux NUT package to gracefully shut down the server when the UPS batteries are exhausted.  Such software should work for an ARM SBC and a small UPS.  You can set up Linux and Earthworm to send email notifications when the system is starting up or shutting down so you will know when that happens.

7. systemd - I also use After=time-sync.target in my systemd configuration fie.  Winston needs Internet time.  I don't know if your software does.

8. The Linux Java run-time engine is slow - The Winston waveserver is written in Java.  Running the Linux Java run-time engine consumed over 90% of the BBB CPU (36 channels, 200 samples/s, 32-bit samples).  When I switched to the Oracle Java run-time engine for embedded ARM processors, CPU usage dropped to under 70%.  That was when Oracle did not restrict the use of Java by charging license fees.  I hope the Linux Java for ARM has improved.  Maybe it has a better JIT compiler by now.  I don't know if there are other alternatives.  Maybe there is something from Apache?

I am puzzled why you did not use ew_linux_arm.bash, which I added to Earthworm specifically for ARM Linux.  I have not compared it with ew_linux.bash in a while.  I hope it is still being maintained by someone.

I have written documentation for all the ARM systems I have developed, not just for Earthworm.  Some are small gateways (Ionics Nimbus, Marvell SheevaPlug).  Most have been the size of a cigarette box (BeagleBone Black).  Sometimes I use slightly larger boxes with room for a SATA SSD or an mSATA card (CodeLathe TonidoPlug, embeddedTS).  (I particularly like embeddedTS, formerly Technologic Systems, as a supplier.  GateWorks may interest you as well.)  All are no bigger than your XetaEdge9.  You are welcome to any of it.  They just keep running and running. :)

Best of luck,

Larry Baker
larry...@stanfordalumni.org

To view this discussion on the web visit https://groups.google.com/d/msgid/earthworm_forum/30366552-ea90-454e-9a8a-52dca21a7aa0n%40googlegroups.com.

[Victor Preatoni]

Hi Larry,

thanks a lot for your feedback!. also for porting EW to the ARM platform.

Neither I am completely happy with Xetawave radios, enclosure is not the best, even for the rugged models.

About hardware internals, Xeta9EC has a Texas Instruments Sitara processor: https://www.ti.com/product/AM3352/part-details/AM3352BZCZ100

SDRAM seems to be an Micron MT41K512M16HA, and eMMC a Kioxia THGBMJG6C1LBAU7. Not familiar with those memory brands, so not sure about reliability.

The U-Boot issue is really interesting. Not only because of noise, but some telemetry product can be connected to the radio serial port, and keep a continuous data flow, which will cause U-Boot to interrupt boot process. I will do some tests about that.

I never figured there was an ew_linux_arm.bash file, my mistake. Just compiled successfully using it.

Thanks,

Victor