Rev'd FLEX-6000 & QS1R skimming/recording guides

1 view
Skip to first unread message

Sam Rose

unread,
Aug 6, 2017, 1:20:17 PM8/6/17
to ham...@googlegroups.com
Hi everyone,

This weekend I attempted to finalize the FLEX-6000 skimming/recording guide, and with the help of Alex, W2AXR, wrote one for the QS1R. With the QS1R, it's actually quite easy to skim and record seven 192KHz-wide slices of RF.

I'm handing these off for review of those of y'all interested in doing so - otherwise, I think it's time to publish and promote, as users still have a couple weeks to get things set up.

This leads me to two questions as well...

1. Have we created much buzz among the Skimmer community, ie on the skimmertalk message board? Alex is a member of this and he's said that he has seen very little here about the Solar Eclipse QSO Party, and nothing he recalls about configuring RBN nodes to do I/Q recording as well.

2. Do we have a similar guide for the Red Pitaya? I looked around the HamSCI site, and the available documentation seems very disjoint to me - there's the "main" guide on how to set up an RBN node with a Red Pitaya, but this doesn't mention anything about recording - it uses the non-CWSL_Tee method. Then there's N6TV's guide which does use this method, but doesn't mention how to run any recorder - the page that links to this guide only mentions that it's possible. And the MIT Haystack Github pages provide little in the way of obvious instruction on how to get things running - unless you really have to go through all of that compiling every time, yikes!! It's quite easy to get lost in what's there now, which, with two weeks to go, is not a situation I like.

I think we need one "central" guide for the Red Pitaya - in my opinion, what's posted in public now is not adequately intuitive for the intelligent non-engineer to set up a combined recording/skimming node - and if we want a lot of recordings to come back, we need something better. Perhaps we modify the main guide to use the CWSL_Tee method, and instruct users to connect CWSL_File (or the new MIT utility if this can be done - and heck, if that'd work with the QS1R then I could modify that guide to call for that also) and consolidate it all into one intuitive guide.

73 -
Sam, KC2LRC
QS1R RBN Node Setup Guide.docx
FLEX-6000 RBN Node Setup Guide.docx

Sam Rose

unread,
Aug 7, 2017, 5:11:47 PM8/7/17
to ham...@googlegroups.com
So, following some off-list discussions, we've decided that we should go post these.

Technically, I have the technology to do this, but I won't be around tonight - would someone else want to? I think they should be linked to on this page under the Reverse Beacon Network heading: http://hamsci.org/projects/2017-total-solar-eclipse/get-involved

I think I will make a small change to the Flex version mentioning that the older PowerSDR radios can be used to do this as well for a single band. I doubt it will garner many users, but since Flex wrote a guide on how to do it, I'll mention it.

Next I think we should make a revision of the Red Pitaya guide to include recording. I think this really needs to be a part of the main setup guide - this page: http://hamsci.org/cw-reverse-beacon-network-how-guide - particularly as it's not clear from anything on our site that you can simultaneously skim to RBN and record, we should modify that to use CWSL_Tee.dll, then add a section on CWSL_File usage if this method works - the procedure would be about the same as what I already wrote for the QS1R. After tonight I should be available to give this a try. There's a lot of good info in N6TV's setup guide, but in general I don't think it's very intuitive as written - it reads more like "engineer's notebook" than a straightforward how-to, particularly as the revisions require jumping up and down the page to read. I can try to make a nice, cohesive guide for all of this while there's still time to put the information out.

73 -
Sam, KC2LRC

Pete Smith N4ZR

unread,
Aug 7, 2017, 6:02:04 PM8/7/17
to Ham...@googlegroups.com, skimmertalk
On August 21, 2017, from 1400 to 2200Z, operators of RBN nodes will have a unique opportunity to contribute to scientific understanding of the ionosphere.  That time-frame straddles the period when the solar eclipse will be visible across North America, and is also when the Solar Eclipse QSO Party will be run.

The RBN's unique contribution, in North America but also worldwide, is to provide as many data points as possible during the 1400-2200Z period.  In order to do this, two departures from normal practice are needed:

1.  It is desirable for science to have much more frequent spots of stations active in the QSO Party than could be produced with the standard 10 minute respot interval. To this end, Alex has provided for adjustabilty in this respect in the current versions of CW Skimmer, CW Skimmer Server and RTTY Skimmer Server.  By adding a single line to the .ini file for your skimming software, you can adjust the respot interval anywhere from the original 10 minutes down to zero.  In the latter case, every repetition of a station's callsign will be reported (provided of course that the station includes CQ or TEST at least twice in each transmission).

2. The RBN archive is not the best source of the scientific data we hope to produce, because the timing is relatively imprecise (nearest minute).  So check your Skimmer computer for a file titled "spots.txt" , which will be found in C:\users\[your username]\Appdata\Roaming\Afreet\Products\Skimsrv.  The spots.txt file gives the time when a spot is actually made, to the nearest second. Regrettably, there is no comparable file in CW Skimmer. For Skimmer ops, your data are welcome too, even if still on a 10-minute scale - after all the eclipse period is hours long!

If you have been operating your node for a while, this file will be quite large - all that we need is 1400-2200Z on the 21st.  You can use a text editor to extract the part we need.  We'll tell you where to send your spot file shortly.

HamSCI is also looking for recordings of digital I/Q data from Skimmer receivers made during the duration of the Solar Eclipse QSO Party. This will allow HamSCI to replay and analyze recordings from specific receivers in greater depth following the contest. HamSCI will be publishing guides shortly on how this can be done with the QS1R or Red Pitaya and CW Skimmer Server, or with any SDR capable of sending data to CW Skimmer using its built-in recording function. Note that this will consume significant hard disk space - up to about 4GB per hour per band. The data will be accepted for upload or in physical form after the contest.

If you're interested in putting your node to work on this project, you're still lacking one thing - the magic formula to put into your Skimsrv's ini file.  Email me, and the secrets of the universe will be revealed.

I'm doing it this way because I want to have some reasonable confidence that people won't start using shorter interval settings with the RBN servers outside the eclipse period.  Even one Skimmer can make a big difference in this regard, and we will not hesitate to block anyone's spots from the RBN server if they violate this rule outside the eclipse period.

73, Pete N4ZR
Check out the Reverse Beacon Network 
at <http://reversebeacon.net>, now 
spotting RTTY activity worldwide. 
For spots, please use your favorite 
"retail" DX cluster.

Robert McGwier

unread,
Aug 7, 2017, 7:56:02 PM8/7/17
to HAMSCI, HAMSCI, skimmertalk
N4HY will be on the air before, during, and after the eclipse with the Virginia Tech team in South Carolina.  We will operate mostly CW but some digital and SSB as well.  Thanks to all of the RBN operators for making the network data available to the scientists and the students.

Alex is, as always, a great contributor to ham radio.  

I might have time to put up a Red Pitaya working on several bands doing WSPR, but no guarantee.

Bob
N4HY


Dr. Robert W. McGwier
rwm...@vt.edu
Chief Scientist, Hume Center for National Security and Technology
Research Professor, Electrical and Computer Engineering
Research Professor, Aerospace and Ocean Engineering
1991 Kraft Drive (0713), Virginia Tech, Blacksburg, VA 24061
Office: (540) 231-2041 Cell: (609)-731-5289 (Leave a voicemail)
Affiliated: Wireless@VT, Space@VT, Senior Member IEEE, ARRL, BOD of AMSAT, TAPR
Amateur Radio Station: N4HY
Faculty Advisor VT Amateur Radio Association
Engineer, Mathematician (Auburn University, Ph.D Brown University)


From: "Pete Smith N4ZR" <pete...@gmail.com>
To: "HAMSCI" <Ham...@googlegroups.com>
Cc: "skimmertalk" <skimm...@contesting.com>
Sent: Monday, August 7, 2017 6:02:06 PM
Subject: [HamSCI] RBN Participation in Solar Eclipse Science

--
You received this message because you are subscribed to the Google Groups "HamSCI" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+un...@googlegroups.com.
To post to this group, send email to ham...@googlegroups.com.
Visit this group at https://groups.google.com/group/hamsci.
To view this discussion on the web visit https://groups.google.com/d/msgid/hamsci/a1a2f2d7-e531-66e6-0d57-be6ada6a7091%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Sam Rose

unread,
Aug 8, 2017, 11:22:18 PM8/8/17
to ham...@googlegroups.com
Posted!

See http://hamsci.org/projects/2017-total-solar-eclipse/get-involved under Reverse Beacon Network - links are there.

I'll do some work on consolidating the Red Pitaya guide soon, but not tonight!

73 -
Sam, KC2LRC

Pete Smith N4ZR

unread,
Aug 9, 2017, 6:11:12 AM8/9/17
to ham...@googlegroups.com, Sam Rose

    Should I give RBN people the zenodo URL for uploading their RBN spots.txt files?  Maybe a folder just for this?

73, Pete N4ZR
Download the new N1MM Logger+ at 
<http://N1MM.hamdocs.com>. Check 
out the Reverse Beacon Network at 
<http://reversebeacon.net>, now 
spotting RTTY activity worldwide. 
For spots, please use your favorite 
"retail" DX cluster.
--
You received this message because you are subscribed to the Google Groups "HamSCI" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+un...@googlegroups.com.
To post to this group, send email to ham...@googlegroups.com.
Visit this group at https://groups.google.com/group/hamsci.

Sam Rose

unread,
Aug 9, 2017, 8:43:36 AM8/9/17
to ham...@googlegroups.com
Hi Pete,

Others can chime in here, but I vote yes. If I understand correctly, zenodo is where we're going to be archiving all of the data this generates, so if we're storing the I/Q data there, then the spots.txt files would be appropriately stored there as well - and of course, it'll destroy quite a few less of their hard drives than the I/Q data will...

Should we be making the announcement to other places that we now have recording guides as well?

73 -
Sam

To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+unsubscribe@googlegroups.com.

To post to this group, send email to ham...@googlegroups.com.
Visit this group at https://groups.google.com/group/hamsci.

--
You received this message because you are subscribed to the Google Groups "HamSCI" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+unsubscribe@googlegroups.com.

To post to this group, send email to ham...@googlegroups.com.
Visit this group at https://groups.google.com/group/hamsci.

Nathaniel A. Frissell

unread,
Aug 9, 2017, 10:48:43 AM8/9/17
to ham...@googlegroups.com
Hi, guys.

Zenodo doesn’t currently support folders, only communities. So, I think uploading them as individual files to zenodo will make things cluttered. I’m pointing people to zenodo especially for sharing larger files.

I think it would be best to have them upload them to a form on hamsci.org. We will post the link to this form at hamsci.org/seqp as soon as it is ready. In addition to the spots.txt, I also want to collect some basic information confirming the location of the receiver and finding out what antenna and receiver they are using.

Once we get all of them, we can post the results to hamsci.org or to zenodo as a single dataset.

Thanks and 73,
Nathaniel W2NAF

To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+un...@googlegroups.com.

To post to this group, send email to ham...@googlegroups.com.
Visit this group at https://groups.google.com/group/hamsci.

Nathaniel A. Frissell

unread,
Aug 9, 2017, 11:45:29 AM8/9/17
to ham...@googlegroups.com
Sam, thank you for writing this! I just read it, and it is well written.

Correct me if I am wrong, but the IQ files produced are not time-stamped. Is that true? Ideally, we would want something like the timestamps that the Digital RF package can give us. But, maybe at this point the best thing to do at this point is to make sure the user reports to us the correct start time of the file. Additionally, we should be able to get some timing by looking at a stations RBN spots in conjunction with their IQ data, but this will be very tedious.

Another big issue is how we get the data. I’m seeing estimates of total data for this event being around 500 GB. Zenodo allows 50 GB uploads per data set. Each individual band recording should be under 50 GB, so we could ask the participants to upload each band separately. The biggest problem here is the time it takes the participant to do this. Even so, I’d really like these recordings posted to Zenodo as it will both archive the data permanently and make it available to everyone.

Do you have any thoughts on this?

If we are good on the Zenodo band-splitting approach, we should add text to your pages telling people to follow the instructions at hamsci.org/eclipse-wideband for submitting the observations.

73,
Nathaniel W2NAF


-- 
You received this message because you are subscribed to the Google Groups "HamSCI" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+un...@googlegroups.com.
To post to this group, send email to ham...@googlegroups.com.
Visit this group at https://groups.google.com/group/hamsci.

Sam Rose

unread,
Aug 9, 2017, 12:29:18 PM8/9/17
to ham...@googlegroups.com
Hi Nathaniel,

>> Correct me if I am wrong, but the IQ files produced are not time-stamped. Is that true?

As far as I know, yes, this is true. It's not ideal, and while I think the Digital RF package produces more ideal data outputs, it would have had to have become available much earlier than it did for it to be useful here - for instance it definitely won't work with the Flex in its current form, nor do I know if it will integrate with CWSL_Tee. The existing recording technology produces files which are time/date stamped at their beginnings, and are broken at intervals so the files only run so long - it's 1 hour by default with CWSL_File. So I'm suggesting that users keep their clocks sync'd so this start time will be meaningful. Having a good start time is a heckuva lot better than having to hand-sync each user's recording.

With 1.5 weeks to go, I'll only be doing minor edits to the guides that are finalized - it's unfortunate that I'm this late to begin with, but vacation mishaps do happen! The only "low-hanging fruit" improvement to timekeeping I can think of would be to specify that the users give CWSL_File the '-Q' argument which would make four files per hour, and therefore increase the number of timestamps generated.

CWSL is open source, so perhaps before the next event where I/Q recording is relevant, we could build some capabilities into it to enabled better file output formats with timestamping, like the MIT format.

>> Zenodo allows 50 GB uploads per data set. Each individual band recording should be under 50 GB, so we could ask the participants to upload each band separately. The biggest problem here is the time it takes the participant to do this.

As for getting the data - yeah, if we're using Zenodo I think splitting by band is the only reasonable way to do it. I've got a couple monster servers at home, so if need be I could accept whole sets from anyone who needs to reclaim their disk space in a hurry, but we'll hold off on stuff like this unless individual users make alternative arrangements with us.

Might be worth a quick page somewhere with explanations of how to submit data that we can link to from elsewhere in the recording guides.

73- 
Sam, KC2LRC

To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "HamSCI" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+unsubscribe@googlegroups.com.

To post to this group, send email to ham...@googlegroups.com.
Visit this group at https://groups.google.com/group/hamsci.

Nathaniel A. Frissell

unread,
Aug 9, 2017, 12:38:27 PM8/9/17
to ham...@googlegroups.com
Hi, Sam.

Very good. Does CWSL_File output a time a frequency stamp into the filename automatically?

For data upload instructions, point everyone to hamsci.org/eclipse-wideband. There are relatively detailed instructions for uploading to zenodo there. We can tweak them as need be. Also, it may be appropriate to include links to your pages on the wideband recording page.

Yes, let’s encourage everyone to submit to zenodo and handle others submissions on a case-by-case basis.

I realize there are a lot of loose ends here… especially with regard to time stamping, a smooth data upload process, etc. I see this Eclipse project as not simply a way of getting science observations, but also a way for us to learn how the ham radio community should grow to make this stuff useful in the future. For instance, I don’t think time and frequency stamping IQ files was something the ham radio community has put much thought into before. Hopefully it will become a standard practice!

73,
Nathaniel W2NAF

----------
----------------------------------------
Nathaniel A. Frissell, Ph.D.
Assistant Research Professor
Center for Solar-Terrestrial Research
New Jersey Institute of Technology
Tiernan Hall, Room 101
161 Warren Street, Newark, NJ. 07102
--------------------------------------------------

To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+un...@googlegroups.com.

To post to this group, send email to ham...@googlegroups.com.
Visit this group at https://groups.google.com/group/hamsci.

Sam Rose

unread,
Aug 9, 2017, 2:04:09 PM8/9/17
to ham...@googlegroups.com
Hi Nathaniel,

>> Does CWSL_File output a time a frequency stamp into the filename automatically?

Yes. It also automatically names them based on their center frequency. Skimmer also timestamps the filenames when saving. So having the computer NTP-locked should be adequate for having at least some time information.

>> I realize there are a lot of loose ends here… especially with regard to time stamping, a smooth data upload process, etc.

Yes, there sure are! I think this will be a great opportunity to learn where we want to go, both in terms of hardware, software, and logistics. Any crowdsourced project like this will have its challenges... later on, for instance, I imagine there will be a time period where some users have wideband-capable (0-60MHz or so) I/Q recorders, but others still are using ones that produce only a number of 192KHz slices. Our ability to benefit from whatever sort of contributions come in will remain key to this project.

73 -
Sam



To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+unsubscribe@googlegroups.com.

To post to this group, send email to ham...@googlegroups.com.
Visit this group at https://groups.google.com/group/hamsci.

--
You received this message because you are subscribed to the Google Groups "HamSCI" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+unsubscribe@googlegroups.com.
To post to this group, send email to ham...@googlegroups.com.
Visit this group at https://groups.google.com/group/hamsci.

Phil Erickson

unread,
Aug 9, 2017, 3:25:13 PM8/9/17
to ham...@googlegroups.com
Hi Sam,

  It is really essential that these systems move towards being able to absolutely time stamp their data with UTC (and know when the triggered acquisition started, down to the resolution of the time standard used in the system).  There are numerous examples of 'meta-instruments' that form the measurement by comparing two separated observations with time tagging, where without the time tagging the information is not usable and the measurement cannot exist.  Examples = GPS navigation signals determining ionospheric content, oblique ionograms where you don't control the TX but you know when it started (and you know the absolute UTC of the recorded data at the RX end), and others.

  Global time sources like GPS make the possibility of knowing what UTC is much easier.  

  As long as I know when the recorded samples happened, even narrow band recordings contain useful information as long as the RF center frequency and bandwidth end up containing the signal of interest!

73
Phil W1PJE



For more options, visit https://groups.google.com/d/optout.



--
----
Phil Erickson
phil.e...@gmail.com

Pete Smith N4ZR

unread,
Aug 9, 2017, 3:28:52 PM8/9/17
to ham...@googlegroups.com, Phil Erickson

It may also be useful to develop a means of comparing the spot data generated by the RBN with the I/Q recordings.  The CW Skimmer Server's spots.txt file has time resolution to the nearest second.

73, Pete N4ZR
Download the new N1MM Logger+ at 
<http://N1MM.hamdocs.com>. Check 
out the Reverse Beacon Network at 
<http://reversebeacon.net>, now 
spotting RTTY activity worldwide. 
For spots, please use your favorite 
"retail" DX cluster.
To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+un...@googlegroups.com.

To post to this group, send email to ham...@googlegroups.com.
Visit this group at https://groups.google.com/group/hamsci.

Sam Rose

unread,
Aug 9, 2017, 9:59:39 PM8/9/17
to ham...@googlegroups.com
Hi Phil,

Certainly, I desire to move towards this going forward - appreciate that, for now, the instructions I made were to leverage what we can do reliably today. Data formats like MIT's (and what I use internally on projects at work with GPS radio clocks) solve many of our issues, as would SDRs with on-board GPS receivers for PPS and such. But we're 1.5 weeks out, and without having software support earlier, integration of these wasn't going to get done in time.

So I see this is a project that goes beyond this event. We had some in-depth discussion earlier about coming up with a new science-oriented SDR architecture, and one of the problems is that we don't know exactly what we want out of it. Anyhow, we're off topic at this point, so these discussions should continue at a later point.

73 -
Sam, KC2LRC

Sam Rose

unread,
Aug 9, 2017, 10:27:29 PM8/9/17
to ham...@googlegroups.com
Added the same blurb to the bottom of each guide I did on how to upload. Okay, now for a little promotion of what's posted. Tomorrow I'll see if recording from a Red Pitaya in Skimmer mode is as easy as doing so with a QS1R, and update the guide if so.

73 -
Sam

On Wed, Aug 9, 2017 at 11:45 AM, Nathaniel A. Frissell <nathaniel....@njit.edu> wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "HamSCI" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+unsubscribe@googlegroups.com.

To post to this group, send email to ham...@googlegroups.com.
Visit this group at https://groups.google.com/group/hamsci.

Phil Erickson

unread,
Aug 9, 2017, 11:45:42 PM8/9/17
to ham...@googlegroups.com
Hi Sam,

  Yes, my comments were more of a long term goal.  For any event as close as Aug. 21, then you have to have frozen features long before this time, so it's great that you can do as much as you've done!

73
Phil W1PJE


For more options, visit https://groups.google.com/d/optout.

Robert McGwier

unread,
Aug 10, 2017, 7:42:26 AM8/10/17
to ham...@googlegroups.com
I discussed this with Steve Hicks yesterday. They are planning time stamps to support contract from SWRI and that will fold into the main product but it isn't in there yet.

Boh

Dr. Robert W. McGwier, Chief Scientist

Hume Center for National Security and Technology
Virginia Tech



Robert McGwier

unread,
Aug 10, 2017, 7:45:55 AM8/10/17
to ham...@googlegroups.com
The time stamping needed has to be done synchronously by the radio using a high performance clock.

Dr. Robert W. McGwier, Chief Scientist
Hume Center for National Security and Technology
Virginia Tech
To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+un...@googlegroups.com.

To post to this group, send email to ham...@googlegroups.com.
Visit this group at https://groups.google.com/group/hamsci.
--
You received this message because you are subscribed to the Google Groups "HamSCI" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hamsci+un...@googlegroups.com.

To post to this group, send email to ham...@googlegroups.com.
Visit this group at https://groups.google.com/group/hamsci.

Tom Clark

unread,
Aug 11, 2017, 5:09:35 PM8/11/17
to ham...@googlegroups.com, Robert McGwier
On 8/10/2017 07:42 AM, Robert McGwier wrote:
I discussed this with Steve Hicks yesterday. They are planning time stamps to support contract from SWRI and that will fold into the main product but it isn't in there yet.

Boh

Dr. Robert W. McGwier, Chief Scientist
Hume Center for National Security and Technology
Virginia Tech

If Steve plans to support time stamps on the 6xxx, it would be a wonderful time for him to support an external 1PPS  input to set the clock. As it is, the only time hack is from the internal GPSDO, but I would like to be able to sync to an external time/frequency standard. Needless to say, I suggest this to allow multiple remote stations to have synchronous recording (a.k.a. VLBI).

73 de Tom, K3IO
Reply all
Reply to author
Forward
0 new messages