Python and Radio Sky Pipe

[Dave Typinski]

Does anyone here have Python code to read (and/or write) Radio Sky Pipe SPD
format data files? If not, I'll write a module for it. Is anyone here
interested in that?
--
Dave

[Michiel Klaassen]

Hi Dave,

Some time ago you asked me if I was ok with using my Cfrad/3pt Python software for Jupiter radio reception, and yes thats ok. 

Also Raydel asked me to use that for the software written by Marcus and that is ok too.

However it is not ok if you are planning to use it to support Radio Sky software. 

That is a commercial enterprise I do not want to support in any way.

Regards,

Michiel

parac.eu

Op za 30 jan. 2021 om 02:44 schreef Dave Typinski <dav...@typnet.net>:

--
--
You received this message because you are subscribed to the Google
Groups "Society of Amateur Radio Astronomers" group.
To post to this group, send email to sara...@googlegroups.com
To unsubscribe from this group, send email to
sara-list-...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sara-list?hl=en
---
You received this message because you are subscribed to the Google Groups "Society of Amateur Radio Astronomers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sara-list+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/6014C809.6030202%40typnet.net.

[Dave Typinski]

Hi Michiel,

That was a whole different ballgame. I just wanted to see how you plotted stuff
in cfrad when I was learning more about matplotlib. Nothing to do with RSP at all.
--
Dave

On 1/30/21 04:30, Michiel Klaassen wrote:
> Hi Dave,
> Some time ago you asked me if I was ok with using my Cfrad/3pt Python software
> for Jupiter radio reception, and yes thats ok.
> Also Raydel asked me to use that for the software written by Marcus and that is
> ok too.
>
> However it is not ok if you are planning to use it to support Radio Sky software.
> That is a commercial enterprise I do not want to support in any way.
>
> Regards,
> Michiel

> parac.eu <http://parac.eu>

>
>
> Op za 30 jan. 2021 om 02:44 schreef Dave Typinski <dav...@typnet.net

> <mailto:dav...@typnet.net>>:

>
> Does anyone here have Python code to read (and/or write) Radio Sky Pipe SPD
> format data files? If not, I'll write a module for it. Is anyone here
> interested in that?
> --
> Dave
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Society of Amateur Radio Astronomers" group.
> To post to this group, send email to sara...@googlegroups.com

> <mailto:sara...@googlegroups.com>

> To unsubscribe from this group, send email to
> sara-list-...@googlegroups.com

> <mailto:sara-list-...@googlegroups.com>

> For more options, visit this group at
> http://groups.google.com/group/sara-list?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Society of Amateur Radio Astronomers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sara-list+...@googlegroups.com

> <mailto:sara-list%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sara-list/6014C809.6030202%40typnet.net.
>

> --
> --
> You received this message because you are subscribed to the Google
> Groups "Society of Amateur Radio Astronomers" group.
> To post to this group, send email to sara...@googlegroups.com
> To unsubscribe from this group, send email to
> sara-list-...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/sara-list?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Society of Amateur Radio Astronomers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to sara-list+...@googlegroups.com

> <mailto:sara-list+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/sara-list/CAABBjRAyJRMCPeMT6YCsCS08hEwS4fmX6Ye6wKZdiDAqKY-7Nw%40mail.gmail.com
> <https://groups.google.com/d/msgid/sara-list/CAABBjRAyJRMCPeMT6YCsCS08hEwS4fmX6Ye6wKZdiDAqKY-7Nw%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Emmett Kyle]

I wrote a script that we use at Hobbs Observatory. Included is a bash script to work on multiple files/directories.

I'm not sure how attach files here, so I'll just include them in the message:

#######################

#!/usr/bin/python
import datetime
import array
import string as string
import os
import csv
import math
from optparse import OptionParser

def spd2txt(in_file, out_file, delim, lineterm, comchar, metadata, min_sz ):
    # Dictionary to hold boiler plate info (meta data)
    bP = {}
    # Channel labels and info
    cI = []
    if os.path.getsize(in_file) < min_sz:
        print 'File:', in_file, '< minimum size:', min_sz
        return

    # Open SPD file
    fp = open(in_file, 'r')

    bP['vers']      = 'version:    '+str(fp.read(10))
    bP['start']     = 'Start time: '+str(vbtime2dtime(arr2double(fp.read(8))))
    bP['finish']    = 'End time:   '+str(vbtime2dtime(arr2double(fp.read(8))))
    bP['lat']       = 'Latitude:   '+dec2minsec(arr2double(fp.read(8)))
    bP['lng']       = 'Longitude:  '+dec2minsec(arr2double(fp.read(8)))
    bP['max_y']     = 'Maximum Y:  '+str(arr2double(fp.read(8)))
    bP['min_y']     = 'Minimum Y:  '+str(arr2double(fp.read(8)))
    bP['time_zone'] = 'Time zone:  '+str(arr2int(fp.read(2)))
    bP['source']    = 'Source:     '+str(fp.read(10))
    bP['author']    = 'Author:     '+str(fp.read(20))
    bP['loc_name']  = 'Local Name: '+str(fp.read(20))
    bP['location']  = 'Location:   '+str(fp.read(40))
    channels        = arr2int(fp.read(2))
    bP['channels']  = '# channels: '+str(channels)
    noteLen         = arr2int(fp.read(4))
    bP['note_len']  = 'Note len:   '+str(noteLen)

    noteBlk         = fp.read(noteLen)
    for i in noteBlk.split('\xff'):
    # Print channel label if present CHL#<label>
        if ('CHL' in i) and (metadata == True):
        cI.append('Channel '+str(int(i[3])+1)+ ': '+i[4:])

        else:
            if 'CHL' in i:
            cI.append(i[4:])

    fSize        = fileSz(in_file)
    bP['f_size'] = 'File size:  '+str(fSize)+' bytes'

    # Data block starts after the "noteBlk" section
    # Data block size in bytes
    dBlock        = fSize-fp.tell()
    bP['d_block'] = 'Data size:  '+str(dBlock)+' bytes'

    # Number of data points
    nEntries        = dBlock/((channels+1)*8)
    bP['n_entries'] = '# Samples:  '+str(nEntries)

    # Parse date time stamp and channel data
    data = []
    for i in range(0,nEntries):
        dataPoint = []
        for j in range(0, channels+1):
        d = arr2double(fp.read(8))
        if j == 0:
        d = vbtime2dtime(d)
            dataPoint.append(d)
    data.append(dataPoint)

    # Close SPD input file
    fp.close()

    # Write output file
    with open(out_file, 'w') as csvFile:
        csvWriter = csv.writer(csvFile, escapechar='', delimiter=delim,\
            lineterminator=lineterm, quoting=csv.QUOTE_NONE)

    if metadata == True:
        # Add boiler plate info
        for bp in bP.values():
            csvFile.write(comchar+' '+bp+lineterm)
        csvFile.write(comchar+lineterm)

            # Add channel label info
        for ch in cI:
            csvFile.write(comchar+' '+ch+lineterm)
        csvFile.write(comchar+lineterm)
        else:
            '''
                Write out a single line header containing each channel name
            '''
            csvFile.write('timestamp,'+','.join(cI)+lineterm)

        # Add date/time data
        for dp in data:
        csvWriter.writerow(dp)

    return 0

def arr2double(a):
    return array.array('d', a)[0]

def arr2int(a):
    return array.array('h', a)[0]

def fileSz(filename):
    st = os.stat(filename)
    return st.st_size

def vbtime2dtime(vb_time):
    ''' Convert visual basic time to "normal".
        Integer value is y/m/d. Fractional value = % of day
    '''
    i,f = divmod(vb_time, 1) # split integer/fractional parts
    ymd = (i - 25569) * 86400.0
    return datetime.datetime.utcfromtimestamp(ymd+(round(f*86400)))
       
def dec2minsec(dec):
    a = divmod(dec, 1)
    deg = a[0]
    b = divmod(a[1]*60,1)
    minutes = b[0]
    seconds = b[1]*60
    return str(deg)+':'+str(minutes)+'"'+str(seconds)+"'"
# test
def main():

    parser = OptionParser()
    parser.add_option('-c', '--comment-char', dest='comment_char', default='#',
            help='Character to comment out boiler plate info.')

    parser.add_option('-d', '--deliminer', dest='delim', default=',',
            help='Separator between fields in output file.')

    parser.add_option('-f', '--in-file', dest='in_file', default=None,
            help='Input SPD file', metavar='FILE')

    parser.add_option('-q', '--quiet', dest='meta_dat', default=True,
            action='store_false',
            help='Do not print meta data in output file. Default is True.')

    parser.add_option('-o', '--out-file', dest='out_file', default=None,
            help='Output TXT file', metavar='FILE')

    parser.add_option('-m', '--min-size', dest='min_sz', default=1000000L,
            help='Minimum input file size', metavar='MINSZ')

    parser.add_option('-t', '--terminator', dest='term', default='\r\n',
            help='Line ending terminator. \\n for unix, \\r\\n for windows')

    (opts, args) = parser.parse_args()

# Check validity of input args.
    if opts.in_file == None:
        print('Error: No input file name supplied')
        return 1

# fix this!
    if opts.out_file == None:
        out_file = 'out.txt'

    spd2txt(opts.in_file, opts.out_file, opts.delim, opts.term,
            opts.comment_char, opts.meta_dat, opts.min_sz)

if __name__ == "__main__":
    # Someone is launching this directly
    main()

#####################################

# convert.sh

#!/bin/bash
# python rsp2txt.py -f *.spd -o *.txt -q -d ,

DEST_DIR=csv_converted

mkdir -p $DEST_DIR

for f in `find . | grep spd | grep -v txt`
do
    out_file=$(basename $(echo $f) | cut -f1 -d '.').txt
    echo "rsp2txt.py -f $f -d , -q -o ${DEST_DIR}/${out_file}"
    ./rsp2txt.py -f $f -d , -q -o ${DEST_DIR}/${out_file}
done

##################################################

Regards,

Emmett Kyle

[Larry Mayfield]

Most email clients have somewhere on the ribbon at the top where it says attach file. Make your file with a word processor or even just not the where the Python file is located. The click on the attach file, and when it asks for the file, browse to it and highlight it and tell it to include or attach. All the email systems are similar so it should be fairly easy to do.  You can send me something, heck try it to me with the python file. I’d be interested and I have python my desk top and also on my Raspberry pi.  Happy to help you get comfortable with attaching and sending files as well as receiving them as well.  Be happy to assist…

 

Larry Mayfield

Pahrump, NV

--

--
You received this message because you are subscribed to the Google
Groups "Society of Amateur Radio Astronomers" group.
To post to this group, send email to sara...@googlegroups.com
To unsubscribe from this group, send email to
sara-list-...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/sara-list?hl=en
---
You received this message because you are subscribed to the Google Groups "Society of Amateur Radio Astronomers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sara-list+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/174d85dc-8560-4757-b011-504dedbbc6dfn%40googlegroups.com.

[Emmett Kyle]

Okay, the first time I tried this, it didn't show an "add attachment" Probably because I wasn't subscribed to the group. Here are the files we use at Hobbs Observatory to convert Radio Skypipe files. In addition to the converter, I have a down sampler script to reduce the data size and one to switch column data.

The next thing I want to do is make a UDS interface to RadioSkypipe that works with GNU Radio.

[Larry Mayfield]

You know you can send to the sara list via your email client also.  I use outlook which came with my MS home and office suite. I just use the sara email addy (like the one above: sara...@googlegroups.com and attach my files through it.  Then send and they are there for the world to see. Unless something has changed recently. I do all my list corresponding from me email account.

An Arduino file is attached, just something I have been rooking on for my yet to be finalized observatory.  Just a  sample….

larry

To view this discussion on the web visit https://groups.google.com/d/msgid/sara-list/841bcf9b-b358-455a-bbe2-2b83251b9506n%40googlegroups.com.

[Dave Typinski]

Hi Emmett,

Wow, thank you very much for the Python code and shell scripts! This is exactly
what I was looking for. Much appreciated!
--
Dave

On 1/30/21 16:57, Emmett Kyle wrote:
> Okay, the first time I tried this, it didn't show an "add attachment" Probably
> because I wasn't subscribed to the group. Here are the files we use at Hobbs
> Observatory to convert Radio Skypipe files. In addition to the converter, I have
> a down sampler script to reduce the data size and one to switch column data.
> The next thing I want to do is make a UDS interface to RadioSkypipe that works
> with GNU Radio.
>
> On Saturday, January 30, 2021 at 3:41:08 PM UTC-6 Lawrence E. Mayfield wrote:
>
> Most email clients have somewhere on the ribbon at the top where it says
> attach file. Make your file with a word processor or even just not the where
> the Python file is located. The click on the attach file, and when it asks
> for the file, browse to it and highlight it and tell it to include or
> attach. All the email systems are similar so it should be fairly easy to
> do. You can send me something, heck try it to me with the python file. I’d
> be interested and I have python my desk top and also on my Raspberry pi.
> Happy to help you get comfortable with attaching and sending files as well

> as receiving them as well. Be happy to assist…____
>
> __ __
>
> Larry Mayfield____
>
> Pahrump, NV____
>
> __ __
>
> *From:*sara...@googlegroups.com [mailto:sara...@googlegroups.com] *On Behalf
> Of *Emmett Kyle
> *Sent:* Saturday, January 30, 2021 12:54 PM
> *To:* Society of Amateur Radio Astronomers <sara...@googlegroups.com>
> *Subject:* [SARA] Re: Python and Radio Sky Pipe____
>
> __ __

>
> I wrote a script that we use at Hobbs Observatory. Included is a bash script

> to work on multiple files/directories.____

>
> I'm not sure how attach files here, so I'll just include them in the

> message:____
>
> #######################____

> main()____
>
> __ __
>
> #####################################____
>
> # convert.sh____

>
> #!/bin/bash
> # python rsp2txt.py -f *.spd -o *.txt -q -d ,
>
> DEST_DIR=csv_converted
>
> mkdir -p $DEST_DIR
>
> for f in `find . | grep spd | grep -v txt`
> do
> out_file=$(basename $(echo $f) | cut -f1 -d '.').txt
> echo "rsp2txt.py -f $f -d , -q -o ${DEST_DIR}/${out_file}"
> ./rsp2txt.py -f $f -d , -q -o ${DEST_DIR}/${out_file}

> done____
>
> __ __
>
> ##################################################____
>
>
> Regards,
>
> Emmett Kyle____
>
> On Friday, January 29, 2021 at 8:44:27 PM UTC-6 Dave Typinski wrote:____

>
> Does anyone here have Python code to read (and/or write) Radio Sky Pipe SPD
> format data files? If not, I'll write a module for it. Is anyone here
> interested in that?
> --

> Dave ____

>
> --
>
> --
> You received this message because you are subscribed to the Google
> Groups "Society of Amateur Radio Astronomers" group.
> To post to this group, send email to sara...@googlegroups.com
> To unsubscribe from this group, send email to
> sara-list-...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/sara-list?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Society of Amateur Radio Astronomers" group.
>
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sara-list+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/sara-list/174d85dc-8560-4757-b011-504dedbbc6dfn%40googlegroups.com

> <https://groups.google.com/d/msgid/sara-list/174d85dc-8560-4757-b011-504dedbbc6dfn%40googlegroups.com?utm_medium=email&utm_source=footer>.____

>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Society of Amateur Radio Astronomers" group.
> To post to this group, send email to sara...@googlegroups.com
> To unsubscribe from this group, send email to
> sara-list-...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/sara-list?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Society of Amateur Radio Astronomers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to sara-list+...@googlegroups.com

> <mailto:sara-list+...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/sara-list/841bcf9b-b358-455a-bbe2-2b83251b9506n%40googlegroups.com
> <https://groups.google.com/d/msgid/sara-list/841bcf9b-b358-455a-bbe2-2b83251b9506n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Alexandre Alves]

Hi Emmett,

Thanks a lot.

Kind regards

Alexandre

[Emmett Kyle]

No problem!