System down

[Philip Gladstone]

I have no idea what is going on. I'm away from a laptop for the next few hours. Hopefully it will recover by itself. 

Sorry

Philip

Philip Gladstone

[AndyZ K1RA]

I notice this wackiness in the PSKr reporting tsv.gz 6-hour file archives at https://pskreporter.info/csv/2026/08/ .  I usually see pairs of _e and _o files, but mostly _o and MUCH more frequent than 6 hours, and not the usually 100-200M size, but instead on the order of bytes, Kbytes or a few M. Attachment #1

It appears this started around 08/09 20Z. Attachment #2

73

andyz - K1RA

[Philip Gladstone]

Yes. For some reason that I do not currently understand, the process that generates the _o files (which are from reporters with Odd ip addresses) burns memory and runs into its memory limit, crashes, and then is restarted. Since this only happens to the Odd process and not the Even process, I think that the issue is to do with something that someone is reporting. I've added lots of diagnostics to the code, but I still can't figure it out. 

It hasn't happened since yesterday evening. There is nothing obvious in the logs. 

Philip

Philip Gladstone

--
You received this message because you are subscribed to the Google Groups "PSK Reporter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to psk-reporter...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/psk-reporter/b4ffe822-ead6-4f35-b8cd-c5dae037e295n%40googlegroups.com.

[AndyZ K1RA]

It appears the system successfully generated a few full size _o & _e tsv.gz files (100-300M), at the top of an hour, though NOT a pair on a 6-hour boundary like has been generated under regular conditions, then went hack to multiple, smaller files, see attached.

I opened some of the shortened files and they either just have a header line OR a handful of station spots.  There was no pattern to the last spot in the files where it cut off though to give any clue what may be causing the problem.

This is happening against both the even and odd files now.

andyz - K1RA

You received this message because you are subscribed to a topic in the Google Groups "PSK Reporter" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/psk-reporter/OT1pyztOp1E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to psk-reporter...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/psk-reporter/CAMd_wWqxMddfpD-ozMDG%2BSS8fG72EfYDu8fD9CqjbxP0EUdMCQ%40mail.gmail.com.

[Philip Gladstone]

Argh! 

Why does stiff only ever happen when I'm on vacation? 

If any author of software that submits spots feels even slightly guilty, then please let me know. It used to be limited to someone with an odd IP address, but now it seems to be both odd and even.

Philip

Philip Gladstone

To view this discussion visit https://groups.google.com/d/msgid/psk-reporter/CAEDj4kU1a%3DUSN_SgytE8V6zquHp5%3DCSZ3%2BCE%3DABPuVZMLeWHGg%40mail.gmail.com.

[AndyZ K1RA]

Philip,

It is probably a long shot, but any chance you could spin up a packet sniffer (tcpdump, tshark/wireshark) and set it up to filter on the incoming TCP/UDP data port number and save it to a PCAP file for an hour? 

I have no idea how much raw data that would be, so maybe a shorter window, but given the frequency of occurrence the hope is that the file time stamp of an aborted / crashed in and_o or _e tsv file overlaps the PCAP file where one could look to identify the IP(s), and callsign(s) and the app reporting.  

From my recollection, I thought most of that posted data was clear text, so it may be a vector to honing in on the culprit.  

If you can do this, just post the PCAP file somewhere and send me a URL and I'll try to help and investigate further.

andyz - K1RA

To view this discussion visit https://groups.google.com/d/msgid/psk-reporter/CAMd_wWoL-pEzyobpCmXKDwF7HgPW-MvA8WNqbVY66oNx-ipT%3Dg%40mail.gmail.com.