Faulty GTFS-RT files: <75% size of the last file

[Nick Spreen]

I've noticed using my own app that sometimes for 30s (which is my update interval) there are no trains listed for a station. I go off of gtfs rt train positions. About 1% of the time gtfs files suddenly contain way fewer trains than at the last poll. The next poll they show up again.

I've started adding a detection for files shrinking down to less than 75% size of the last download. This has mitigated the issue for my application, but of course it doesn't solve the underlying problem.

I first noticed this on the L train a few days ago. Since adding the detection to my server it has only happened on the 123

2024-12-04 23:14:31.968 Skipping suspicious download for: 123.bin (size: 102809 bytes)
2024-12-04 23:04:51.432 Skipping suspicious download for: 123.bin (size: 109746 bytes)
2024-12-04 22:12:53.805 Skipping suspicious download for: 123.bin (size: 126917 bytes)
2024-12-04 22:06:46.350 Skipping suspicious download for: 123.bin (size: 132357 bytes)
2024-12-04 22:02:41.914 Skipping suspicious download for: 123.bin (size: 132454 bytes)
2024-12-04 21:58:36.456 Skipping suspicious download for: 123.bin (size: 131781 bytes)
2024-12-04 21:56:03.870 Skipping suspicious download for: 123.bin (size: 132287 bytes)
2024-12-04 21:54:01.737 Skipping suspicious download for: 123.bin (size: 136531 bytes)
2024-12-04 21:49:27.096 Skipping suspicious download for: 123.bin (size: 133207 bytes)
2024-12-04 21:46:53.238 Skipping suspicious download for: 123.bin (size: 138536 bytes)
2024-12-04 21:42:49.282 Skipping suspicious download for: 123.bin (size: 139657 bytes)
2024-12-04 21:40:16.673 Skipping suspicious download for: 123.bin (size: 136315 bytes)
2024-12-04 21:38:11.140 Skipping suspicious download for: 123.bin (size: 139885 bytes)
2024-12-04 21:36:39.424 Skipping suspicious download for: 123.bin (size: 143010 bytes)
2024-12-04 21:35:07.812 Skipping suspicious download for: 123.bin (size: 141625 bytes)
2024-12-04 21:32:35.216 Skipping suspicious download for: 123.bin (size: 146475 bytes)
2024-12-04 21:29:59.716 Skipping suspicious download for: 123.bin (size: 139420 bytes)
2024-12-04 21:27:57.384 Skipping suspicious download for: 123.bin (size: 142846 bytes)
2024-12-04 21:26:25.710 Skipping suspicious download for: 123.bin (size: 146118 bytes)
2024-12-04 21:23:53.294 Skipping suspicious download for: 123.bin (size: 146644 bytes)
2024-12-04 21:22:52.236 Skipping suspicious download for: 123.bin (size: 149072 bytes)
2024-12-04 21:21:16.586 Skipping suspicious download for: 123.bin (size: 152395 bytes)
2024-12-04 21:11:34.991 Skipping suspicious download for: 123.bin (size: 158212 bytes)
2024-12-04 21:09:32.707 Skipping suspicious download for: 123.bin (size: 150977 bytes)
2024-12-04 21:08:31.663 Skipping suspicious download for: 123.bin (size: 154359 bytes)
2024-12-04 21:07:30.600 Skipping suspicious download for: 123.bin (size: 156884 bytes)
2024-12-04 21:06:29.428 Skipping suspicious download for: 123.bin (size: 159580 bytes)
2024-12-04 21:04:55.117 Skipping suspicious download for: 123.bin (size: 155962 bytes)
2024-12-04 21:03:52.832 Skipping suspicious download for: 123.bin (size: 158945 bytes)
2024-12-04 21:01:20.082 Skipping suspicious download for: 123.bin (size: 11815 bytes)
2024-12-04 11:18:26.472 Skipping suspicious download for: 123.bin (size: 185434 bytes)
2024-12-04 11:14:22.020 Skipping suspicious download for: 123.bin (size: 180778 bytes)
2024-12-04 11:11:18.292 Skipping suspicious download for: 123.bin (size: 179071 bytes)
2024-12-04 11:07:13.104 Skipping suspicious download for: 123.bin (size: 188145 bytes)
2024-12-04 11:04:40.176 Skipping suspicious download for: 123.bin (size: 179612 bytes)
2024-12-04 11:03:08.494 Skipping suspicious download for: 123.bin (size: 183183 bytes)
2024-12-04 11:00:04.511 Skipping suspicious download for: 123.bin (size: 201906 bytes)
2024-12-04 10:56:30.592 Skipping suspicious download for: 123.bin (size: 209556 bytes)
2024-12-04 10:53:57.549 Skipping suspicious download for: 123.bin (size: 204770 bytes)
2024-12-04 10:50:54.050 Skipping suspicious download for: 123.bin (size: 198653 bytes)
2024-12-04 10:48:20.626 Skipping suspicious download for: 123.bin (size: 204287 bytes)
2024-12-04 10:46:18.221 Skipping suspicious download for: 123.bin (size: 199159 bytes)
2024-12-04 10:42:13.724 Skipping suspicious download for: 123.bin (size: 208724 bytes)
2024-12-04 10:39:40.450 Skipping suspicious download for: 123.bin (size: 204143 bytes)
2024-12-04 10:35:05.791 Skipping suspicious download for: 123.bin (size: 200963 bytes)
2024-12-04 10:31:31.718 Skipping suspicious download for: 123.bin (size: 210020 bytes)
2024-12-04 10:27:26.723 Skipping suspicious download for: 123.bin (size: 211721 bytes)
2024-12-04 10:21:50.698 Skipping suspicious download for: 123.bin (size: 210676 bytes)
2024-12-04 10:19:47.947 Skipping suspicious download for: 123.bin (size: 206308 bytes)
2024-12-04 10:15:43.518 Skipping suspicious download for: 123.bin (size: 205037 bytes)
2024-12-04 10:09:05.269 Skipping suspicious download for: 123.bin (size: 209294 bytes)
2024-12-04 10:05:31.367 Skipping suspicious download for: 123.bin (size: 207991 bytes)
2024-12-04 10:01:26.664 Skipping suspicious download for: 123.bin (size: 217733 bytes)
2024-12-04 09:46:38.037 Skipping suspicious download for: 123.bin (size: 98101 bytes)
2024-12-04 03:39:35.882 Skipping suspicious download for: 123.bin (size: 55851 bytes)

it seems the errors are more frequent around 8am and 8pm. times are NYC local time