Badges stop collecting/updating data

[jope...@hotmail.com]

I have now made a setup that constinsts of a Ubuntu server, three raspberry pi hubs, 17 beacon/room badges and 11 user badges (+9 extra in case we need them).

This is supposed to be a week long test and everything has so far went seemingly okay but I realized that some of the badges have stopped sending data to the hubs. They do connect to the hubs (As seen in the "Last Seen") but they have not sent any data for several hours despite the badges being on and near the hubs. So far three badges have started doing this (Badges  4-6).

I remember having a similiar issue back when we were doing our first small scale experiment when we used a standalone hub. I could see that the hub connecting to a badge but not fetching any data. Turning the badge on and off did not solve the issue, only the hard reset of the machine fixed it. Is this a known issue? The server and the hubs have been running for little over 24 hours now.

I attached the admin page to illustrate the problem

[Oren Lederman]

it's been a while since I troubleshoot these problems. 

Which version of the firmware are they running? I'm guessing v2 (not 3) because I can see the voltage.

Can you get the hub logs when it tries to connect to the badge?

When you say "hard reset", do you mean the badge, hub or server?

--
You received this message because you are subscribed to the Google Groups "Rhythm Badges" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rhythm-badge...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rhythm-badges/afaf0033-4b94-4882-8a29-abe68c53da72n%40googlegroups.com.

[jope...@hotmail.com]

It should be v3 (when programming the badges I used the badge_03v6 tag).

When I said hard reset I was talking about our earlier experiment where we had a standalone hub (ubuntu machine that worked as a single hub) and badges but no Pi hubs. I had a similiar problem back then where some badges would stop transmitting data (they would connect to the hub). It was fixed by rebooting the Ubuntu hub.

Now we have a local server, badges and 3 Pi hubs. Every single badge is connecting to the hub but at least 3 badges have stopped transmitting data according to the admin dashboard. I have now tried resetting the server, resetting all of the hubs and also rebooting the hubs. I have not rebooted the server yet.

I have attached a screenshot from one of the connection attems. The upper one is a working badge (id1) and the middle one is not working (id6)

[jope...@hotmail.com]

I tried to reboot everything but this did not fix the issue. I also tested that the badges do work by adding a new user with the same badge address and it was able to collect the data

[oren.l...@gmail.com]

03v6 is the hardware version (yes, confusing, should have separated the hardware and firmware repos). Do you mean that you used the release called "Hardware update, 03v6" (tag v2.5) or "New firmware implementation (v3.0) (tag v3.0)" ?

 see it reporting voltages of 0.00 in the chunks, which indicates that it's using firmware version 3.0 and not 2.X. But when they send data to the server, it says the last voltage is 2.880. I don't remember the protocol, so perhaps that's ok. But it does seem to indicate that you are using the newer 3.0 version. You can tell which version it is based on how it blinks in when you turn it on - version 3 does (I think) - Red on for 100ms, and then blinks the green led three times. 

My current guess:
* An unknown bug in V3 of the firmware that we don't know. I did all of my live experiments with v 2.6

* Data corruption on the badge. Either because there was old data there from v2.6 or, something went wrong. Wiping the memory is a bit tricky ... I can't remember how. 

I'd switch the "broken" badges with backup badges. You can do this by changing the MAC addresses but keeping the same name. This way, the new badge will get the same "observed ID" so the analysis is easier. Don't forget to turn off the broken badges because they'll keep broadcasting the ID otherwise.

[jope...@hotmail.com]

When programming the badges, this was my full command:

sudo docker-compose run nrg make badge_03v6_noDebug flashUnlock flashErase flashS130 flashApp

I just did this according to the guide in the main repo. The badges hardware wise are definitely the V3 version (says 03V6 on the board)

[jope...@hotmail.com]

Oh and even thou in the dashboard it does show voltage, it does not show voltage in the data file (it is 0.0)

[jope...@hotmail.com]

I was able to "fix" (read: a hack) it by temporary changing the badge mac address to another badge and after it successfully started pulling data, I switched it back to original mac address.

While this is not ideal, atleast I now know how to fix it for the time being