FEDs auto-restarting

[Taaseen Rahman]

Hi Everyone,

Has anyone had trouble with FEDs auto-restarting when kept in cages for long durations while plugged in? I've had issues where some of my FEDs automatically restart and sometimes I end up being forced to exclude those datasets unfortunately. I was wondering why this might be happening, and if anyone might have any way to tackle this.

Sincerely,

Taaseen

[Matt Gaidica]

I don't have a FED in front of me to confirm, but I believe this may be due to memory leaks/overruns. This occurs when memory is not freed. I would include the following function somewhere in your FED routine and see if the free heap is decreasing over time. If it is, you will eventually run out of memory and it will reboot.

extern "C" char* sbrk(int incr);

void setup() {
  Serial.begin(115200);
}

void loop() {
  delay(5000);
  printHeapStatus(); // example usage
}

void printHeapStatus() {
  char stack_dummy = 0;
  char* heap_end = sbrk(0);
  int free_memory = &stack_dummy - heap_end;

  Serial.print("Free heap memory: ");
  Serial.print(free_memory);
  Serial.println(" bytes");
}

[Lex]

In addition to Matt's idea can you post more info about your setup?  What do you mean by "long durations"?  Are you leaving it plugged in for hours, days, or weeks? 

From your post, it sounds like the resets may only happen when you leave it plugged in in the cage - is that true? The FED will reset if the power line shorts to ground - I'm speculating, but I could see moisture building up in the USB port in a humid cage, which could cause a short.  If these resets only happen when plugged in, can you run on battery and then plug in every week or so to charge? 

Can you also explain more about why you need to discard your dataset if there's a reset?  I know you want to solve this issue, but so you don't lose data you've already collected, can you concatenate sessions together across the reset?

--
You received this message because you are subscribed to the Google Groups "FEDforum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedforum+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/fedforum/9dd8532e-4819-4f25-adff-3ed64b784aa8n%40googlegroups.com.

[Taaseen Rahman]

Hi Matt,

Thanks for this. I’ll try incorporating this. Once the free heap reduces, I just change the SD card right?

--

[Taaseen Rahman]

Hi Lex,

I have it plugged in for weeks (slightly over two weeks). I’ve noticed two or three FEDs running into this problem repeatedly, and I’ve checked the SD cards too and they seemed fine. One time I saw it auto restart too. And yes, I do leave them plugged in throughout this time. I am guessing I’m experiencing what you’ve described because I haven’t really noticed any memory issues when I use them. 

I’ve previously concatenated files when this happened but I can’t use this if an auto restart influences my experiments unfortunately. This time around I’m training the mice on a deterministic bandit task session and restarting it throws the sequence of probability switches off. I would imagine such auto restarts to be a problem in the actual bandit task too as we don’t want any block probabilities associated with the poke-ports to repeat consecutively.  I could maybe still use the data by concatenating and ensuring all blocks don’t have the same probabilities for left and right pokes, but I’m not sure if that’s the most efficient way to go about it. 

I could try leaving them unplugged and plug them in less frequently actually. But I’m also worried that some FEDs may not hold charge as well as others, so I’m also open to any other suggestions that would allow me to leave the FEDs plugged in continuously. 

To view this discussion visit https://groups.google.com/d/msgid/fedforum/CAPM2s1Sfmz1v5aQJJV2jzSFruns8nkd3grCaK1QPAmi5mxgNVw%40mail.gmail.com.

[Lex]

The heap issue is related to the processor free memory, not the SD card storage. If there's an issue we'd have to revisit the library code to try to nail down what's causing it. Please post if you find an issue!

Regarding concatenating files, we run a lot of deterministic and probabalistic bandit tasks, we get ~8-10 switches per day so I don't worry about it if a switch is shorter or longer due to an occasional technical or user error.  Most of our metrics are looking at one trial back (ie: win stay) which shouldn't be affected.   If you wanted to be conservative you may be able to filter out the incomplete blocks around a reset and then concatenate?  Just throwing out ideas because I hate throwing out data 🙃

[Taaseen Rahman]

Oh I see. I’ll take a look at the heap issue too. Funnily enough, I’ve never experienced this when I ran closed economy for over two weeks. But I did experience this with a modified version of the left-right sequence, and a customized stop sequence one too. Resets in the former resulted in some mice experiencing a reset in the time interval required to finish a left-right poke, (which was meant to reduce to 3 seconds after the mice reached 320 pellets, but the reset caused it to go back to the starting 60 seconds interval). And I’ve had some issues with this in the stop sequence trials that I couldn’t quite resolve by concatenating the files too (the concatenation worked fine but when I tried running the files through another python code, it wasn’t giving me the correct summary data unfortunately). 

I haven’t taken a closer look at the code but I’ll check if there’s something different in the closedecon code vs the left-right and stop sequence ones that might be responsible? Although I wouldn’t think that’d be the case.

The suggestion about filtering out incomplete blocks seems pretty good actually. I’ll try that out if need be. I think that way I’d still be able to use all the data.

I’ll definitely try troubleshooting this further. I’ll play around with a spare FED and see if it’s the heap issue or if it’s something else. 

[Matt Gaidica]

Free heap is the amount of memory you have hanging around to do stuff with. It's natural to decrease during the first few invocations of unique functions/utilities, but if it is a steady decline, your Arduino will eventually run out of memory and its recovery method is to reset entirely. You can do some napkin math to determine if it is going down, at what rate, and see if that aligns with your observed reset duration.

[Taaseen Rahman]

Hi Matt,

That sounds helpful actually. I did notice that the FEDs that have these problems tend to restart around after a week or so. It could be because of what you've described.

Sincerely,

Taaseen

To view this discussion visit https://groups.google.com/d/msgid/fedforum/8481bdb2-1569-4882-8f85-de0d9800ce1dn%40googlegroups.com.

[Lex]

Interesting!  Taaseen if you can check the heap memory that with be useful to know. If it's an issue we can try to address it!

To view this discussion visit https://groups.google.com/d/msgid/fedforum/CANaGbXq4d8pv8OQgojNciVM0LdPCd59Rg_o62ZFf%3DkiConJPAQ%40mail.gmail.com.

[Taaseen Rahman]

Sorry my last reply didn't have the forum tagged. I'll try this out on a spare FED. But where would I implement the heap memory code:

extern "C" char* sbrk(int incr);

void setup() {
  Serial.begin(115200);
}

void loop() {
  delay(5000);
  printHeapStatus(); // example usage
}

void printHeapStatus() {
  char stack_dummy = 0;
  char* heap_end = sbrk(0);
  int free_memory = &stack_dummy - heap_end;

  Serial.print("Free heap memory: ");
  Serial.print(free_memory);
  Serial.println(" bytes");
}