FED3 display/program freezes

[Hwei Ee Tan]

Hi

With v1.1.48 and v1.8.3, has anyone encountered FED3 just freezing out of the blue?

Battery level is fine >=3 bars, and there is no jam or blockage. We will just discover the device (on free feeding) in the morning at 10am with no pellets in the tray, but the screen clock will show a past timing (e.g. 1am) where the pellet was dispensed. We also tried to purposely put in a pellet (or occlude the sensor) and remove the pellet, but nothing seems to get detected. When we restart the FED3 device, then it starts dispensing and detecting pellets and running again as normal.

I have been looking into modifying the FED3 library cpp (from v1.8.3) and Classic FED script hoping to solve this problem, as well as other customizations (e.g. logging all pokes to SD card even in free feeding, logging pellet-dispensed and pellet-retrived events separately, etc). Would you have any advice on a good way to debug? I thought of perhaps getting each function to display something on the screen when it is running, so at least I can get a clue of which part of the script did it get frozen at. I wonder if it could be some loop or condition it got stuck in.

Thanks very much!

Hwei

[Lex Kravitz]

Hi Hwei,

Thanks for writing!  The only way I've seen FED3 freeze up is if the pellet well is dusty causing FED3 to think there is a pellet in the well. When a pellet is in the well the device enters a sleep mode to save battery life, and at that point both pokes become unresponsive and the time on the display stops updating until the pellet is removed.  This would be my prime suspect. Can you clean both sides of the pellet sensor with a cotton swab and some water or alcohol and see if that fixes it? 

Is this always happening with the same device?  If so that would also suggest a hardware issue with that specific device and not a code issue.

To keep the forum topics organized, can you start another thread with what functionality you're trying to add via the library and I can advise you on that?  Thanks!  -Lex.

[Lex Kravitz]

Reposting question from Renée Poelma:

Dear all,

I've been trying to habituate my mice to eat from the FED3 in freefeeding mode. I used the FreeFeeding protocol from the FED3 library. I also tested the devide for jamming (I tested 200 pellets), and everything worked fine. There would be a pellet jam sometimes but usually the device would solve this by itself within 10 min. 

The problem I have now is that when I put the FED3 in the cage with the mouse everything works fine, but when I come to check the next day, or even 6 hours later, the FF mode is not working anymore. My mouse knows how to take the pellet out, for example the pellet count is 28 when I come to check. But there is no pellet in the well and when I move a tweezer trough the well (to mimic taking a pellet out) nothing happends. When I restart the FED3 everything works again.  Does the FED3 go into sleeping mode if my mouse hasn't used it for a certain time? And how can  Icange this?

Kind regards,

Renée Poelman

[Lex Kravitz]

Hi Renée,

The FED3 will go to sleep whenever it thinks there is a pellet in the well.  It is not dependent on time, the sleep mode is activated as soon as the device boots up.  I suspect somehow your device is going to sleep *thinking* there is a pellet in the well, even though there is not.  Can you try a couple things?

1) Can you use compressed air to clean out both side of the sensor in the pellet well?  If the sensor gets dusty it can become unreliable and think there is a pellet blocking it when there is not.  You can also try a cotton swab with ethanol and wipe both sides of the sensor.

2) What kind of bedding are you using?  If you have finely ground bedding small pieces can get lodged in the pellet sensor and also cause the FED3 to think it is sensing a pellet.  We often use Isopads, see post about them here: https://groups.google.com/g/fedforum/c/Bayii5EYSh8/m/aqLqU56ZCgAJ.  We also have some bedding in our facility that is ground into larger pieces, and we can get away with using that. 

3) Do you have more than one FED3 device?  If the problem is specific to a single device that would help narrow the problem down.

You also asked about turning off sleep mode.  If you want to try this, insert this line right after fed3.begin() and re-flash the device.  Battery life will take a big hit without the sleep mode but it may help you with troubleshooting this:

fed3.disableSleep();

See more info at: https://github.com/KravitzLabDevices/FED3_library/wiki/3.-Functions

Please write back with an update! Best, -Lex

[Matias Andina]

Not sure if related, but I have found one unit that has a very similar issue. 

I cleaned and dusted it. I inspected the photointerrupters and they seem to be working, as in, I can measure current with the multimeter on the PCB. 

When using the menu to choose the program the FED will run, pokes work right until the program is selected. Then, the unit stops responding to all pokes and events (doesn't try to move the motor, thinks there's a pellet in the well, I can see the pellet stopwatch running on the screen). I tried several programs but they all happen to have the same problem. This only happened with one unit and I couldn't pinpoint what's wrong with it, I will disable sleep and try again.

--
You received this message because you are subscribed to the Google Groups "FEDforum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedforum+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fedforum/f69a437a-2363-4ce0-91d5-340835b53c06n%40googlegroups.com.

[Lex Kravitz]

On last idea could be to try a different SD card - I have experienced some SD cards that work fine when plugged into a computer but do not work with microcontrollers.  If the SD card crashes it can hang the device.  It may be worth trying to swap the SD card out for one you know is working in the FED3.

[Renée Poelman]

Hi all,

First, thank you Lex and Matias for helping! 

Here is an update after I tried the suggestions from Lex: 

1) I used both compressed air and a Qtip with ethanol to clean the pellet well. 

2) We use standard bedding so our lab might need to change that. I cleaned and checked the FED3 for bedding. I am testing the FED3 in the office now and I keep having the same problem. When I start the FED3 in FF mode, a pellet will dispense, when I take the pellet out a new pellet will dispense etc. However, when I take a the pellet out ~20 min after it was dispensed, there will be no new pellet dispension. When I restart the FED3 it repeats this process.

3) We are currently waiting for our order placement so we only have one deivce in the lab now, so we only have 2 FED3 devices. We tested the second device in the office and that one doesn't have this problem. So I guess its a specific problem for this device. 

Best,

Renée

Op maandag 17 januari 2022 om 19:40:19 UTC+1 schreef lex.k...@gmail.com:

[Lex Kravitz]

Hi  Renée,

Given that you have a working device it confirms that this issue is not a problem with the code.  It also sounds like it's not an issue with bedding, given that you can replicate the issue on your desk.  Having a working device is great, it means you can swap some parts to try to track down the issue.  A couple more ideas:

1) The simplest thing it could be is the SD card - I have had SD cards go bad before, and a if the SD crashes it can cause the FED3 to become unresponsive.  I don't know why that would happen after 20 minutes of operation but can you swap the SD cards between the FEDs to check if it's that?

2) You can also remove the Adalogger boards (black board in below photo, it is mounted on headers so it can be pulled out), and swap them between the FEDs.  If the bad FED3 is still problematic with the Adalogger from the good FED3 it suggests a problem with the main circuit board.  If you purchased this from OEPS I would suggest contacting them for a replacement at that point, as this is unfortunately not something that is easy to repair.

Please write back if either of these helps!  -Lex

[Renée Poelman]

Hi Lex,

Yes we tried to switch the SDcards but it didn't fix the problem. I'm afraid of messing up our second (working) FED if I'm going to move the Adalogger boards around, so we haven't tried that here. However, I have another update:

We put the  fed3.EnableSleep (); function in the void setup, after fed3.begin(); and set it to false:

After flashing it to the FED3 the FreeFeeding mode seems to keep going instead of freezing every hour. I'm not sure if this fixes the bigger problem, but it might help.

Best,

Renée

Op vrijdag 21 januari 2022 om 21:10:40 UTC+1 schreef lex.k...@gmail.com:

[Lex Kravitz]

OK we're making progress on this detective story!  This seems to suggest that this specific FED3 device is having an issue where sometimes when it goes to sleep it does not wake up.  This would explain why this device works normally when you disable sleep.  This is fine BTW, the battery life will take a hit but there's nothing wrong with running the device with sleep mode disabled.  I would test is with a mouse in that mode to confirm that you have really eliminated the issue by turning off sleep.

It will be tricky to track down the reason that the sleep mode is causing issues for this device.  You could purchase another Adalogger board from Adafruit, so you can see if the board is the issue without taking apart your working FED3.  That might be the first thing I'd recommend.  Or if you purchased it from OEPS I'd reach out to them and see if they'll replace it.

[Samuel Walker]

Hi Lex/all,

I've had a similar problem to this with all of my FED3s recently. I didn't use them for a couple of months, and when I came to use them again, every one at some point during the night would stop dispensing pellets. I've noticed that when this happens, the clock timer freezes, and the FED3 is unresponsive to pokes. If I turn it off and on again, it works just fine, and the device still has plenty of battery.

If I turn on the FED3 in free feeding mode and collect pellets manually, it will keep delivering them, even if there is a gap of up to an hour between pellet retrievals. However, when I waited 3 hours between pellets, I found that the FED3 had frozen in the meantime and did not deliver a new pellet.

Any idea what could be causing this freezing? As I said, it happens on all the FED3s. I don't think it's the code, as I have used the same code before with no issues.

Thanks!
Sam

[Lex]

Hi Sam - just so I'm clear:

Did these FEDs work fine last time you used them, and you're certain no one re-flashed them before they started freezing?  If so, it would suggest something in the hardware has changed since you last used them.  Are there any other clues like a bad clock time or anything else unusual going on?  By "plenty of battery" do you mean it's fully charged?  If not, can you charge the batteries fully to make sure it's not a low battery?

Assuming it's a hardware issue that came up while the devices were sitting dormant: one possibility is that the photo-interrupters became dirty or dusty while sitting and are giving unreliable readings.  Can you clean the inside of the pokes and pellet well with a cotton swap and 70% ethanol to make sure they didn't get dusty?  Compressed air can help here too.  The logic of the FED3 in free feeding mode is that it drops a pellet and monitors the pellet for 60s so it can time how long the mouse takes to retrieve it.  If the pellet (or dust) is still blocking the well after 60s the FED3 thinks the pellet was not retrieved and goes to sleep to save battery life. It wakes up again when the pellet is removed.  So it sounds like your FEDs are going to sleep while a pellet is in the well, but not waking up when it's removed.  One possibility is that dust in the pellet well is causing unreliable readings so it's not able to detect the removal of the pellet.  Can you see if cleaning it helps?

As a work-around "hack" if this doesn't work and you need to get going on your experiments quickly, you can disable sleep entirely and that should avoid this specific issue.  Please post back if you figure this out!   -Lex

To view this discussion on the web visit https://groups.google.com/d/msgid/fedforum/21e1c17e-15dc-49fc-aaa7-d048b8095ae1n%40googlegroups.com.

[Samuel Walker]

Hi Lex,

I'm sure it's not a code issue, because I also re-flashed them with the original "Menu-Example" code from the FED3 library (and fully charged them), and all 5 devices still displayed the same issue.

The clock time is "fine" - meaning, up until the time the device freezes, it keeps track of the time; and after freezing if I turn it off and on again the time is still aligned with real time (so the internal clock was still running as the device was frozen).

When operating normally, if there is still a pellet in the well but the device is asleep, the clock on the display keeps ticking, and it remains responsive to pokes. The weird thing I am experiencing is that at some point, each device will freeze, on-screen clock time stops (which also means I can see the exact time that the device froze), and nose pokes in either well won't register either on screen or in the log file.

Based on this, I figured it wasn't a sleep issue. However, you were totally right - disabling sleep prevents the issue. But I have no idea why - so any more ideas welcome! In any case, for the time being I can just run the FED with sleep disabled and buy a load of spare batteries so that I can keep running multi-day experiments. Thanks for the help!

Sam