Concatenating data from two FEDs

[Ivan Alcantara]

Hi all,

I'm monitoring feeding over days and I'm using two FEDs per cage just in case one of them gets jammed. What do you think is the best way to combine data from both FEDs?

I tried using FED3viz to concatenate csv files from both FEDs but then it says that's not possible because some dates overlap. I tried manually combining the rows in Excel and then sorting the rows based on the dates. When I convert the resulting file to a csv, FED3viz doesn't recognize it as FED data. I'm guessing maybe it's because the values in the "inter-pellet interval" column are not accurate anymore?

Any help on this would be appreciated! I'm also trying to figure out how to concatenate FED data on Python and plot pellets consumed over time. If anyone has a Python script that could do something like that, please let me know!

Thank you,

Ivan

[Lex Kravitz]

Hi Ivan!

Can you attach a couple datafiles you want to join?  This sounds like something that would have to be taken care of in Python, but once they are concatenated the resulting file could probably go back through FED3VIZ. 

Were there multiple mice in each cage or just one?  I'm curious how you'd get exactly the same time-stamp on both files.  Anyway please post back a couple files, I can take a look!  -Lex

[Ivan Alcantara]

Hi Lex,

There's only one mouse per cage. Here are a couple of files. I was able to concatenate files together in Python but when I sort the rows based on the time stamp using 

df = df.sort_values(by=['MM:DD:YYYY hh:mm:ss'])

it didn't completely work. The rows are in order based on the dates but chunks of the times are jumbled. Is the MM:DD:YYYY hh:mm:ss format not completely recognizable by Python?

Thanks, Lex!

[Ames Sutton]

Hey all, 

I've been thinking about this for some experiments I am doing, too!  I do think for this to work you would have to somehow time lock the two FEDs (maybe via TTLs sent to the same computers?, but I'm interested to hear others thoughts. 

Ames

--
You received this message because you are subscribed to the Google Groups "FEDforum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedforum+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fedforum/86210e8c-2c32-4500-ad21-b733ed7188c5n%40googlegroups.com.

[Ivan Alcantara]

Ah, that's a good point Ames. So the FEDs are not time-locked but I did set up the RTC, so the internal clocks of the FEDs should be very close to each other (maybe shifted by a couple of seconds). But I guess even with that small discrepancy, it would be possible that two pellets from two FEDs were taken at the same exact time (according to their individual clocks), so that may be the issue. 

[Ivan Alcantara]

Yep, just checked those two files on python and there's 10 rows out of 1945 with the same time stamps. What do you think I should do with those rows?

[Matias Andina]

Shameless plug here, but consider using FEDWatcher https://github.com/matiasandina/FEDWatcher.

The key function here is that you could have the two feds sending info through serial to a RPi (or anything that can read a serial input) and timestamp all info using that clock instead of the one on the FEDs.

FEDWatcher essentially rebuilds the dataset line by line but with new timestamps. You can bind the resulting data.frames using the time from the Pi (which we add to the FED3 data) and they should naturally sort.

You can do up to 4 FEDs into one Pi. Happy to talk more about this.

Best,

Matias

To view this discussion on the web visit https://groups.google.com/d/msgid/fedforum/e30207f8-f8e1-48e5-af03-450b28bb1b44n%40googlegroups.com.

[Lex]

If you mostly care about total pellets and the RTCs are set roughly the same, I would just add 1 second to those 10 pellets and see if that allows you to concatenate the files and send them through FED3VIZ.  Post back if that works!

Also to follow up on Ames' point, it would be possible to set up two FED3s up to work as a pair, with one FED3 logging its own pellets and also pellets from the other in the same datafile.  Let me know if you want to do this, I don't think it would be hard to do.

To view this discussion on the web visit https://groups.google.com/d/msgid/fedforum/09c07811-88f5-441a-addd-26fbf961dcd8n%40googlegroups.com.

[Alcantara, Ivan]

Lex - I'll try your suggestion of adding 1 second to those duplicated pellets. Thanks.

But actually, I looked at the data again and some of these pellets with the same timestamps actually come from the same FED. The inter-pellet interval is then logged as 0. How is that possible? In this case, should I just delete these pellets from the analysis?

--

Ivan C. Alcantara

Neuroscience PhD Student

Brown University - National Institutes of Health

[Lex]

I was wondering that too - some of your pellets even have ITIs of 1s, and it seems hard to believe a mouse could take 2 pellets 1s apart, it often takes longer than 1s for it to dispense.  I'm not sure why this is happening, but you may want to filter these low ITI pellets as well. Or at least keep track of them and try to determine if they are really pellets being taken or if your mice are tricking the FED3 into registering two pellets somehow, for instance by bouncing the pellet around in the well before taking it.  We designed the pellet well with a small opening to reduce this possibility but it may be something to consider. 

One simple way to stop this could be to introduce a 10s timeout after each pellet removal or something like that.

[Alcantara, Ivan]

Finally figured it out! If anyone else is interested in doing this, here's what I did:

1. Clean up individual FED data files to remove rows (i.e. pellets) that have an inter-pellet interval of less than a certain threshold.

2. Concatenate the FED files you want to merge.

3. Sort the rows based on the timestamps. If you're using Python, the timestamps in the original FED files (column MM:DD:YY hh:mm:ss) are read as strings, not objects, so you can't sort based on these values. Instead, convert the timestamps to objects, save these new values in a new date/time column, and sort the rows based on this new column.

4. Reset the indexing of the rows. The updated indices can then be used to calculate the total pellet count. Use these values to fill the Pellet_Count column. 

5. If you care about inter-pellet intervals, you'd have to recalculate these also and update the InterPelletInterval column. 

6. Delete the date/time column you created, and the new file can now be used in FED3WIZ!

There's probably a more efficient way of doing this but I hope this helps! Thanks everyone for your advice. 

[Ames Sutton]

Hey Ivan (and others),

How do you calculate IPI if your time stamps aren’t somehow using a central clock that aligns them? In my hands even if the two FEDs were set on the same computer they won’t be perfectly aligned. 

Ames 

To view this discussion on the web visit https://groups.google.com/d/msgid/fedforum/CAD0XzD5L77a8Nxb1H6YCUAdDDCaaHNUhabS_FCj3hFftytHD1w%40mail.gmail.com.

--

Ames Sutton, PhD

Postdoctoral Fellow, Krashes Lab

NIDDK, National Institutes of Health

cell: 317-435-4631

[Alcantara, Ivan]

I think one easy way would be to grab pellets from two FEDs at the exact same time, then look at the data files to see what the discrepancy is, and adjust the timestamps for one FED accordingly.