fed3viz problem uploading files

[Martina Maria De Cristofaro]

Hi All! I am having an issue trying to use fed3viz. When I try to upload my data (by clicking on "load files"), the software gives me this error message (see attached). Does anyone have any idea how to solve this problem?

Thank you! 

[Tom Earnest]

Hi Martina!

I believe this looks like an issue I've encountered before where there are some missing entries (either in individual cells or whole rows) within the FED3 data.  Would you be able to send me or share one of the files which is unable to load, and I could verify this?

Best,

Tom

[Martina Maria De Cristofaro]

Hi Tom! thank you for your answer. I don't think the problem is the file, as I am able to upload it on one of the computer I have. However, we have that message error on all the other computers we have in our lab - this is why I believe it has to be with something about the software and not the file. 

[Tom Earnest]

Hi Martina,

If its something that is happening with FED3VIZ on some computers and not others, I would hope that is a difference in the version of the program across computers.  Can you confirm what version you are running (it should say on the "About" tab)?  And also whether its the same version on all the computers you are using?  If its not that, then it could be something related to the operating system, which I would know less about how to diagnose (maybe unless it's a Mac/PC issue).

I looked back at an email chain I had with a different user, and I believe they were having the same issue as you (at least they saw an identical error message, I believe).  Their issue ended up being that the CSV had all these empty rows at the end - this sort of thing was not visible in Excel, but is in a text editor (I have attached a picture).  You could check if your data also looks like that?  I can say that the error message you shared is pointing to a line which is trying to populate the "End Time" column in the File View.  It is looking at the last timestamp in the file, and trying to turn it into a nicer string (e.g. "1/29/2021 16:08" > "Jan 29th, 2021 16:08").  But it is finding an empty value instead (the "NaTType"), causing an error.  So your message sounds consistent with the problem this other user saw - it might not be the same, but it is worth ruling out.

Would you still be able to send one of the problem CSV files?  If I am able to reproduce the error, I have a much better chance of being able to determine what the issue is.  If I am able to load your file, that will also be helpful information.  

Best, 

Tom

[Martina Maria De Cristofaro]

Hi Tom! The FED3viz software is the same version across all computers and is version 0.5.1. I attach one our files so you can look at the data yourself. 

thank you so much!

[Tom Earnest]

Hi again,

Thanks for sending, and confirming the version!

It looks like this data has that same problem where null rows have been created at the end.  These are what the last five rows of the data you sent me look like (I opened with Notepad):

6/24/2022 7:42,1.12.0,Free_feed,0,3.93,1,0,Pellet,Left,168,154,822,0,Timed_out,512,nan
6/24/2022 7:50,1.12.0,Free_feed,0,3.93,1,0,Pellet,Left,168,154,823,0,Timed_out,524,nan
6/24/2022 7:53,1.12.0,Free_feed,0,3.93,1,0,Pellet,Left,168,154,824,0,Timed_out,141,nan
,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,

The last two rows are the problem.  They won't show up in Excel (or maybe other programs) b/c they do some invisible cleaning of loaded data.  FED3VIZ is reading stuff "as is", so it tries to interpret those empty rows and runs into the error you see.  I also get the same error you reported when trying to load the file.

The immediate fix for this is to manually remove those rows with a text editor (Notepad on Windows/TextEdit on Mac, e.g.).    Or if you are familiar with Python/R, this would be a fast operation to apply for many files.  I am attaching a copy of your file for which I have done so - I am able to load this into the program and it seems to be working.  Just make sure you restart FED3Viz before trying to load it.  

You are the second user I have seen with data like this.  I'm not really sure where it comes from - directly from the FED3 logging, for from some other interaction with the data after.  I would like to update FED3VIZ to anticipate this problem.  I'm unsure of an ETA on that, however.  If you have a lot of data that you can't load b/c of this error, I can help with automating the cleaning if that would be helpful.  

Let me know if you have any other questions?

Best, 

Tom

[Martina Maria De Cristofaro]

Hi Tom, sorry for my super delayed answer. Your help have been precious. thank you so much!

Best, Martina.