CSV import completed but no data imported

84 views
Skip to first unread message

sally-an...@york.ac.uk

unread,
Nov 11, 2021, 8:03:46 AM11/11/21
to AtoM Users
Hi,

I've had an issue today with trying to import a csv file.  When I imported it, the job report completed instantly (as in, the same second) but no data was actually imported.   The report from this import is below, Unsuccessful csv import.JPG

I then tried the same csv on our test site and it worked normally (job report below)

Successful csv import.JPG

Our AtoM IT support, Jim Adamson, has looked at my csv and not found any issues with it (csv attached).  He noted that the job report on our production version of AtoM was missing the various steps associated with building the nested sets.  Based on the documentation he suggested running the following: php symfony propel:build-nested-set --exclude-tables="term,menu" as advised when the nested set has become corrupted, but he thought it would be a good idea to check with the group and if anyone else has had this issue?  If so, did you find this solved the problem?

I am loathe to test any further imports to our production version in case they all appear at once and I have to delete a lot of entries!

Thanks for any guidance,

Sally

  
B9_8 final csv delete after - Sheet1.csv

Dan Gillean

unread,
Nov 12, 2021, 5:26:00 PM11/12/21
to ICA-AtoM Users
Hi Sally, 

Our support team has seen this a few times in larger production installations, and we are investigating. So far however, we have had problems reproducing this issue consistently or determining the root cause. 

I did look at your CSV and I also don't see anything there that would lead to this kind of error. I would advise you to follow Jim's suggestion and try rebuilding the nested set - and also repopulating the search index. Might as well clear the application cache and restart PHP-FPM as well, to ensure you are not seeing any cached results returned. I think it would be important to confirm whether or not the import simply doesn't run at all or whether the records exist in the database but don't display in the UI because building the nested set and indexing them fail. 

Make sure when you review the site that you are logged in (in case the records are draft), and also clear your web browser cache or else test in a private/incognito browser window. If they don't show up after running those maintenance tasks etc, then I suspect that the entire process is simply failing to launch the import. 

We are tracking this issue internally and doing everything we can to get to the underlying issue and resolve it for 2.7. It doesn't seem to happen consistently, so if you perform the above maintenance tasks and review and find no records, you could try the import again and it will likely succeed. 

Dan Gillean, MAS, MLIS

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/213e29ac-ea2b-476e-913a-b64b2b2161c7n%40googlegroups.com.

Dan Gillean

unread,
Nov 15, 2021, 9:11:54 AM11/15/21
to ICA-AtoM Users
Hi again Sally (and Jim), 

I've followed up with some of our team members who have been looking into this issue. While we still haven't identified a permanent fix, one extra tip: After you've done the above, if you have determined that no records were imported, make sure you restart the atom-worker before attempting to reimport. From what we've seen so far, it seems like the issue may repeat unless the worker is restarted as well. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him

sally-an...@york.ac.uk

unread,
Nov 18, 2021, 11:17:12 AM11/18/21
to AtoM Users
Hi Dan,  apologies for the delay replying, I've been on leave.  Jim followed his original suggestion but it still wasn't working until we spotted your additional tip to restart the atom-worker. I'm relieved to say that did the trick and I've done several normal imports this afternoon.  Thank you very much for your help (as always!)

Sally

Jim Adamson

unread,
May 4, 2022, 7:38:12 AMMay 4
to AtoM Users
Hi Dan,

We've had a couple of incidents of this problem recently, one just this morning. Interestingly, merely running systemctl restart atom-worker resolved it.

I was wondering whether there had been any movement on this bug recently? Do you have a Redmine URL for the bug, or is it tracked elsewhere?

Thanks, Jim

Dan Gillean

unread,
May 4, 2022, 10:15:09 AMMay 4
to ICA-AtoM Users
Hi Jim, 

Our team is pretty confident that this issue is caused by out of memory issues - previously run or retried jobs are not always releasing memory. This is something you can check by getting more information from the worker log, or checking the Ubuntu syslog. Here are commands that our Support team has been using to check for these types of issues: 
  • journalctl -u atom-worker --since today
  • sudo grep memory /var/log/syslog
They have also suggested that the following command can be used to return the memory usage of a worker: 
  • sudo ps -p $(systemctl show --property MainPID atom-worker | cut -d\= -f2) --no-headers -o %mem
We've been doing some work related to this on the following tickets: 
The first ticket has some example configurations using these new worker parameters from issue 13573, and our team also recently added the following to our Ansible roles (see the linked README for more information): 
For now, our team is also using Prometheus and Grafana to create monitoring dashboards with automatic alerts when memory exceeds a certain threshold on an atom-worker, at which point proactively restarting the worker seems to release any held memory. Long-term, we still need to do more analysis and investigation to determine the best way to improve memory use and management in AtoM overall, but the new worker --max-mem-usage parameter included in the upcoming 2.7 release should allow restarts to be automated when a target threshold is reached. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him


Jim Adamson

unread,
May 4, 2022, 10:46:32 AMMay 4
to ica-ato...@googlegroups.com
Hi Dan,

Thank you, that's really helpful. Nothing immediately apparent in syslog or the output of journalctl, but I will keep an eye on worker memory usage using the command you supplied.

Assuming our issue is unreleased memory, an out-of-hours cron task to restart the worker might be enough to keep these incidents at bay.

Thanks, Jim



--
Jim Adamson
Systems Administrator/Developer
Facilities Management Systems
IT Services
LFA/023 | Harry Fairhurst building | University of York | Heslington | York | YO10 5DD

Reply all
Reply to author
Forward
0 new messages