CSV import completed but no data imported

[sally-an...@york.ac.uk]

Hi,

I've had an issue today with trying to import a csv file.  When I imported it, the job report completed instantly (as in, the same second) but no data was actually imported.   The report from this import is below, 

I then tried the same csv on our test site and it worked normally (job report below)

Our AtoM IT support, Jim Adamson, has looked at my csv and not found any issues with it (csv attached).  He noted that the job report on our production version of AtoM was missing the various steps associated with building the nested sets.  Based on the documentation he suggested running the following: php symfony propel:build-nested-set --exclude-tables="term,menu" as advised when the nested set has become corrupted, but he thought it would be a good idea to check with the group and if anyone else has had this issue?  If so, did you find this solved the problem?

I am loathe to test any further imports to our production version in case they all appear at once and I have to delete a lot of entries!

Thanks for any guidance,

Sally

  

[Dan Gillean]

Hi Sally, 

Our support team has seen this a few times in larger production installations, and we are investigating. So far however, we have had problems reproducing this issue consistently or determining the root cause. 

I did look at your CSV and I also don't see anything there that would lead to this kind of error. I would advise you to follow Jim's suggestion and try rebuilding the nested set - and also repopulating the search index. Might as well clear the application cache and restart PHP-FPM as well, to ensure you are not seeing any cached results returned. I think it would be important to confirm whether or not the import simply doesn't run at all or whether the records exist in the database but don't display in the UI because building the nested set and indexing them fail. 

Make sure when you review the site that you are logged in (in case the records are draft), and also clear your web browser cache or else test in a private/incognito browser window. If they don't show up after running those maintenance tasks etc, then I suspect that the entire process is simply failing to launch the import. 

We are tracking this issue internally and doing everything we can to get to the underlying issue and resolve it for 2.7. It doesn't seem to happen consistently, so if you perform the above maintenance tasks and review and find no records, you could try the import again and it will likely succeed. 

Dan Gillean, MAS, MLIS

AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/213e29ac-ea2b-476e-913a-b64b2b2161c7n%40googlegroups.com.

[Dan Gillean]

Hi again Sally (and Jim), 

I've followed up with some of our team members who have been looking into this issue. While we still haven't identified a permanent fix, one extra tip: After you've done the above, if you have determined that no records were imported, make sure you restart the atom-worker before attempting to reimport. From what we've seen so far, it seems like the issue may repeat unless the worker is restarted as well. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

[sally-an...@york.ac.uk]

Hi Dan,  apologies for the delay replying, I've been on leave.  Jim followed his original suggestion but it still wasn't working until we spotted your additional tip to restart the atom-worker. I'm relieved to say that did the trick and I've done several normal imports this afternoon.  Thank you very much for your help (as always!)

Sally

[Jim Adamson]

Hi Dan,

We've had a couple of incidents of this problem recently, one just this morning. Interestingly, merely running systemctl restart atom-worker resolved it.

I was wondering whether there had been any movement on this bug recently? Do you have a Redmine URL for the bug, or is it tracked elsewhere?

Thanks, Jim

[Dan Gillean]

Hi Jim, 

Our team is pretty confident that this issue is caused by out of memory issues - previously run or retried jobs are not always releasing memory. This is something you can check by getting more information from the worker log, or checking the Ubuntu syslog. Here are commands that our Support team has been using to check for these types of issues: 

• journalctl -u atom-worker --since today
  
• sudo grep memory /var/log/syslog

They have also suggested that the following command can be used to return the memory usage of a worker: 

• sudo ps -p $(systemctl show --property MainPID atom-worker | cut -d\= -f2) --no-headers -o %mem
  

We've been doing some work related to this on the following tickets: 

• https://projects.artefactual.com/issues/13573
  
• https://projects.artefactual.com/issues/13575
  

The first ticket has some example configurations using these new worker parameters from issue 13573, and our team also recently added the following to our Ansible roles (see the linked README for more information): 

• https://github.com/artefactual-labs/ansible-atom#install-and-configure-memprof-extension-and-use-max-job-count-and-max-mem-limit-features

For now, our team is also using Prometheus and Grafana to create monitoring dashboards with automatic alerts when memory exceeds a certain threshold on an atom-worker, at which point proactively restarting the worker seems to release any held memory. Long-term, we still need to do more analysis and investigation to determine the best way to improve memory use and management in AtoM overall, but the new worker --max-mem-usage parameter included in the upcoming 2.7 release should allow restarts to be automated when a target threshold is reached. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/47b83b38-4f46-4ee1-ade1-51cddd377aa2n%40googlegroups.com.

[Jim Adamson]

Hi Dan,

Thank you, that's really helpful. Nothing immediately apparent in syslog or the output of journalctl, but I will keep an eye on worker memory usage using the command you supplied.

Assuming our issue is unreleased memory, an out-of-hours cron task to restart the worker might be enough to keep these incidents at bay.

Thanks, Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZJM-3DwaJcJjMaXxj5q%2BF4uAgbkSdgXaW%3DtqvWkSKEBuA%40mail.gmail.com.

--

Jim Adamson

Systems Administrator/Developer
Facilities Management Systems
IT Services

LFA/023 | Harry Fairhurst building | University of York | Heslington | York | YO10 5DD

+44 (0)1904 323859

Email disclaimer: https://www.york.ac.uk/docs/disclaimer/email.htm