--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/b2c70a4bd4e84eb7a0c6e589164245b6%40Mail04.usask.ca.
For more options, visit https://groups.google.com/d/optout.
Hi Darryl,Wow! That's a lot of rows! 😮
To post to this group, send email to ica-ato...@googlegroups.com.
Wow! That's a lot of rows! 😮
Yah Microsoft! I’ve seen Excel add empty lines to an export before, but never a million.
Once the process is killed you can kill the job queue with (from the AtoM directory):
./symfony jobs:clear
Thanks! This was the piece I was missing. I have a Bash script that clears the cache and restarts all the daemons; I noticed that when I did that the import job re-appeared. This helped!
Yes, the fastest method to roll back the import is to restore a backup. You can try and delete the blank descriptions, but it's complicated because data is added to several MySQL tables for each row imported, e.g. object, information_object, information_object_i18n, object_term_relation, slug, status.
Ya, I assumed it would be a lot of work. I restored from a previous nightly backup and all seems well. I’m wondering about Elasticsearch though. Will the indexes have data in them that doesn’t match the restored MySQL database? Should I run search:populate (or any other commands) after a DB restore?
Thanks for help David.
- Darryl
Thanks David.
If I were to look at this and submit a PR, would you want it against the qa/2.5.x branch or stable/2.4.x?
- Darryl
--
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/b2c70a4bd4e84eb7a0c6e589164245b6%40Mail04.usask.ca.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to
ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CABMx%3Dej3A%2BGOW5FYCpdrvq6ubjAsYyw45sakEFKi6-DX9kaPVQ%40mail.gmail.com.
--
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/b2c70a4bd4e84eb7a0c6e589164245b6%40Mail04.usask.ca.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CABMx%3Dej3A%2BGOW5FYCpdrvq6ubjAsYyw45sakEFKi6-DX9kaPVQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/e819ceae33d64ccca36d036c2942ea83%40Mail04.usask.ca.
Phone: +61 3 6324 3061
Email: Glenn.M...@utas.edu.au
Mail: Locked Bag 1312, Launceston TAS 7250Hi Darryl
Would be interested to know if your CSV imports have worked subsequently.
Was there an issue with the UTF-8 encoding or an absence of the "/n" characters at the end of lines that may have triggered the extensive upload ?
We are about to start uploading large numbers of bulk Archival Descriptions using CSV import.
Previously, we have used LibreOffice-generated CSVs encoded in UTF-8 or MS Excel for MacOS-created CSVs encoded in UTF-8.
Normally, the uploads take only a few seconds, when using only a few dozen lines of spreadsheets for Archival Descriptions.
I had interpreted the user guidelines that the limits of 50 or 100 lines of a CSV were to prevent timeout errors.
We always use CSVs to import using qubitParentSlug as the first column, so that we are only populating child-level ADs as "levelOfDescription" Items to an already existing Collection level AD.
But now, we are experimenting with using CSVs encoded in UTF-8 and created in MS Excel for Windows, for which most of our users have been given new updated Office suites that will cope with UTF-8.
regards
Glenn Mulcaster
Librarian (Access and Discovery)University of Tasmania Library
Phone: +61 3 6324 3061
Email: Glenn.Mulcaster@utas.edu.au
Mail: Locked Bag 1312, Launceston TAS 7250
On Thursday, October 26, 2017 at 3:40:39 AM UTC+11, Darryl Friesen wrote:One of our archives staff imported a CSV file (that she exported from Excel) that she thought had 415 records into our AtoM 2.4 test instance. She was excited to try a web import using a file with more than 100 records (which I gather was a limitation of AtoM pre 2.4??)Excel, in its infinite wisdom, somehow exported not just the 415 lines of her spreadsheet that had data, but an additional 1,048,161 completely empty lines, yielding a CSV file with 1,048,576 records, most of which were simply this:,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,She was unaware of the extra blank records and uploaded this CSV file for import. Only after she realized this morning that the import was still running (after, like, 16 hours) did we notice the problem.Firstly, I’d like to suggest there might be a bug in AtoM. Rather than skip these lines which obviously contain no data, AtoM happily turned them into empty records/objects. LWhich leads me into some questions:What’s the proper way to terminate an import? I can see an active job in the web interface, but there’s no method to terminate it there. On the server I see a couple of symphony processes running that are obviously responsible for the import – is it “safe” to just kill those? I needed to restart Percona for a config change to take effect, and the act of doing that actually seemed to terminate the import (but that left the job in the AtoM web interface listed as “Running” when it is not).Is there a way to remove that phantom “Running” job from the list of active jobs?And lastly, what’s my best option for cleaning out the several hundred thousand empty objects that were created? I’m assuming the quickest method is a restore from backup? I’m doing nightly backups of the Percona databases (using something I found on GitHub https://github.com/sixninetynine/surrogate) so that shouldn’t be a big deal; just curious if there was some other method of undoing an import (I assume constructing SQL queries to “rollback”/undo an import would be more trouble than it’s worth??)Thankfully this was all on a test server!- Darryl----------------------------------------------------------------------Darryl Friesen, B.Sc., Programmer/Analyst Darryl....@usask.caLibrary Systems & Information Technology, http://library.usask.ca/University of Saskatchewan Library----------------------------------------------------------------------"Go not to the Elves for counsel, for they will say both no and yes"
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/a40676c6-2af7-4aa0-a300-23c4838af1fb%40googlegroups.com.