Unable to process uploaded file

[Ruth Barros De Paula]

Good evening,

I am having issues running LocusZoom using "Variant from columns" data format. The error message is "We were unable to process your uploaded GWAS file. Please review the logs for the cause, and contact us if you believe the problem is an error."

I thought it could be a problem with the format of my uploaded file, but I tested LocusZoom using a previously uploaded file (that had worked) and I got the same error message.

I wonder if this error is showing up because the server might be overloaded. Any thoughts?

Best,

Ruth

[Andy Boughton]

Thanks for the inquiry.

First, each failed upload should have a log file if you visit the website. (we really should revisit and add a link in the email!) This can help identify common mistakes, like if you chose different “where to find key data” options when you uploaded the file this time. Can you comment on whether any such errors are revealed in the upload log for your file?

1. See list of your uploads, including failed files: https://my.locuszoom.org/profile/

2. On the study in question, see the “ingest logs” button in the top right corner of the page.

One example: sometimes people accidentally specify a marker ID like “chr:pos:ref:alt” as the “chromosome” column. The automated “suggest columns” feature tries to catch common problems, but sometimes people have new file formats that are able to confuse the parser in truly awful ways. It’s worth double checking.

If you still see problems, can you privately send me a my.locuszoom.org link to the failed upload? I’d be happy to take a look at the logs/ contents. We plan to release a new update soon that will make chromosome names more restrictive, and if you have a file that is having trouble, I’d like to make sure we support it in an upcoming release!

-Andy Boughton

abo...@umich.edu

--
You received this message because you are subscribed to the Google Groups "locuszoom" group.
To unsubscribe from this group and stop receiving emails from it, send an email to locuszoom+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/locuszoom/3d9cb83c-f7d6-4c5d-838d-5630bf82f95en%40googlegroups.com.

[Juulia Partanen]

Hi,

I'm having similar issues as Ruth and the log files seem fine:

[ingest][2022-05-23T08:22:32+00:00] Performing upload step: Calculate SHA256
[success][2022-05-23T08:22:33+00:00] Step completed
[success] The GWAS file passed validation. Read the logs carefully, in case any specific lines failed to parse.
[ingest][2022-05-23T08:22:33+00:00] Performing upload step: Normalize GWAS file format
[success][2022-05-23T08:30:11+00:00] Step completed
[ingest][2022-05-23T08:30:11+00:00] Performing upload step: QQ plots and top hit detection
[success][2022-05-23T08:34:53+00:00] Step completed
[ingest][2022-05-23T08:34:53+00:00] Performing upload step: Prepare a manhattan plot
[success][2022-05-23T08:37:51+00:00] Step completed

I've sent you Andy a link to one of the failed uploads.

Best,

Juulia

[Kate Burley]

I've been having exactly the same message over the weekend, with identical log file output...

[Ruth Barros De Paula]

Same, my log files are also the same - basically, look normal.

[Andy Boughton]

Thanks all! I very much appreciate the timely and helpful error reports. I'll spend some time after my morning meeting to debug these examples in depth, and will report back once the issue has been identified and cleared.

Over the last week, we've had a series of malformed files and queries that caused massive memory usage; at least some of these files may have been collateral damage as ingest workers had to be killed due to time / resource limits. I'll triage the ingest failures and try to restart files where appropriate- don't be alarmed if you receive emails for files that were already uploaded. More info will follow later today, and we will be issuing a website update this week to prevent (some of) the problems from recurring.

-Andy Boughton
abo...@umich.edu

Senior Applications Programmer/Analyst
Center for Statistical Genetics
University of Michigan

To view this discussion on the web visit https://groups.google.com/d/msgid/locuszoom/426dea23-cbfd-4f99-a04b-f0ab8a7051b2n%40googlegroups.com.

[Andy Boughton]

Update: it appears that many of the affected files are ok, and may have been affected by a network/email issue. If you see problems in the log, you may get things working by clicking the "retry" button on the file page without any changes (assuming that the log otherwise looks normal). 

I was able to verify that several affected files could be ingested simply by re-trying manually. To avoid spamming users with emails, I'll let each person decide whether to retry their own files. :)

Root cause: we have been seeing some intermittent failures with the "email user on upload step" due to some mail server network issues over the weekend. To avoid any chance that a file gets into an inconsistent state, the code presently marks a file as failing ingest on any error..... including the final "success" email step. (sigh) We will revisit this in the future.

Please let me know if problems continue, and I'll keep an eye on network and email errors over the next few days. Sincere apologies for the inconvenience.

-Andy Boughton
abo...@umich.edu

Senior Applications Programmer/Analyst
Center for Statistical Genetics
University of Michigan

[Kate Burley]

Thank you Andy - working for me now!

[Shafa Talitha]

Good evening Andy, I'm having similar issues as Ruth. But my log files looks like this :

Excluded row 960300 from output due to parse error: list index out of range
Excluded row 960301 from output due to parse error: list index out of range
Excluded row 960302 from output due to parse error: list index out of range
Excluded row 960303 from output due to parse error: list index out of range
[failure] Could not create normalized GWAS file.
[ingest][2024-09-05T11:53:56+00:00] Performing upload step: Normalize GWAS file format
[failure][2024-09-05T11:54:05+00:00] An error prevented this step from completing
Too many lines in the file failed to parse; stopping

I don't understand what's wrong with this one, since the files I've uploaded were practically the same as the others that have succeeded.  Any thoughts?

Best,

Shafa

[Andrew Boughton]

Hmm. The only reason I can think of to see that error would be if some of the rows of your file were missing certain columns. 

Eg, sometimes people will process their data and append two files together, which could cause a series of records late in the file to fail parsing, even if everything before that parsed ok.

-Andy Boughton

abo...@umich.edu

On Sep 5, 2024, at 8:09 AM, Shafa Talitha <shafata...@gmail.com> wrote:

Good evening Andy, I'm having similar issues as Ruth. But my log files looks like this :

To view this discussion on the web visit https://groups.google.com/d/msgid/locuszoom/5a2f8a56-5821-4716-a66e-bdd3244e8b54n%40googlegroups.com.

[Shafa Talitha]

thank you! i figured it out

You received this message because you are subscribed to a topic in the Google Groups "locuszoom" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/locuszoom/OUOWLKayCZw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to locuszoom+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/locuszoom/C721F06F-E33B-45F9-B4D2-3FAFA8B35475%40umich.edu.