"Ingest in progress" displayed despite ingest being disabled

[Philipp Conzett]

We have turned of tabular file ingest in our Dataverse installation, as described in the installation guide. I verified: the value for :TabularIngestSizeLimit in the database i 0. However, in a recently submitted dataset, there are several tabular files, both csv and RData, that show the notification "Ingest in progress. Does anyone have an idea why this happens?

Best, Philipp

[Philip Durbin]

Hmm, is a restart of Payara required? The guide doesn't say so.

On Mon, Feb 19, 2024 at 8:14 AM Philipp Conzett <uit.p...@gmail.com> wrote:

We have turned of tabular file ingest in our Dataverse installation, as described in the installation guide. I verified: the value for :TabularIngestSizeLimit in the database i 0. However, in a recently submitted dataset, there are several tabular files, both csv and RData, that show the notification "Ingest in progress. Does anyone have an idea why this happens?

Best, Philipp

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/f6abdf91-2258-46a3-827c-cf287a5b8869n%40googlegroups.com.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

[Donald Sizemore II]

Philipp,

We just encountered similar behavior on ACSS Dataverse (running v5.14). In our case, there were three tabular datafiles in three separate formats (Stata, CSV, TSV) uploaded, likely at the same time. Leonid wonders whether that may contribute to the behavior.

I'm unable to reproduce this on demo.dataverse.org (v6.1), using one tabular file or several.

Thank you,

Don

[Leonid Andreev]

Hi Philipp,

The issue we saw on Don's server does not appear to be related to yours after all. 

Could you please check SELECT ingeststatus FROM datafile WHERE id=...

for the files in question? 

I have a guess as to what may be happening, and it's likely relatively harmless (the files are NOT really being ingested, i.e. only the status flags that are wrong), but I need to confirm. 

best, 

-L.A.

[Philipp Conzett]

Thanks, Don and Leonid, the ingest status of the files at stake is "B".

- Philipp

[Leonid Andreev]

OK, yes, I think I know what's going on. 

Once again, the files are not really getting ingested. I also believe these labels would disappear next time any other files are added to the version. 

I just opened a GitHub issue. We may be able to address it in 6.2. But you should be able to prevent this from happening again with the following workaround: replicate that size limit=0 as a JVM option for the data store (or each one, if you're using multiple stores), for ex.: 

<jvm-options>-Ddataverse.files.file.ingestsizelimit=0</jvm-options>

or

<jvm-options>-Ddataverse.files.s3.ingestsizelimit=0</jvm-options> 

etc. 

Finally, I'm realizing that we never explain how to reset these flags anywhere in the guide. 

Unfortunately, there's no way to do that other than by direct database manipulation: 

UPDATE datafile SET ingeststatus='A' WHERE id=...

best, 

-L.A.

[Philipp Conzett]

Hi Leonid,

We're testing v6.2 and I just discovered datasets displaying File Ingest in Progress. Do you have any updates on whether the fix will make into v6.3? Could you point me to the GitHub issue? Thanks!

Best,

Philipp

[James Myers]

Philipp,

I’m guessing the issue is https://github.com/IQSS/dataverse/issues/10378 - I don’t see any work on it (and it’s not on any board), so I think the workaround below is the only option for now.

 

-- Jim

--

You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/eab955b3-51e8-4054-bb90-cadfb34227can%40googlegroups.com.

[Philipp Conzett]

Thanks, Jim! Is the workaround a one-off task? I guess once issue #10378 is fixed, the fix will also apply to / not mess up with an installation which fixed the problem with the workaround?

Best, Philipp

[Philipp Conzett]

Another question: Will the workaround (and finally the fix in issue #10378) also reset the flags we currently see on some tabular files? Or do we need to identify all the flags and reset them manually?

[James Myers]

The workaround (setting the ingest size limits to 0) won’t clear the existing flags, so I think the db update to switch ingest status from ‘B’ to ‘A’ is still required.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/bcd1e103-dc80-4bc2-a47a-eb8352e508b2n%40googlegroups.com.

[Philipp Conzett]

Thanks for confirming!