Share Level1 data pool among several projects - best practice

[Cornelius Senf]

Hi all!

We want to share a Level1 data pool among projects working in distinct study sites and we ran into some data management issues where I would like to ask for your experience. 

Each of our projects creates their one Level2 data cube. However, when creating a data cube for a project, the force-level2 routine will run through all images set as QUEUED in the Level1 data pool txt file. This takes up quite some time as the Level1 data pool is rather large and the routine has to skip many images. Is there any way of skipping the Level1 data not needed a priori without messing with the Level1 data pool txt file (i.e., manually setting the images not needed as DONE)? I imagine something like the tile allow list in the Level2 parameter file, but for Landsat/Sentinel scene ids. Moreover, is there a better way for QUEUING already processed images than find/replace? 

Sorry if this is a stupid questions and/or I misunderstood something. I just want to set up our data cube in a sustainable way without creating too much redundancy in data.

Best

Cornelius

[david frantz]

Hi Cornelius,

for spatially distinct study sites, I commonly use different pools to keep things simple. But one pool is okay, too, of course. Just more management.

How many images do you have that this becomes an issue? FORCE uses "grep" to filter for the "QUEUED" images. This is a fairly efficient and standard Unix operation.

If you want to extract some subset, you could use: grep T33UUU queue.txt > newqueue.txt

You can also set not-needed images to anything else, e.g. IGNORE: sed -i '/T33UUU/s/QUEUED/IGNORE/g'

For re-queueing all files, you can do : sed -i 's/DONE/QUEUED/' queue.txt

For re-queueing specific files, use: sed -i '/T33UUU/s/DONE/QUEUED/g'

Basically, a variety of "grep" and "sed" commands work very well in most cases. If you need something more advanced, I recommend using a scripting language.

Does this help?

Cheers,

David

[Cornelius Senf]

Hi David, thanks for the tips. I think we will proceed with one data pool and I will develop some scripts that reset the images needed to QUEUED.

Cornelius