gsutil docker

[Ma Wo]

Hello everyone!

I'm new to FORCE and to Docker... so bear with me!

While the download of Landsat data works without any problems, I'm encountering some problems when I want to download Sentinel-2 data.

I followed the instructions to configure gsutil and have a .boto file, but I'm not 100% sure how to use the command that is described here as a docker work around. Am I right to assume, that I'll just have to integrate that part into my command like so:

sudo docker run -it -v /home/user/:/credentials --env BOTO_CONFIG=/credentials/.boto -v /media/user/WD500/force_data/:/data fegyi001/force force-level1-csd -c 0,80 -d 20190601,20190630 -s S2A,S2B /data/meta_catalogues /data/level-1-pool queue /data/shape_4326.gpkg

Unfortunately I always end up with the following error:

Downloading S2B_MSIL1C_20190601T102029_N0207_R065_T32UPC_20190601T123553(1 of 80)...
tput: No value for $TERM and no -T specified
|         |   0 %                                                                   Traceback (most recent call last):
  File "/usr/local/bin/gsutil", line 7, in <module>
    from gslib.__main__ import main
  File "/usr/local/lib/python3.6/dist-packages/gslib/__main__.py", line 53, in <module>
    import boto
  File "/usr/local/lib/python3.6/dist-packages/gslib/vendored/boto/boto/__init__.py", line 53, in <module>
    config = Config()
  File "/usr/local/lib/python3.6/dist-packages/gslib/vendored/boto/boto/pyami/config.py", line 63, in __init__
    self.read(BotoConfigLocations)
  File "/usr/lib/python3.6/configparser.py", line 697, in read
    self._read(fp, filename)
  File "/usr/lib/python3.6/configparser.py", line 1015, in _read
    for lineno, line in enumerate(fp, start=1):
  File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3148: ordinal not in range(128)

If that isn't the right way, I'd happy if someone could enlighten me :)

Cheers and regards,

Marco

[Stefan]

Hello Marco,

if your boto file was created properly and everything else is set up correctly...this may be a problem with locales.

What locale is the docker environment and the environment where you created the boto file? Did you use any special characters (i.e., Umlaut)?

Stefan

[Stefan]

I'm assuming the 'user' in ' /home/user/:/credentials ' was just replaced in your post to not reveal your real username? Otherwise the error may be there.

[Ma Wo]

Hello Stefan,

yes you're right about the 'user' part and I'm pretty sure that my .boto file is okay and free of any special characters or Umlaute. I mean, the gsutil config process is pretty straightforward or am I missing something?

However, I just repeated the process to be sure that I did everything correctly and noticed that the [docs](https://force-eo.readthedocs.io/en/latest/howto/level1-csd.html#gsutil-configuration) describe that a project name is not required and can be skipped by pressing enter. I'm not asked for a name, but a project-id. This can't be skipped, so I created a new project to provide an id. Is it possible that the config process changed or something like that?

My docker environment and the environment in which I created the .boto file are the same. Output of the 'locale' command:

LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

[Stefan]

Sorry, I think the documentation is a bit unclear at this point. The config asks for a project id, not a project name. You can skip creating a project by simply typing in anything.

Have you tried setting LANGUAGE and LC_ALL in the docker to en_US.UTF-8?

[Ma Wo]

Okay, it seems like I fixed this problem.

First of all, the output from my earlier message was from the system environment.

The Locale inside the docker container was set to 'POSIX', so I assume that the Locale setting for the image was not set properly.

I just added ENV LC_ALL=C.UTF-8 to the Dockerfile and did a local build instead of pulling the fegyi001/force image. This successfully set the Locale to C.UTF-8.

Unfortunately, I immediately stumbled into the next error after it downloaded the first tile: 

Downloading S2A_MSIL1C_20190626T102031_N0207_R065_T32UPC_20190626T122046(1 of 21)...

tput: No value for $TERM and no -T specified

|         |   0 %                                                                   lockfile creation failed: cannot create temporary lockfile

I'm pretty sure that's an unrelated error. Any idea about this one?

[Stefan]

Did you check whether you can properly write to the output directory?

[Ma Wo]

Yes, that was my first thought as well.

However, it was just a simple copy-paste mistake, where I forgot to fix the path to the queue textfile to be created :)

Seems like queue file and temporary lockfile are related? If the directory doesn't exist, it can't create those files... makes sense.

Thanks for your help!

[Stefan]

The lockfile makes sure that the queue is not touched by any other process while FORCE writes to it.

I thought I had implemented a check to see if the directories of all outputs exist and are writeable, but apparently I missed something. Thanks for bringing this to my attention!