gsutil docker

73 views
Skip to first unread message

Ma Wo

unread,
Dec 7, 2020, 9:04:41 AM12/7/20
to FORCE
Hello everyone!

I'm new to FORCE and to Docker... so bear with me!
While the download of Landsat data works without any problems, I'm encountering some problems when I want to download Sentinel-2 data.

I followed the instructions to configure gsutil and have a .boto file, but I'm not 100% sure how to use the command that is described here as a docker work around. Am I right to assume, that I'll just have to integrate that part into my command like so:

sudo docker run -it -v /home/user/:/credentials --env BOTO_CONFIG=/credentials/.boto -v /media/user/WD500/force_data/:/data fegyi001/force force-level1-csd -c 0,80 -d 20190601,20190630 -s S2A,S2B /data/meta_catalogues /data/level-1-pool queue /data/shape_4326.gpkg

Unfortunately I always end up with the following error:

Downloading S2B_MSIL1C_20190601T102029_N0207_R065_T32UPC_20190601T123553(1 of 80)...
tput: No value for $TERM and no -T specified
|         |   0 %                                                                   Traceback (most recent call last):
  File "/usr/local/bin/gsutil", line 7, in <module>
    from gslib.__main__ import main
  File "/usr/local/lib/python3.6/dist-packages/gslib/__main__.py", line 53, in <module>
    import boto
  File "/usr/local/lib/python3.6/dist-packages/gslib/vendored/boto/boto/__init__.py", line 53, in <module>
    config = Config()
  File "/usr/local/lib/python3.6/dist-packages/gslib/vendored/boto/boto/pyami/config.py", line 63, in __init__
    self.read(BotoConfigLocations)
  File "/usr/lib/python3.6/configparser.py", line 697, in read
    self._read(fp, filename)
  File "/usr/lib/python3.6/configparser.py", line 1015, in _read
    for lineno, line in enumerate(fp, start=1):
  File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 3148: ordinal not in range(128)

If that isn't the right way, I'd happy if someone could enlighten me :)

Cheers and regards,
Marco

Stefan

unread,
Dec 8, 2020, 6:16:32 AM12/8/20
to FORCE
Hello Marco,

if your boto file was created properly and everything else is set up correctly...this may be a problem with locales.
What locale is the docker environment and the environment where you created the boto file? Did you use any special characters (i.e., Umlaut)?


Stefan

Stefan

unread,
Dec 8, 2020, 6:18:51 AM12/8/20
to FORCE
I'm assuming the 'user' in ' /home/user/:/credentials ' was just replaced in your post to not reveal your real username? Otherwise the error may be there.

Ma Wo

unread,
Dec 8, 2020, 10:19:54 AM12/8/20
to FORCE
Hello Stefan,

yes you're right about the 'user' part and I'm pretty sure that my .boto file is okay and free of any special characters or Umlaute. I mean, the gsutil config process is pretty straightforward or am I missing something?
However, I just repeated the process to be sure that I did everything correctly and noticed that the [docs](https://force-eo.readthedocs.io/en/latest/howto/level1-csd.html#gsutil-configuration) describe that a project name is not required and can be skipped by pressing enter. I'm not asked for a name, but a project-id. This can't be skipped, so I created a new project to provide an id. Is it possible that the config process changed or something like that?

My docker environment and the environment in which I created the .boto file are the same. Output of the 'locale' command:

LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="en_US.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="en_US.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

Stefan

unread,
Dec 8, 2020, 11:03:07 AM12/8/20
to FORCE
Sorry, I think the documentation is a bit unclear at this point. The config asks for a project id, not a project name. You can skip creating a project by simply typing in anything.

Have you tried setting LANGUAGE and LC_ALL in the docker to en_US.UTF-8?

Ma Wo

unread,
Dec 9, 2020, 11:23:55 AM12/9/20
to FORCE
Okay, it seems like I fixed this problem.

First of all, the output from my earlier message was from the system environment.
The Locale inside the docker container was set to 'POSIX', so I assume that the Locale setting for the image was not set properly.

I just added ENV LC_ALL=C.UTF-8 to the Dockerfile and did a local build instead of pulling the fegyi001/force image. This successfully set the Locale to C.UTF-8.

Unfortunately, I immediately stumbled into the next error after it downloaded the first tile: 

Downloading S2A_MSIL1C_20190626T102031_N0207_R065_T32UPC_20190626T122046(1 of 21)...

tput: No value for $TERM and no -T specified
|         |   0 %                                                                   lockfile creation failed: cannot create temporary lockfile


I'm pretty sure that's an unrelated error. Any idea about this one?

Stefan

unread,
Dec 10, 2020, 5:42:58 AM12/10/20
to FORCE
Did you check whether you can properly write to the output directory?

Ma Wo

unread,
Dec 10, 2020, 9:45:15 AM12/10/20
to FORCE
Yes, that was my first thought as well.
However, it was just a simple copy-paste mistake, where I forgot to fix the path to the queue textfile to be created :)
Seems like queue file and temporary lockfile are related? If the directory doesn't exist, it can't create those files... makes sense.

Thanks for your help!

Stefan

unread,
Dec 10, 2020, 2:51:34 PM12/10/20
to FORCE
The lockfile makes sure that the queue is not touched by any other process while FORCE writes to it.
I thought I had implemented a check to see if the directories of all outputs exist and are writeable, but apparently I missed something. Thanks for bringing this to my attention!
Reply all
Reply to author
Forward
0 new messages