Problems running Scylla with volume on Windows 10

[benjamintestapp0 benjamintestapp0]

I already asked the question on stackoverflow, reddit and Slack, but no response. I hope this way will work.

I am having a problem running a Scylla instance in docker-compose. I am on windows 10 and this is the errors I am getting when running `$ docker logs b-scylla`:

    INFO  2021-03-03 20:28:33,030 [shard 0] storage_service - Stop transport: starts
    INFO  2021-03-03 20:28:33,030 [shard 0] storage_service - Stop transport: shutdown rpc and cql server done
    INFO  2021-03-03 20:28:33,030 [shard 0] gossip - gossip is already stopped
    INFO  2021-03-03 20:28:33,030 [shard 0] storage_service - Stop transport: stop_gossiping done
    ERROR 2021-03-03 20:28:33,030 [shard 0] init - Directory '"/var/lib/scylla/commitlog"' cannot be initialized. Tried to do it but failed with: storage_io_error (Storage I/O error: 22: Invalid argument)
    ERROR 2021-03-03 20:28:33,030 [shard 0] init - Directory '"/var/lib/scylla/data"' cannot be initialized. Tried to do it but failed with: storage_io_error (Storage I/O error: 22: Invalid argument)
    ERROR 2021-03-03 20:28:33,030 [shard 0] init - Directory '"/var/lib/scylla/hints/0"' cannot be initialized. Tried to do it but failed with: storage_io_error (Storage I/O error: 22: Invalid argument)
    ERROR 2021-03-03 20:28:33,030 [shard 0] init - Directory '"/var/lib/scylla/hints/1"' cannot be initialized. Tried to do it but failed with: storage_io_error (Storage I/O error: 22: Invalid argument)
    ERROR 2021-03-03 20:28:33,030 [shard 0] init - Directory '"/var/lib/scylla/hints/2"' cannot be initialized. Tried to do it but failed with: storage_io_error (Storage I/O error: 22: Invalid argument)
    ERROR 2021-03-03 20:28:33,030 [shard 0] init - Directory '"/var/lib/scylla/hints/3"' cannot be initialized. Tried to do it but failed with: storage_io_error (Storage I/O error: 22: Invalid argument)
    ERROR 2021-03-03 20:28:33,030 [shard 0] init - Directory '"/var/lib/scylla/view_hints/0"' cannot be initialized. Tried to do it but failed with: storage_io_error (Storage I/O error: 22: Invalid argument)
    ERROR 2021-03-03 20:28:33,031 [shard 0] init - Directory '"/var/lib/scylla/view_hints/1"' cannot be initialized. Tried to do it but failed with: storage_io_error (Storage I/O error: 22: Invalid argument)
    ERROR 2021-03-03 20:28:33,031 [shard 0] init - Directory '"/var/lib/scylla/view_hints/2"' cannot be initialized. Tried to do it but failed with: storage_io_error (Storage I/O error: 22: Invalid argument)
    ERROR 2021-03-03 20:28:33,031 [shard 0] init - Directory '"/var/lib/scylla/view_hints/3"' cannot be initialized. Tried to do it but failed with: storage_io_error (Storage I/O error: 22: Invalid argument)
    INFO  2021-03-03 20:28:33,031 [shard 0] init - Shutting down database
    INFO  2021-03-03 20:28:33,031 [shard 0] storage_service - messaging_service stopped
    INFO  2021-03-03 20:28:33,031 [shard 0] storage_service - Stop transport: shutdown messaging_service done
    INFO  2021-03-03 20:28:33,031 [shard 0] storage_service - stream_manager stopped
    INFO  2021-03-03 20:28:33,031 [shard 0] storage_service - Stop transport: shutdown stream_manager done
    INFO  2021-03-03 20:28:33,031 [shard 0] storage_service - Stop transport: done
    INFO  2021-03-03 20:28:33,032 [shard 0] init - Shutting down database was successful

Oddly enough, I see that the dirs are actually created on my local machine (I have a volume).


This is my docker-compose.yml:

    version: '3'
    
    services:
      b-scylla:
        image: scylladb/scylla:4.3.1
        container_name: b-scylla
        volumes:
          - ./scylla:/var/lib/scylla
          - ./:/project
        ports:
          - 127.0.0.1:9042:9042
          - 127.0.0.1:9160:9160

I replaced './scylla/' with an absolute path, but the result didn't change. I have this in my .env file, in the same folder as docker-compose.yml:

    COMPOSE_CONVERT_WINDOWS_PATHS=1

[benjamintestapp0 benjamintestapp0]

The reddit link should be this: https://www.reddit.com/r/docker/comments/lx3u3y/directory_can_not_be_initialized_storage_io_error/

Op donderdag 4 maart 2021 om 10:03:05 UTC+1 schreef benjamintestapp0 benjamintestapp0:

[Pekka Enberg]

Hi,

I unfortunately have no experience with Windows 10 and Docker, but it looks like some Docker setup issue. Let's CC Ivan and Lucas who I think know more.

- Pekka 

[benjamintestapp0 benjamintestapp0]

Thanks for the quick response! I don't have problems running Elasticsearch on my Windows with a persistent volume, so my guess is that there is something wrong with Scylla

Op donderdag 4 maart 2021 om 10:32:40 UTC+1 schreef pen...@scylladb.com:

[Avi Kivity]

Probably the underlying storage does not support the lockf [1] API correctly. Scylla creates a .lock file under each of these directories, and locks it to ensure no two parallel instances of Scylla are operating on the same data.

You can try running a stanrard Linux container with such a mounted volume, and call the fcntl.lockf() Python function [2] on a file in that volume, to see if I am correct.

[1] https://man7.org/linux/man-pages/man3/lockf.3.html

[2] https://docs.python.org/3/library/fcntl.html

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/6d39c377-a668-49f7-ab45-0128edc5ce53n%40googlegroups.com.

[Pekka Enberg]

Hi,

On Thu, Mar 4, 2021 at 11:49 AM Avi Kivity <a...@scylladb.com> wrote:

Probably the underlying storage does not support the lockf [1] API correctly. Scylla creates a .lock file under each of these directories, and locks it to ensure no two parallel instances of Scylla are operating on the same data.

You can try running a stanrard Linux container with such a mounted volume, and call the fcntl.lockf() Python function [2] on a file in that volume, to see if I am correct.

[1] https://man7.org/linux/man-pages/man3/lockf.3.html

[2] https://docs.python.org/3/library/fcntl.html

I think Ivan managed to get it running without bind mounts (how do those even work on Windows?):

https://stackoverflow.com/a/66472567/153224

- Pekka

[Ivan Prisyazhnyy]

> Probably the underlying storage does not support the lockf [1] API correctly. Scylla creates a .lock file under each of these directories, and locks it to ensure no two parallel instances of Scylla are operating on the same data.

Probably. Lock files however are present.

> (how do those even work on Windows?):

Linux containers on Windows are run by Hyper-V IIRC