K8S Simple configuraiton - unable to create database

[Patryk]

Hey, 

I have simple two-node setup with k8s. 
As data volume glusterfs is used and everything seems to work fine. RavenDb is creating all system files etc. Everything works unless new database is created [Exception BELOW].

My PVC is mounting volume properly (in ReadWriteOnly mode access) so no other pods / nodes can access volume:

 k describe pvc data-ravendb-0

Name:          data-ravendb-0

Namespace:     production

StorageClass:  gluster-heketi

Status:        Bound

Volume:        pvc-107f1c47-cd55-419f-afcc-c1ed9defd500

Labels:        app=ravendb

Annotations:   pv.kubernetes.io/bind-completed: yes

               pv.kubernetes.io/bound-by-controller: yes

               volume.beta.kubernetes.io/storage-class: gluster-heketi

               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/glusterfs

Finalizers:    [kubernetes.io/pvc-protection]

Capacity:      25Gi

Access Modes:  RWO

VolumeMode:    Filesystem

Used By:       ravendb-0

Events:

I've tried to run only one pod on the only node, but no luck - still the same. 

2021-02-21T22:01:03.5951780Z, 51, Information, Server,  EXCEPTION: Raven.Client.Exceptions.Database.DatabaseLoadFailureException: Failed to start database TestDb

At /data/RavenData/Databases/TestDb

 ---> System.InvalidOperationException: Cannot open database because RavenDB was unable create file lock on: '/data/RavenData/Databases/ TestDb  /db.lock'. File system type: fuse

 ---> System.IO.IOException: The process cannot access the file '/data/RavenData/Databases/ TestDb  /db.lock' because it is being used by another process.

   at System.IO.FileStream.CheckFileCall(Int64 result, Boolean ignoreNotSupported)

   at System.IO.FileStream.Lock(Int64 position, Int64 length)

   at Raven.Server.Utils.FileLocker.TryAcquireWriteLock(Logger logger) in C:\Builds\RavenDB-Stable-5.1\51016\src\Raven.Server\Utils\FileLocker.cs:line 81

   --- End of inner exception stack trace ---

   at Raven.Server.Utils.FileLocker.TryAcquireWriteLock(Logger logger) in C:\Builds\RavenDB-Stable-5.1\51016\src\Raven.Server\Utils\FileLocker.cs:line 81

   at Raven.Server.Documents.DocumentDatabase..ctor(String name, RavenConfiguration configuration, ServerStore serverStore, Action`1 addToInitLog) in C:\Builds\RavenDB-Stable-5.1\51016\src\Raven.Server\Documents\DocumentDatabase.cs:line 179

   at Raven.Server.Documents.DatabasesLandlord.CreateDocumentsStorage(StringSegment databaseName, RavenConfiguration config, Nullable`1 wakeup) in C:\Builds\RavenDB-Stable-5.1\51016\src\Raven.Server\Documents\DatabasesLandlord.cs:line 778

   --- End of inner exception stack trace ---

   at Raven.Server.Documents.DatabasesLandlord.CreateDocumentsStorage(StringSegment databaseName, RavenConfiguration config, Nullable`1 wakeup) in C:\Builds\RavenDB-Stable-5.1\51016\src\Raven.Server\Documents\DatabasesLandlord.cs:line 778

   at Raven.Server.Documents.DatabasesLandlord.ActuallyCreateDatabase(StringSegment databaseName, RavenConfiguration config, Nullable`1 wakeup) in C:\Builds\RavenDB-Stable-5.1\51016\src\Raven.Server\Documents\DatabasesLandlord.cs:line 685

   at System.Threading.Tasks.Task`1.InnerInvoke()

   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)

--- End of stack trace from previous location ---

   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)

--- End of stack trace from previous location ---

   at Raven.Server.Routing.RouteInformation.UnlikelyWaitForDatabaseToLoad(RequestHandlerContext context, Task`1 database, DatabasesLandlord databasesLandlord, StringSegment databaseName) in C:\Builds\RavenDB-Stable-5.1\51016\src\Raven.Server\Routing\RouteInformation.cs:line 179

   at Raven.Server.Routing.RouteInformation.WaitForDb(Task databaseLoading) in C:\Builds\RavenDB-Stable-5.1\51016\src\Raven.Server\Routing\RouteInformation.cs:line 218

   at Raven.Server.Routing.RequestRouter.HandlePath(RequestHandlerContext reqCtx) in C:\Builds\RavenDB-Stable-5.1\51016\src\Raven.Server\Routing\RequestRouter.cs:line 196

   at Raven.Server.RavenServerStartup.RequestHandler(HttpContext context) in C:\Builds\RavenDB-Stable-5.1\51016\src\Raven.Server\RavenServerStartup.cs:line 242

[Patryk]

Eh it seems as it is GlusterFS  and heketi issue - emptyDir volume works w/o problems...

[Oren Eini (Ayende Rahien)]

What version are you using? IIRC, this is something that we recently fixed

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/58312267-9b4c-42bb-8cc4-9779aea86e27n%40googlegroups.com.

[Patryk]

I am using the newest image from the DockerHub

ravendb/ravendb:5.1.4-ubuntu.20.04-x64

[Oren Eini]

Are you able to run strace on the database as it is creating the database? I would like to inspect what call is failing.

[Patryk]

I'll try to reproduce it on our dev cluster.
As for now, we have ended up with external SSD storage on each node - for performance reasons also (our vnet was not fast enough so writing / reading files from PersistentVolume based on glusterfas was pain in the ass). 

[Oren Eini (Ayende Rahien)]

Yes, that is recommended in general

To view this discussion on the web visit https://groups.google.com/d/msgid/ravendb/0a1b9cac-91d3-4ceb-a852-80e2d3fc5c89n%40googlegroups.com.