How to host a secured raven instance in Kubernetes

886 views
Skip to first unread message

Morten Christiansen

unread,
Jun 4, 2018, 7:11:55 AM6/4/18
to RavenDB - 2nd generation document database
Hi,

I'm trying to see if I can get a RavenDB 4 cluster up and running inside a Kubernetes cluster in Azure. So far I have managed to run a raven instance unsecured, but when I supply an admin certificate, the server fails to load with the following message:

Unhandled Exception: System.InvalidOperationException: When the server certificate in either `Security.Certificate.Path` or `Security.Certificate.Exec` is specified, the `ServerUrl` must be using https, but was http://raven-1-598c698bb4-mmjrt:8080

I have configured a service with a single pod containing the database as per this yaml:

apiVersion: v1
kind: Service
metadata:
name: raven-1
namespace: default
labels:
project: campaign-manager
environment: staging
spec:
selector:
name: raven-1
type: ClusterIP
ports:
- name: https
protocol: TCP
port: 443
targetPort: 443
- name: tcp
protocol: TCP
port: 38888
targetPort: 38888
selector:
environment: staging
project: campaign-manager
name: raven-1
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: raven-1
namespace: default
spec:
replicas: 1
minReadySeconds: 5
revisionHistoryLimit: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
template:
metadata:
labels:
project: campaign-manager
environment: staging
name: raven-1
spec:
containers:
- name: ravendb
image: ravendb/ravendb:4.0.4-patch-40038-ubuntu.16.04-x64
env:
- name: RAVEN_ARGS
value: '--Setup.Mode=None'
- name: RAVEN_License_Eula_Accepted
value: 'true'
- name: RAVEN_Security_Certificate_Path
value: '/config/certs/ravendb-admin.pfx'
resources:
limits:
cpu: 1
memory: 512Mi
requests:
cpu: 250m
memory: 512Mi
ports:
- containerPort: 443
volumeMounts:
- name: certs
mountPath: "/config/certs"
readOnly: true
volumes:
- name: certs
secret:
secretName: db-admin-cert
imagePullSecrets:
- name: regsecret

Can anyone tell me how to satisfy the pod with an HTTPS url? I'm new to Kubernetes so there might be a simple and obvious solution to this but it just escapes me. I have an Ingress controller in front of the raven service which currently terminates the TLS connection and I suppose I will need to find a way for it to pass the HTTPS connection on to raven without terminating it, but first I need to get the raven server to start up without errors.

Regards,
Morten Christiansen

Oren Eini (Ayende Rahien)

unread,
Jun 4, 2018, 7:39:39 AM6/4/18
to ravendb
You cannot provide an admin cert without also setting up security.
You _have_ to setup security properly with a cert for this to work.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 


--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Morten Christiansen

unread,
Jun 4, 2018, 7:56:29 AM6/4/18
to RavenDB - 2nd generation document database
Yes, and what I'm looking for is help on how to do that in Kubernetes.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

Oren Eini (Ayende Rahien)

unread,
Jun 4, 2018, 8:00:13 AM6/4/18
to ravendb
The easiest way to do that is to handle the setup on a side machine, then take the settings.json and the certificate and point the instance in K8s to them.
There is the "-c" flag that tell us where to look for the config file.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Morten Christiansen

unread,
Jun 4, 2018, 8:30:04 AM6/4/18
to RavenDB - 2nd generation document database
Currently, my problem is that I don't know how to give the pod containing the raven server an https hostname. In other words, how do I change http://raven-1-598c698bb4-mmjrt:8080 to something with HTTPS as raven expects. This is more a Kubernetes issue, so I accept if this is the wrong place to ask - though someone here might have solved the same issue.

Grzegorz Lachowski

unread,
Jun 6, 2018, 3:39:09 AM6/6/18
to RavenDB - 2nd generation document database
Hi Morten,

I think we can fix that - here's a related ticket http://issues.hibernatingrhinos.com/issue/RavenDB-11302

For the time being to override the server url you can use RAVEN_ARGS env variable (that's concatenated to command line params when server is started) like so:
RAVEN_ARGS="–ServerUrl https://..."

since you don't know your hostname before it starts I would use RAVEN_ARGS="–ServerUrl https://0.0.0.0:8080" to bind to ALL interfaces in the container.

Thanks,
Greg

Morten Christiansen

unread,
Jun 6, 2018, 5:22:51 AM6/6/18
to RavenDB - 2nd generation document database
Hi Greg,

That explains why it did not have any effect when I tried setting the RAVEN_ServerUrl variable. I sidestepped the issue by adding authorization to ingress instead of securing the raven database but I will give it another go when I get the time.

Thanks,
Morten

Andreas Kappel

unread,
Jul 19, 2018, 3:53:15 AM7/19/18
to RavenDB - 2nd generation document database
Hi Christiansen,

have you achieved something? I am facing the exact same problem but I haven´t figured out how to configure my ingress properly. Maybe you can share your config.

Thanks
Andreas

Oren Eini (Ayende Rahien)

unread,
Jul 19, 2018, 1:56:40 PM7/19/18
to ravendb
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Andreas Kappel

unread,
Aug 13, 2018, 10:29:34 AM8/13/18
to RavenDB - 2nd generation document database
@Oren: This url isn´t available anymore

Andreas Kappel

unread,
Aug 14, 2018, 2:33:51 AM8/14/18
to RavenDB - 2nd generation document database
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    name: ravendb
  name: ravendb
  namespace: ravendb
spec:
  replicas: 1
  selector:
    matchLabels:
      name: ravendb
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: ravendb
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/os
                operator: In
                values:
                - linux
      containers:
      - env:
        - name: RAVEN_Security_Certificate_Path
          value: /opt/RavenDB/cert/ravendb.pfx
        - name: RAVEN_ARGS
          value: --ServerUrl=https://0.0.0.0:8080
        - name: RAVEN_PublicServerUrl
          value: https://ravendb.mycompany.com
        - name: RAVEN_Security_Certificate_Password
          value: certpass
        image: ravendb/ravendb:4.0.6-patch-40047-ubuntu.16.04-x64
        imagePullPolicy: IfNotPresent
        name: ravendb
        ports:
        - containerPort: 8080
          name: 8080
          protocol: TCP
        resources:
          limits:
            cpu: 1
            memory: 512Mi
          requests:
            cpu: 250m
            memory: 512Mi
        volumeMounts:
        - mountPath: /opt/RavenDB/cert
          name: ravendb
          subPath: cert
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      volumes:
      - name: ravendb
        nfs:
          path: /path/to/ravendb/
          server: nfsserver
---
apiVersion: v1
kind: Service
metadata:
  annotations:
  name: ravendb
  namespace: ravendb
spec:
  ports:
  - nodePort: 30739
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    name: ravendb
  sessionAffinity: None
  type: NodePort
---
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Ingress
  metadata:
    annotations:
    name: ravendb
    namespace: ravendb
  spec:
    rules:
    - host: ravendb.mycompany.com
      http:
        paths:
        - backend:
            serviceName: ravendb
            servicePort: 8080
    tls:
    - hosts:
      secretName: certificate

Thats my configurationfile I am using at the moment. 
When I use HTTPS Internally, I get a connection timeout
When I use HTTP to the pod, and HTTPS on the Ingress I get "Running in a potentially unsafe mode"

Oren Eini (Ayende Rahien)

unread,
Aug 14, 2018, 3:34:50 AM8/14/18
to ravendb
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Oren Eini (Ayende Rahien)

unread,
Aug 14, 2018, 3:36:17 AM8/14/18
to ravendb
You don't map the 443 port to 8080 ?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Andreas Kappel

unread,
Aug 14, 2018, 4:59:27 AM8/14/18
to RavenDB - 2nd generation document database
Accordinng to the documentation, if I specify a port for ServerUrl, it´ll be exposed on that port.
I also tried to set the RAVEN_Security_UnsecuredAccessAllowed Variable to PublicNetwork but nothing changes.

Output from the container
       _____                       _____  ____
      |  __ \                     |  __ \|  _ \
      | |__) |__ ___   _____ _ __ | |  | | |_) |
      |  _  // _` \ \ / / _ \ '_ \| |  | |  _ <
      | | \ \ (_| |\ V /  __/ | | | |__| | |_) |
      |_|  \_\__,_| \_/ \___|_| |_|_____/|____/
      Safe by default, optimized for efficiency
 Build 40047, Version 4.0, SemVer 4.0.6-patch-40047, Commit 75ef43a
 PID 9, 64 bits, 1 Cores, Phys Mem 512 MBytes, Arch: X64
 Source Code (git repo): https://github.com/ravendb/ravendb
 Built with love by Hibernating Rhinos and awesome contributors!
+---------------------------------------------------------------+
Warning: Admin Channel is not available:System.InvalidOperationException: Unable to set the proper path for the admin pipe, admin channel will not be available
   at Raven.Server.Utils.Pipes.WorkaroundPipePathForPosix(NamedPipeServerStream pipe, String pipeName) in C:\Builds\RavenDB-4.0-Patch\src\Raven.Server\Utils\Pipes.cs:line 211
   at Raven.Server.Utils.Pipes.OpenLogStreamPipe() in C:\Builds\RavenDB-4.0-Patch\src\Raven.Server\Utils\Pipes.cs:line 161
   at Raven.Server.RavenServer.OpenPipes() in C:\Builds\RavenDB-4.0-Patch\src\Raven.Server\RavenServer.cs:line 1580
   at Raven.Server.Program.Main(String[] args) in C:\Builds\RavenDB-4.0-Patch\src\Raven.Server\Program.cs:line 137
Using GC in server concurrent mode retaining memory from the OS.
Server available on: https://ravendb.mycompany.com
Tcp listening on 0.0.0.0:38888
Server started, listening to requests...
TIP: type 'help' to list the available commands.
Running non-interactive.

content of logfile inside container
Time,   Thread, Level,  Source, Logger, Message,        Exception
2018-08-14T08:56:01.6564396Z, 1, Operations, Server, Raven.Server.Program,  Build 40047, Version 4.0, SemVer 4.0.6-patch-40047, Commit 75ef43a
 PID 9, 64 bits, 1 Cores, Arch: X64
 512 MBytes Physical Memory, 499.74 MBytes Available Memory
 Using GC in server concurrent mode retaining memory from the OS.
2018-08-14T08:56:02.6900206Z, 1, Operations, Server, Sparrow.Platform.Posix.KernelVirtualFileSystemUtils, no swap defined on this system according to /proc/swaps
2018-08-14T08:56:04.6581167Z, 1, Operations, Server, Raven.Server.RavenServer, During server initialization, validating that the server can authenticate with itself using https://ravendb.mycompany.com.

Oren Eini (Ayende Rahien)

unread,
Aug 14, 2018, 5:02:15 AM8/14/18
to ravendb
You are using the public server url without a port, which means it will use port 443
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Andreas Kappel

unread,
Aug 14, 2018, 5:49:27 AM8/14/18
to RavenDB - 2nd generation document database
I have changed now the ServerUrl to "https://0.0.0.0:443", PublicServerUrl to "https://ravendb.mycompany.com:443" and adapted the NodePort and Ingress accordingly, but still I the connection times out when I access it via the browser. If I connect to the NodePort "<nodeip>:30327" I get the message "Attempted to access an HTTPS server using HTTP, did you forget to change 'http://' to 'https://' ?"

Oren Eini (Ayende Rahien)

unread,
Aug 14, 2018, 6:12:36 AM8/14/18
to ravendb
What are the ports you have open?
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Andreas Kappel

unread,
Aug 14, 2018, 6:26:47 AM8/14/18
to RavenDB - 2nd generation document database
Just 443

Fitzchak Yitzchaki

unread,
Aug 14, 2018, 7:16:15 AM8/14/18
to <ravendb@googlegroups.com>
Hi Andreas,

Have you tried to run the RavenDB service in unsecure mode on port 8080 and expose it with certificate using the ingress which would map from https to service on port 8080?

Best Regards,

Hibernating Rhinos Ltd  cid:image001.png@01CF95E2.8ED1B7D0

Fitzchak Yitzchaki l Senior Software Engineer l Mobile:+972-58-345-9538

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Andreas Kappel

unread,
Aug 14, 2018, 7:29:04 AM8/14/18
to RavenDB - 2nd generation document database
RAVEN_ARGS  => --ServerUrl=http://0.0.0.0:8080
RAVEN_PublicServerUrl => http://ravendb.mycompany.com:8080
RAVEN_Security_UnsecuredAccessAllowed => PrivateNetwork

SSL is handled from the ingress and ingress is pointing to 8080 from the pod
When I access ravendb via https://ravendb.mycompany.com I get this warning:

Untitled.png

Oren Eini (Ayende Rahien)

unread,
Aug 14, 2018, 7:44:47 AM8/14/18
to ravendb
RAVEN_Security_UnsecuredAccessAllowed => PublicNetwork


To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Fitzchak Yitzchaki

unread,
Aug 14, 2018, 7:51:18 AM8/14/18
to <ravendb@googlegroups.com>
Actually you should have also configure ravendb itself to run in secure mode and pass the certificate from the ingress by enable ssl passthrough.

The configuration should be:

Setup.Mode: None
DataDir: /data/RavenData
Security.Certificate.Path: /path/to/ssl/certificate (secret)
ServerUrl.Tcp: tcp://0.0.0.0:38888
PublicServerUrl.Tcp: tcp://tcp.ravendb.mycompany.com
License.Path: /path/to/license.json (secret)
License.Eula.Accepted: true


Best Regards,

Hibernating Rhinos Ltd  cid:image001.png@01CF95E2.8ED1B7D0

Fitzchak Yitzchaki l Senior Software Engineer l Mobile:+972-58-345-9538


To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Andreas Kappel

unread,
Aug 14, 2018, 10:42:05 AM8/14/18
to RavenDB - 2nd generation document database
I switched now from env vars to the json config file and that´s its content
{
  "Setup.Mode": "None",
  "DataDir": "/var/lib/ravendb",
  "Security.Certificate.Path": "/opt/RavenDB/cert/ravendb.pfx",
  "Security.Certificate.Password": "password",
  "ServerUrl": "https://0.0.0.0:8080",
  "ServerUrl.Tcp": "tcp://0.0.0.0:38888",
  "PublicServerUrl": "https://ravendb.mycompany.com",
  "PublicServerUrl.Tcp": "https://ravendb.mycompany.com:38888",
  "License.Eula.Accepted": "true",
  "Security.UnsecuredAccessAllowed": "PublicNetwork" (this setting doesn´t change anything)
}

the command openssl s_client -connect ravendb.mycompany.com:443 from inside the container returns my valid certificate

Andreas Kappel

unread,
Aug 14, 2018, 10:45:15 AM8/14/18
to RavenDB - 2nd generation document database
maybe the server is stuck during validation because the last line in the logfile always says
During server initialization, validating that the server can authenticate with itself using https://ravendb.mycompany.com.



Andreas
Currently, my problem is that I don't know how to give the pod containing the raven server an https hostname. In other words, how do I change <a href="http://www.google.com/url?q=http%3A%2F%2Fraven-1-598c698bb4-mmjrt%3A8080&sa=D&sntz=1&usg=AFQjCNH0bv64KmYvJSuPSwEG80xF-nmP4A" rel="nofollow" style="font-family:"Roboto Mono Regular",monospace;font-size:14px;letter-spacing:0.14px;white-space:pre-wrap" target="_blank" onmousedown="this.href='http://www.google.com/url?q\x3dhttp%3A%2F%2Fraven-1-598c698bb4-mmjrt%3A8080\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH0bv64KmYvJSuPSwEG8

Oren Eini (Ayende Rahien)

unread,
Aug 14, 2018, 10:46:19 AM8/14/18
to ravendb
If you try curl to that, what do you get?

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Oren Eini (Ayende Rahien)

unread,
Aug 14, 2018, 10:46:25 AM8/14/18
to or...@ravendb.net, ravendb
from inside the machine?

Andreas Kappel

unread,
Aug 20, 2018, 7:25:16 AM8/20/18
to RavenDB - 2nd generation document database
curl ravendb.mycompany.com

this command executed in the container just gets stuck
from inside the machine?
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

Oren Eini (Ayende Rahien)

unread,
Aug 20, 2018, 7:37:55 AM8/20/18
to ravendb
Okay, try openssl s_clent -connect  ravendb.mycompany.com:PORT



To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Andreas Kappel

unread,
Aug 20, 2018, 8:56:18 AM8/20/18
to RavenDB - 2nd generation document database
I did that already and it returns my valid certificate from digicert.com

Oren Eini (Ayende Rahien)

unread,
Aug 20, 2018, 9:40:51 AM8/20/18
to ravendb
Okay, and curl -v?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Andreas Kappel

unread,
Aug 20, 2018, 10:04:36 AM8/20/18
to RavenDB - 2nd generation document database
Hi,

thank you so much for that hint. There was a prolem with the TLS termination. I fixed that by adding the output of this command to /etc/ssl/certs/ca-certificates.crt
echo -n | openssl s_client -showcerts -connect ravendb.mycompany.com:443 \
  2>/dev/null  | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'

Now, everything works as expected. If anybody is interested, I can share my kubernetes yaml files.
Okay, and curl -v?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more op

Oren Eini (Ayende Rahien)

unread,
Aug 20, 2018, 10:07:48 AM8/20/18
to ravendb
That would be great, yes.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Andreas Kappel

unread,
Aug 20, 2018, 12:15:17 PM8/20/18
to RavenDB - 2nd generation document database
something is still not correct. When i create a new database via the Web Interface, I get this exception 

Raven.Client.Exceptions.Database.DatabaseLoadFailureException: Failed to start database testmctestface
At /var/lib/ravendb/Databases/testmctestface ---> System.InvalidOperationException: Cannot open database because RavenDB was unable create file lock on: /var/lib/ravendb/Databases/testmctestface/db.lock ---> System.IO.IOException: The process cannot access the file '/var/lib/ravendb/Databases/testmctestface/db.lock' because it is being used by another process.
   at System.IO.FileStream.CheckFileCall(Int64 result, Boolean ignoreNotSupported)
   at System.IO.FileStream.Lock(Int64 position, Int64 length)
   at Raven.Server.Documents.DocumentDatabase.TryAcquireWriteLock() in C:\Builds\RavenDB-4.1-RC\src\Raven.Server\Documents\DocumentDatabase.cs:line 340
   --- End of inner exception stack trace ---
   at Raven.Server.Documents.DocumentDatabase.TryAcquireWriteLock() in C:\Builds\RavenDB-4.1-RC\src\Raven.Server\Documents\DocumentDatabase.cs:line 349
   at Raven.Server.Documents.DocumentDatabase..ctor(String name, RavenConfiguration configuration, ServerStore serverStore, Action`1 addToInitLog) in C:\Builds\RavenDB-4.1-RC\src\Raven.Server\Documents\DocumentDatabase.cs:line 105
   at Raven.Server.Documents.DatabasesLandlord.CreateDocumentsStorage(StringSegment databaseName, RavenConfiguration config) in C:\Builds\RavenDB-4.1-RC\src\Raven.Server\Documents\DatabasesLandlord.cs:line 603
   --- End of inner exception stack trace ---
   at Raven.Server.Documents.DatabasesLandlord.CreateDocumentsStorage(StringSegment databaseName, RavenConfiguration config) in C:\Builds\RavenDB-4.1-RC\src\Raven.Server\Documents\DatabasesLandlord.cs:line 622
   at Raven.Server.Documents.DatabasesLandlord.ActuallyCreateDatabase(StringSegment databaseName, RavenConfiguration config) in C:\Builds\RavenDB-4.1-RC\src\Raven.Server\Documents\DatabasesLandlord.cs:line 516
   at Raven.Server.Documents.DatabasesLandlord.<>c__DisplayClass33_0.<CreateDatabaseUnderResourceSemaphore>b__0() in C:\Builds\RavenDB-4.1-RC\src\Raven.Server\Documents\DatabasesLandlord.cs:line 486
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
   at Raven.Server.Routing.RouteInformation.UnlikelyWaitForDatabaseToLoad(RequestHandlerContext context, Task`1 database, DatabasesLandlord databasesLandlord, StringSegment databaseName) in C:\Builds\RavenDB-4.1-RC\src\Raven.Server\Routing\RouteInformation.cs:line 126
   at Raven.Server.Routing.RouteInformation.WaitForDb(Task databaseLoading) in C:\Builds\RavenDB-4.1-RC\src\Raven.Server\Routing\RouteInformation.cs:line 164
   at Raven.Server.Routing.RequestRouter.HandlePath(HttpContext context, String method, String path) in C:\Builds\RavenDB-4.1-RC\src\Raven.Server\Routing\RequestRouter.cs:line 73
   at Raven.Server.RavenServerStartup.RequestHandler(HttpContext context) in C:\Builds\RavenDB-4.1-RC\src\Raven.Server\RavenServerStartup.cs:line 162

Oren Eini (Ayende Rahien)

unread,
Aug 20, 2018, 2:38:38 PM8/20/18
to ravendb
Can you check if something else is locking this file ( there shouldn't be ).
If you try to create this again, does this happen again?

--

Andreas Kappel

unread,
Aug 20, 2018, 3:15:18 PM8/20/18
to RavenDB - 2nd generation document database
if I create this file manually e.g. echo "bla" > db.lock it gets deleted after a few seconds but I don´t know by whom

Oren Eini (Ayende Rahien)

unread,
Aug 20, 2018, 3:23:56 PM8/20/18
to ravendb
Ha? 
That is really strange, can you use strace to figure this out?

--

Andreas Kappel

unread,
Aug 21, 2018, 2:54:01 AM8/21/18
to RavenDB - 2nd generation document database
What arguments should I use? 
If I specify a non mounted folder as database dir, it is getting created succesfully - but it won´t survive a restart

These are my volumeMounts

...
volumeMounts:
       - mountPath: /opt/RavenDB/cert
         name: ravendb
         subPath: cert
       - mountPath: /opt/RavenDB/config
         name: ravendb
         subPath: config
       - mountPath: /opt/RavenDB/Server/Logs
         name: ravendb
         subPath: logs
       - mountPath: /etc/ssl/certs/ca-certificates.crt
         name: ravendb
         subPath: cert/ca-certificates.crt
       - mountPath: /var/lib/ravendb
         name: ravendb
         subPath: data
...   
volumes:
- name: ravendb
  nfs:
    path: /<path>/
    server: <nfsserver>


Okay, and curl -v?

from inside the machine?
</di

Oren Eini (Ayende Rahien)

unread,
Aug 21, 2018, 1:22:00 PM8/21/18
to ravendb
What host are you running this on?

Oren Eini
CEO   /   Hibernating Rhinos LTD
Skype:  ayenderahien
Support:  sup...@ravendb.net

--

Andreas Kappel

unread,
Aug 21, 2018, 2:51:00 PM8/21/18
to RavenDB - 2nd generation document database
the container host is running ubuntu 16.04

Oren Eini (Ayende Rahien)

unread,
Aug 22, 2018, 3:36:11 AM8/22/18
to ravendb
Can you send your K8s configuration and the steps to reproduce this?

--

Andreas Kappel

unread,
Aug 22, 2018, 8:44:05 AM8/22/18
to RavenDB - 2nd generation document database

K8s config:
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ravendb
  name: ravendb
  namespace: ravendb
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  selector:
    matchLabels:
      app: ravendb
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: 
      labels:
        app: ravendb
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/os
                operator: In
                values:
                - linux
      containers:
      - env:
        - name: RAVEN_ARGS
          value: --config-path=/opt/RavenDB/config/settings.json
        - name: http_proxy
          value: http://squid.mycompany.com:8080
        - name: https_proxy
          value: http://squid.mycompany.com:8080
        - name: no_proxy
          value: .mycompany.com
        imagePullPolicy: IfNotPresent
        name: ravendb
        resources:
          limits:
            cpu: 1
            memory: 512Mi
          requests:
            cpu: 250m
            memory: 512Mi
        securityContext:
          allowPrivilegeEscalation: true
          privileged: true
        volumeMounts:
        - mountPath: /opt/RavenDB/cert
          name: ravendb
          subPath: cert
        - mountPath: /opt/RavenDB/config
          name: ravendb
          subPath: config
        - mountPath: /opt/RavenDB/Server/Logs
          name: ravendb
          subPath: logs
        - mountPath: /etc/ssl/certs/ca-certificates.crt
          name: ravendb
          subPath: cert/ca-certificates.crt
        - mountPath: /mnt/ravendb
          name: ravendb
          subPath: data
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      volumes:
      - name: ravendb
        nfs:
          path: /path/misc/ravendb/ravendb/
          server: storage.mycompanycorp.lan
---
apiVersion: v1
kind: Service
metadata:
  annotations:
  name: svc-ravendb
  namespace: ravendb
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: ravendb
  type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
  name: ravendb
  namespace: ravendb
spec:
  rules:
    http:
      paths:
      - backend:
          serviceName: svc-ravendb
          servicePort: 8080
        path: /
  tls:
  - hosts:

settings.json:
{
    "Setup.Mode": "None",
"DataDir": "/mnt/ravendb",

Andreas Kappel

unread,
Aug 22, 2018, 8:48:53 AM8/22/18
to RavenDB - 2nd generation document database
steps to reproduce:

Not working:
  1. login
  2. New database
  3. Enter database name
  4. leave Path as it is (in my case /mnt/ravendb)
  5. Error: Can't fetch collection stats
working but without persistent data:
  1. same as above until step 4
  2. set custom path (e.g. /var/lib/ravendb/mydb)
  3. DB is online and usable

Adi Avivi

unread,
Aug 27, 2018, 4:32:22 AM8/27/18
to RavenDB - 2nd generation document database
Hi,
Can you share the following bash commands output:

mount
ls -la /var/lib/ravendb
ls -la /mnt/ravendb
groups
groups root
ls -la /var/lib/ravendb/Databases/testmctestface



. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Hibernating Rhinos Ltd                       cid:image001.png@01CF95E2.8ED1B7D0
Avivi Adi l Core Team
RavenDB paving the way to "Data Made Simple"   http://ravendb.net

--

Andreas Kappel

unread,
Aug 27, 2018, 7:06:46 AM8/27/18
to RavenDB - 2nd generation document database
root@ravendb-84cb78d757-7xcnc:/opt/RavenDB/Server# mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/XVDIN4YZXEKL6QEEWKRL6KM5XP:/var/lib/docker/overlay2/l/LYGPFSUWZZC4QUJFS2KGLFRSXC:/var/lib/docker/overlay2/l/5Z573AGQKC6QXXOLHNCDT5OPY2:/var/lib/docker/overlay2/l/QQ5UD2XBKTH6RB5ENMXKPO4RLB:/var/lib/docker/overlay2/l/IJ4WQ23G3XZZYPAFH2Z6AGZQA2:/var/lib/docker/overlay2/l/ZRALATDAWU2WYA5RQDLRJEBV4G:/var/lib/docker/overlay2/l/A4O5LBBGWDVKVSWLBCIFBXN4O5:/var/lib/docker/overlay2/l/3EZXZB5QGXILUPV7KOL477K56H:/var/lib/docker/overlay2/l/D3C3IWUG5D2FNOBS3WFMAF5HMM:/var/lib/docker/overlay2/l/LXT4JNZCYX5XZ2RJLPUMQIGPGE:/var/lib/docker/overlay2/l/IG5B2QWN467BJHXVDOE5VIFZWF,upperdir=/var/lib/docker/overlay2/1f1c8a2b44c97948debaa825ec202657eceedf421733e0aeed24fc3d8ce6324d/diff,workdir=/var/lib/docker/overlay2/1f1c8a2b44c97948debaa825ec202657eceedf421733e0aeed24fc3d8ce6324d/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
storage.mycompanycorp.lan:/path/misc/ravendb/ravendb//data on /mnt/ravendb type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.12.X.Y,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=10.12.X.Y)
/dev/mapper/vg01-root on /dev/termination-log type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/vg01-root on /etc/resolv.conf type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/vg01-root on /etc/hostname type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/mapper/vg01-root on /etc/hosts type ext4 (rw,relatime,errors=remount-ro,data=ordered)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
storage.mycompanycorp.lan:/path/misc/ravendb/ravendb//cert on /opt/RavenDB/cert type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.12.X.Y,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=10.12.X.Y)
storage.mycompanycorp.lan:/path/misc/ravendb/ravendb//config on /opt/RavenDB/config type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.12.X.Y,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=10.12.X.Y)
/dev/mapper/vg01-root on /opt/RavenDB/Server/RavenData type ext4 (rw,relatime,errors=remount-ro,data=ordered)
storage.mycompanycorp.lan:/path/misc/ravendb/ravendb//logs on /opt/RavenDB/Server/Logs type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.12.X.Y,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=10.12.X.Y)
storage.mycompanycorp.lan:/path/misc/ravendb/ravendb//cert/ca-certificates.crt on /etc/ssl/certs/ca-certificates.crt type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.12.X.Y,mountvers=3,mountport=635,mountproto=udp,local_lock=none,addr=10.12.X.Y)
tmpfs on /run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime)

##########################################################

root@ravendb9-6c6cd6696d-zlhp6:/# ls -la /mnt/ravendb/
total 20
drwxr-xr-x 5 root root 4096 Aug 22 10:15 .
drwxr-xr-x 1 root root 4096 Aug 21 06:35 ..
drwxr-xr-x 2 root root 4096 Aug 22 12:49 Databases
drwxr-xr-x 4 root root 4096 Aug 21 06:35 System
drwxr-xr-x 7 root root 4096 Aug 22 10:15 backup

##########################################################

root@ravendb9-6c6cd6696d-zlhp6:/# ls -la /var/lib/ravendb/
total 65572
drwxr-xr-x 6 root root     4096 Aug 22 13:06 .
drwxr-xr-x 1 root root     4096 Aug 21 07:24 ..
drwxr-xr-x 4 root root     4096 Aug 22 12:49 Configuration
drwxr-xr-x 4 root root     4096 Aug 21 09:56 Indexes
drwxr-xr-x 2 root root     4096 Aug 21 10:24 Journals
-rw------- 1 root root 67108864 Aug 22 12:49 Raven.voron
drwxr-xr-x 2 root root     4096 Aug 22 12:49 Temp
-rw------- 1 root root      162 Aug 22 10:05 headers.one
-rw------- 1 root root      162 Aug 22 12:49 headers.two

##########################################################

root@ravendb9-6c6cd6696d-zlhp6:/# groups
root

##########################################################

root@ravendb9-6c6cd6696d-zlhp6:/# groups root
root : root

##########################################################

root@ravendb9-6c6cd6696d-zlhp6:/# ls -la /var/lib/ravendb/Databases/testmctestface/
total 100
drwxr-xr-x 7 root root  4096 Aug 27 11:03 .
drwxr-xr-x 3 root root  4096 Aug 27 11:03 ..
drwxr-xr-x 4 root root  4096 Aug 27 11:03 Configuration
drwxr-xr-x 2 root root  4096 Aug 27 11:03 Indexes
drwxr-xr-x 2 root root  4096 Aug 27 11:03 Journals
drwxr-xr-x 2 root root  4096 Aug 27 11:03 PeriodicBackupTemp
-rw------- 1 root root 65536 Aug 27 11:03 Raven.voron
drwxr-xr-x 2 root root  4096 Aug 27 11:03 Temp
-rw-r--r-- 1 root root     1 Aug 27 11:03 db.lock
-rw------- 1 root root   162 Aug 27 11:03 headers.one
-rw------- 1 root root   162 Aug 27 11:03 headers.two
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

Adi Avivi

unread,
Aug 27, 2018, 7:33:57 AM8/27/18
to RavenDB - 2nd generation document database
Your /mnt/ravendb is a `nfs` filesystem.
I presume this is the cause of the failure (the 'created' file is not 'created' yet because of the way nfs works)

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Hibernating Rhinos Ltd                       cid:image001.png@01CF95E2.8ED1B7D0
Avivi Adi l Core Team
Office: +972-4-622-7811 l Fax: +972-153-4-622-7811
RavenDB paving the way to "Data Made Simple"   http://ravendb.net

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

Andreas Kappel

unread,
Aug 27, 2018, 7:56:07 AM8/27/18
to RavenDB - 2nd generation document database
I run the exact same setting for lots of our production services which also include other database systems like mongodb, mariadb and postgres.

What kind of setting would you recommend? I do not want to store the database on the host itself and our production environment is not ready for persistent volume claims yet
steps to reproduce:

Not working:
Okay, and curl -v?

from inside the machine?

Hibernating Rhinos Ltd  

Oren Eini l <font color

Adi Avivi

unread,
Aug 28, 2018, 10:39:11 AM8/28/18
to RavenDB - 2nd generation document database
There's no REAL recommendation when you rule out persistent volume, however, we can do something with this.. we can have few retries on the specific error, and hopefully it can overcome the situation.
I guess we might add configuration option for controlling how many retries / how much time to retry (with defaults 0).


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Hibernating Rhinos Ltd                       cid:image001.png@01CF95E2.8ED1B7D0
Avivi Adi l Core Team
Office: +972-4-622-7811 l Fax: +972-153-4-622-7811
RavenDB paving the way to "Data Made Simple"   http://ravendb.net

--

Jesús López

unread,
Mar 12, 2019, 11:26:34 AM3/12/19
to RavenDB - 2nd generation document database
A guide to deploy ravendb on Kubernetes would be great.

After reading this thread four times, I still don't know how to make this to work.

I spent an entire day to make a ravendb secured instance running correctly on my Docker Desktop and I'm afraid I would spend more than a week to make this work on Kubernetes. Yes, I'm a docker and kubernetes newbie, but I need to learn.

Thank you

El martes, 14 de agosto de 2018, 9:34:50 (UTC+2), Oren Eini escribió:

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 


On Mon, Aug 13, 2018 at 5:29 PM, Andreas Kappel <and.k...@gmail.com> wrote:
@Oren: This url isn´t available anymore
Am Donnerstag, 19. Juli 2018 19:56:40 UTC+2 schrieb Oren Eini:

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 


On Wed, Jul 18, 2018 at 5:23 PM, Andreas Kappel <and.k...@gmail.com> wrote:
Hi Christiansen,

have you achieved something? I am facing the exact same problem but I haven´t figured out how to configure my ingress properly. Maybe you can share your config.

Thanks
Andreas


Am Mittwoch, 6. Juni 2018 11:22:51 UTC+2 schrieb Morten Christiansen:
Hi Greg,

That explains why it did not have any effect when I tried setting the RAVEN_ServerUrl variable. I sidestepped the issue by adding authorization to ingress instead of securing the raven database but I will give it another go when I get the time.

Thanks,
Morten


On Wednesday, June 6, 2018 at 9:39:09 AM UTC+2, Grzegorz Lachowski wrote:
Hi Morten,

I think we can fix that - here's a related ticket http://issues.hibernatingrhinos.com/issue/RavenDB-11302

For the time being to override the server url you can use RAVEN_ARGS env variable (that's concatenated to command line params when server is started) like so:
RAVEN_ARGS="–ServerUrl https://..."

since you don't know your hostname before it starts I would use RAVEN_ARGS="–ServerUrl https://0.0.0.0:8080" to bind to ALL interfaces in the container.

Thanks,
Greg

On Monday, June 4, 2018 at 2:30:04 PM UTC+2, Morten Christiansen wrote:
Currently, my problem is that I don't know how to give the pod containing the raven server an https hostname. In other words, how do I change http://raven-1-598c698bb4-mmjrt:8080 to something with HTTPS as raven expects. This is more a Kubernetes issue, so I accept if this is the wrong place to ask - though someone here might have solved the same issue.


On Monday, June 4, 2018 at 2:00:13 PM UTC+2, Oren Eini wrote:
The easiest way to do that is to handle the setup on a side machine, then take the settings.json and the certificate and point the instance in K8s to them.
There is the "-c" flag that tell us where to look for the config file.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 


On Mon, Jun 4, 2018 at 2:56 PM, Morten Christiansen <withoutp...@gmail.com> wrote:
Yes, and what I'm looking for is help on how to do that in Kubernetes.


On Monday, June 4, 2018 at 1:39:39 PM UTC+2, Oren Eini wrote:
You cannot provide an admin cert without also setting up security.
You _have_ to setup security properly with a cert for this to work.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 


On Mon, Jun 4, 2018 at 2:11 PM, Morten Christiansen <withoutp...@gmail.com> wrote:
Hi,

I'm trying to see if I can get a RavenDB 4 cluster up and running inside a Kubernetes cluster in Azure. So far I have managed to run a raven instance unsecured, but when I supply an admin certificate, the server fails to load with the following message:

Unhandled Exception: System.InvalidOperationException: When the server certificate in either `Security.Certificate.Path` or `Security.Certificate.Exec` is specified, the `ServerUrl` must be using https, but was http://raven-1-598c698bb4-mmjrt:8080

I have configured a service with a single pod containing the database as per this yaml:

apiVersion: v1
kind: Service
metadata:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
template:
metadata:
labels:
project: campaign-manager
environment: staging
name: raven-1
spec:
containers:
- name: ravendb
image: ravendb/ravendb:4.0.4-patch-40038-ubuntu.16.04-x64
env:
- name: RAVEN_ARGS
value: '--Setup.Mode=None'
- name: RAVEN_License_Eula_Accepted
value: 'true'
- name: RAVEN_Security_Certificate_Path
value: '/config/certs/ravendb-admin.pfx'
resources:
limits:
cpu: 1
memory: 512Mi
requests:
cpu: 250m
memory: 512Mi
ports:
- containerPort: 443
volumeMounts:
- name: certs
mountPath: "/config/certs"
readOnly: true
volumes:
- name: certs
secret:
secretName: db-admin-cert
imagePullSecrets:
- name: regsecret

Can anyone tell me how to satisfy the pod with an HTTPS url? I'm new to Kubernetes so there might be a simple and obvious solution to this but it just escapes me. I have an Ingress controller in front of the raven service which currently terminates the TLS connection and I suppose I will need to find a way for it to pass the HTTPS connection on to raven without terminating it, but first I need to get the raven server to start up without errors.

Regards,
Morten Christiansen

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

Oren Eini (Ayende Rahien)

unread,
Mar 12, 2019, 1:33:26 PM3/12/19
to ravendb
We are actively working on them.
These are still in draft mode, but you can still see how things are working, check:

Jesús López

unread,
Mar 21, 2019, 7:01:11 AM3/21/19
to RavenDB - 2nd generation document database
I followed the guide and got the cluster up and running without problems, thank you very much.

I just had to add 6 dns A records because my provider doesn't support star character in the host name.

One suggestion: use kubectl apply instead of kubectl create. It's very difficult to change the configuration after kubctl create if you didn't include --save-config flag. 
Reply all
Reply to author
Forward
0 new messages