Docker issues

[Jens Pettersson]

Hello.

We're currently trying to use RavenDB in a docker setup (with Docker for Windows) only for development on our local dev machines. However, we're running into a couple of issues with the raven db container... A brand new container, built on ravendb/ravendb-nightly (20180116-0500 as this is written), with only one super simple database with just a couple of documents in it. No applications using it, just the Raven Studio... After a couple of minutes uptime (around 10 minutes) we start getting exceptions from raven studio:

Voron.Exceptions.VoronUnrecoverableErrorException: Error syncing the data file. The last sync tx is 203, but the journal's last tx id is 200, possible file corruption?

After this, the server get unresponsive and dies and we have to restart the container. After restart, everything works great again for a couple of minutes...

As I said, there's no real load at all, just my browser using Raven Studio... The docker container has got 3 CPUs and 4gb+ memory. Running with community license.

At the time when the errors started showing in the Raven Studio UI the total memory usage was around 750mb and raven used less than 200mb. No CPU load as well.

We're using docker-compose and volume mounts for our data folders. This is our docker-compose file as well as the custom settings.json:

version: '3.1'

services:

  ravendocker:

    image: ravendb/ravendb-nightly

    environment:

      - RAVEN_ARGS='--config-path=/opt/RavenDB/config/settings.json'

    ports:

        - 4040:8080

        - 38888:38888

    volumes:

        - c:/docker/raven/config:/opt/RavenDB/config

        - c:/docker/raven/data:/var/lib/ravendb/

{

  "ServerUrl": "http://ravendocker:8080",

  "Setup.Mode": "None",

  "Security.UnsecuredAccessAllowed": "PrivateNetwork",

  "License.Eula.Accepted": true,

  "DataDir": "/var/lib/ravendb/"

}

Any thoughts?

//J

[Oren Eini (Ayende Rahien)]

Can you check the night from today? Yesterday's has a known issue (but note related to this, though).

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jens Pettersson]

Thanks for the quick response. Got the latest nightly (20180117-0500) and restarted and now it's been running for 20 minutes without any issues. Will keep watching it for the rest of the day and see if it behaves.

//J

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Jens Pettersson]

Spoke too soon... Started a client using this instance and it immediately threw the same exception again and now it's unresponsive again.. Well, the Raven Server is "responding" but nothing is shown except for the error notification...

Error syncing the data file. The last sync tx is 234, but the journal's last tx id is 225, possible file corruption?

//J

Den onsdag 17 januari 2018 kl. 13:47:41 UTC+1 skrev Oren Eini:

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Jens Pettersson]

Testing with 4.0.0-rc40025 but I get the same error there as well... Works for a couple (5 - 20ish) minutes, then it throws...

[Oren Eini (Ayende Rahien)]

the 400025 is a few weeks old.

Can you try this with 20180117-0500 ?

Also, how are you running this? On what file system?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--

You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Jens Pettersson]

Already tried that, see a couple of messages up... It first looked like it worked (20+ minutes!) but then the same issues occured. That's why I downgraded to RC-40025 to test that one as well...

The database that I was using was migrated from another RavenDB server (one of our test machines, NON docker) via the Import Data > From another RavenDB Server (great feature btw!)... 

I'm now testing to run a new docker container, with volume mounts, but with only a newly created database with only two manually created docs in it... I'll get back to you.

//J

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.

[Jens Pettersson]

I'm now testing to run a new docker container, with volume mounts, but with only a newly created database with only two manually created docs in it... I'll get back to you.

Unfortunately, the same thing happend again with no migrated databases at all... Next up: Trying without any volume mounts.

[Oren Eini (Ayende Rahien)]

Can you send the docker instance here?

As well as the reprint?

To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+unsubscribe@googlegroups.com.

[Jens Pettersson]

By sending the docker instance, you mean a docker export <container_id> when it starts to act up?

And what do you mean by "the reprint"?

[Jens Pettersson]

FYI. It seems to be related to the volume mounts we're doing... I've been running without any volumes now and it's been running smoothly, and under quite some load, for an hour now. Can you see anything strange with our docker-compose file regarding the mounts?

version: '3.1'

services:

 ravendocker:

   image: ravendb/ravendb-nightly

   environment:

     - RAVEN_ARGS='--config-path=/opt/RavenDB/config/settings.json'

   ports:

       - 4040:8080

       - 38888:38888

   volumes:

       - c:/docker/raven/config:/opt/RavenDB/config

       - c:/docker/raven/data:/var/lib/ravendb/

Got to shut down for the day so I won't be able to test more today.

//J

[Grzegorz Lachowski]

Hi Jens,

Can you tell us the docker version you use and whether or not you use LCoW (Linux Containers on Windows)?

Thanks,

Greg

[Jens Pettersson]

Docker version 17.12.0-ce, build c97c6d6 (from the latest stable channel: https://docs.docker.com/docker-for-windows/install/)

And yes, we're using LCoW - Running on Windows 10 Enterprise, 64 bit.

As a side note, we're using RabbitMQ containers in the same way with volume mounts to both log and data folders and that seems to work (or at least we don't see any errors).

[Grzegorz Lachowski]

Thanks Jens. We've tried LCoW when it shown up on the Docker for Win Edge version and had some trouble running with it. There's an issue yet to be investigated on our bug tracker: http://issues.hibernatingrhinos.com/issue/RavenDB-9562

I wasn't aware it got to Docker for Win Stable channel already. We'll look into the issue.

Thanks,

Greg

[Jens Pettersson]

Thanks. We'll get around this for now as we only wanted to use this in a development environment with minimal impact on each developers machine (i.e. "run this docker-compose to get EVERYTHING up and running required to develop"). When we deploy, the RavenDB instance will probably not be hosted in docker but on a dedicated server on prem.

Keep us posted here if you find anything. Thanks!

//J