Today, our OrientDB crashed without any errors.

[Erik Pragt]

Hi all,

As the question title suggests, our OrientDB database crashed today. We got notified by our monitoring server that our sites aren't accepting any requests anymore, which was quickly traced to OrientDB being unavailable, i.e., not running anymore.

However, when checking the logs (orientdb.err/orientdb.log), there was no indication at all of any crash. What can I do to investigate this issue, and, even better, prevent OrientDB from crashing again? This isn't the first time OrientDB crashes, but usually it's with an OutOfMemory exception, which we fixed(?) by giving it more memory. Currently, the DB hardly does anything (1 request per 10 seconds?), and we're a bit worried that once we do get some requests, that OrientDB might crash more often.

Our current settings to run OrientDB look like this:

ORIENTDB_SETTINGS="-Dprofiler.enabled=true"

JAVA_OPTS_SCRIPT="-Djna.nosys=true -XX:+HeapDumpOnOutOfMemoryError -Djava.awt.headless=true -Dfile.encoding=UTF8 -Drhino.opt.level=9"

# ORIENTDB MAXIMUM HEAP. USE SYNTAX -Xmx<memory>, WHERE <memory> HAS THE TOTAL MEMORY AND SIZE UNIT. EXAMPLE: -Xmx512m

MAXHEAP=-Xmx1024m

# ORIENTDB MAXIMUM DISKCACHE IN MB, EXAMPLE, ENTER -Dstorage.diskCache.bufferSize=8192 FOR 8GB

#MAXDISKCACHE=""

MAXDISKCACHE="-Dstorage.diskCache.bufferSize=8192"

If we need to provide some more information, please let me know.

Kind regards,

Erik Pragt

[Erik Pragt]

Btw, running OrientDB 2.1.9.

[Luigi Dell'Aquila]

Hi Erik,

It's hard to give you an answer here, this is the first report we have from a system that crashes without emitting any logs at all.

In other cases, when we had an unexpected crash with few information, it was because of some OS related procedures (eg. process kill under heavy load), but also in that case we had a log stating that the server was killed.

Could you please share your logs? Maybe we will find some hints in the history

Thanks

Luigi

--

---
You received this message because you are subscribed to the Google Groups "OrientDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luca Garulli]

Hi Erik,

If OrientDB dies with no log, it means the OS killed it. The most common reason is the out of memory.

Look at: http://stackoverflow.com/questions/726690/who-killed-my-process-and-why

A few questions for you:

• How much physical RAM do you have on that server and
• How much is available before you start OrientDB server?
  
• Did you configure any swap for your server?

Best Regards,

Luca Garulli

Founder & CEO

OrientDB

[Erik Pragt]

Hi Luca,

Thanks for your response.

- I'm running on Digital Ocean, 2 core 2 gig version. 

- Around 1500 mb.

- What kind of swap do you mean? On an OS level?

Thanks for the help!

Erik

[Erik Pragt]

Hi Luigi,

I can certainly share my logs. I'll send you an email. Thanks for the help!

Erik

[Luca Garulli]

Hi Erik,

Do you have 2GB of RAM on that server? if this is the case, you cannot have 8GB of RAM for the Diskcache:

-Dstorage.diskCache.bufferSize=8192

So this must be the reason. This is the formula:

diskcache + heap < available ram

So with 1,5GB of free RAM, try these settings:

MAXHEAP=-Xmx700m and -Dstorage.diskCache.bufferSize=512

Best Regards,

Luca Garulli

Founder & CEO

OrientDB

[Erik Pragt]

Hi Luca,

I'm sorry, that was an error on my part: I changed that after the crash, before the crash it was on a very low setting, around 400. I'll change it back anyway!

Erik

You received this message because you are subscribed to a topic in the Google Groups "OrientDB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/orient-database/Xith1NqSX0k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to orient-databa...@googlegroups.com.

[Andrey Lomakin]

Hi Erik,

If system error happens (that is the only reason which we can think why it was shutted down without any exceptions) your OS log should contain information about

that error.

Which OS do you use ?

Best regards,
Andrey Lomakin, R&D lead. 

OrientDB Ltd

twitter:@Andrey_Lomakin linkedin:https://ua.linkedin.com/in/andreylomakin

[Erik Pragt]

Thanks! I'm using Ubuntu, I think 14.x

[Andrey Lomakin]

Hi Erik,

Could you inspect following files /var/log/syslog and /var/log/kern.log using grep for example  grep -i orient /var/log/kern.log and send us results.

[Erik Pragt]

Thanks Andrey.

I checked both files, and there's nothing of internet. This is my complete kern.log file for example:

Apr  8 10:10:04 localhost kernel: [1282413.300235] audit_printk_skb: 24 callbacks suppressed

Apr  8 10:10:04 localhost kernel: [1282413.300241] type=1400 audit(1460103004.823:19): apparmor="STATUS" operation="profile_replace" parent=21949 profile="unconfined" name="/usr/sbin/mysqld" pid=21956 comm="apparmor_parser"

Apr  8 10:11:27 localhost kernel: [1282496.440652] type=1400 audit(1460103087.963:20): apparmor="STATUS" operation="profile_replace" parent=22456 profile="unconfined" name="/usr/sbin/mysqld" pid=22460 comm="apparmor_parser"

and my /var/log/syslog also doesn't contain anything exciting....

[scott molinari]

Is MySql also running on the same server? Just wondering. 

Scott

[Erik Pragt]

It used to run on the same server indeed, but it's not used, so I uninstalled it.

On Sun, Apr 10, 2016 at 3:21 PM, 'scott molinari' via OrientDB <orient-...@googlegroups.com> wrote:

Is MySql also running on the same server? Just wondering. 

Scott

--

[Andrey Lomakin]

Hi,
OK let's  look at /var/log/messages then.

If your application  is exited without your intention OS has to log it somewhere .

So let's keep  digging. 

Also what about DB server logs .

Could you send them to us ?

вс, 10 Апр 2016, 8:54 Erik Pragt <erik....@jworks.nl>:

You received this message because you are subscribed to the Google Groups "OrientDB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to orient-databa...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Erik Pragt]

On 10 Apr 2016, at 19:08, Andrey Lomakin <lomakin...@gmail.com> wrote:

Hi,
OK let's  look at /var/log/messages then.

I’m not sure if that’s normal, but I don’t have a /var/log/messages…. Any other location I could look?

If your application  is exited without your intention OS has to log it somewhere .

So let's keep  digging. 

Also what about DB server logs .

Could you send them to us ?

I already sent the server logs…but there was really nothing interesting in it.

[Andrey Lomakin]

Hi,

OK let's go further. 

1. Did you open database after server was shutted down ?
   
2. If so do you have exact copy of this database before server was started again ?

[Andrey Lomakin]

Also do  you know time when process was shutted down ? Is it possible to send us system logs in interval of hour before and after shutdown ?

[Erik Pragt]

On 11 Apr 2016, at 20:19, Andrey Lomakin <lomakin...@gmail.com> wrote:

Hi,

OK let's go further. 

1. Did you open database after server was shutted down ?
   

What do you mean exactly? Access it through the console? No, I didn’t. I got a message from my monitoring solution, checked if I could access the site myself, I couldn’t, checked the server, saw that OrientDb wasn’t running anymore, checked the server logs of my app and orientdb, started OrientDB and restarted my app, and that was it.

2. If so do you have exact copy of this database before server was started again ?

No, unfortunately not. Actually, the databases is completely different now, in terms of data and structure. I think right now, it’s hard to continue with this, without any information. If it happens again next time, I’ll make a copy of the DB, the logs, etc, and go from there….okay?

Erik

[Erik Pragt]

On 11 Apr 2016, at 20:21, Andrey Lomakin <lomakin...@gmail.com> wrote:

Also do  you know time when process was shutted down ? Is it possible to send us system logs in interval of hour before and after shutdown ?

Yes, I know the time. Sure, I can do that, but which systems logs?

Erik

[Andrey Lomakin]

Hi, 

Is it possible to send us /var/log logs but if possibly only for this time ?