[error] emulator Error in process

[tpbradley]

Hi all,

I'm using zotonic 0.9dev with the twitter module. After a while the server will crash with the following messages in the log files. These message repeat approximately 550 times before the zotonic process dies. From what I can see, there is no erl_crash_dump file with additional information. Our server has also been periodically hitting 100% CPU usage for approximately 2 hours.

Does anyone have any thoughts?

Thanks

From console.log

2012-11-06 02:52:09.491 [error] emulator Error in process <0.29101.21> on node 'zotonic001@localhost' with exit value: {undef,[{ssl_session_cache,delete,[{{"www.surveymonkey.com",443},<<32 bytes>>}]}]}

2012-11-06 02:53:09.491 [error] emulator Error in process <0.30026.21> on node 'zotonic001@localhost' with exit value: {undef,[{ssl_session_cache,delete,[{{"www.surveymonkey.com",443},<<32 bytes>>}]}]}

2012-11-06 02:53:09.492 [error] emulator Error in process <0.30027.21> on node 'zotonic001@localhost' with exit value: {undef,[{ssl_session_cache,delete,[{{"stream.twitter.com",443},<<32 bytes>>}]}]}

2012-11-06 02:54:09.492 [error] emulator Error in process <0.30935.21> on node 'zotonic001@localhost' with exit value: {undef,[{ssl_session_cache,delete,[{{"stream.twitter.com",443},<<32 bytes>>}]}]}

From crash.log

2012-11-06 02:53:09 =ERROR REPORT====

Error in process <0.30026.21> on node 'zotonic001@localhost' with exit value: {undef,[{ssl_session_cache,delete,[{{"www.surveymonkey.com",443},<<32 bytes>>}]}]}

2012-11-06 02:53:09 =ERROR REPORT====

Error in process <0.30027.21> on node 'zotonic001@localhost' with exit value: {undef,[{ssl_session_cache,delete,[{{"stream.twitter.com",443},<<32 bytes>>}]}]}

2012-11-06 02:54:09 =ERROR REPORT====

Error in process <0.30935.21> on node 'zotonic001@localhost' with exit value: {undef,[{ssl_session_cache,delete,[{{"stream.twitter.com",443},<<32 bytes>>}]}]}

2012-11-06 02:54:09 =ERROR REPORT====

Error in process <0.30934.21> on node 'zotonic001@localhost' with exit value: {undef,[{ssl_session_cache,delete,[{{"www.surveymonkey.com",443},<<32 bytes>>}]}]}

From error.log

2012-11-06 02:53:09.491 [error] emulator Error in process <0.30026.21> on node 'zotonic001@localhost' with exit value: {undef,[{ssl_session_cache,delete,[{{"www.surveymonkey.com",443},<<32 bytes>>}]}]}

2012-11-06 02:53:09.492 [error] emulator Error in process <0.30027.21> on node 'zotonic001@localhost' with exit value: {undef,[{ssl_session_cache,delete,[{{"stream.twitter.com",443},<<32 bytes>>}]}]}

2012-11-06 02:54:09.492 [error] emulator Error in process <0.30935.21> on node 'zotonic001@localhost' with exit value: {undef,[{ssl_session_cache,delete,[{{"stream.twitter.com",443},<<32 bytes>>}]}]}

2012-11-06 02:54:09.493 [error] emulator Error in process <0.30934.21> on node 'zotonic001@localhost' with exit value: {undef,[{ssl_session_cache,delete,[{{"www.surveymonkey.com",443},<<32 bytes>>}]}]}

[Marc Worrell]

Hi,

That ssl_session_cache error you see is mosty a nuisance.

Though it does consume lots of disk space (logs) and cpu just generating/handling it.

It was introduced in the R14B03 (or 02?) release,  R14B04 doesn't have that error.

Try upgrading your Erlang install to a newer release.

We are running 0.9dev in production and it is stable.

At least I don't see any CPU spikes, we are running on R14B04.

- Marc

[tpbradley]

Ahh, I didn't realise there was a new version of erlang... I'm currently on R14B03 so I'll give R14B04 a try.

Thanks Marc!

[tpbradley]

Hi Marc,

I updated my erlang install to version R14B04 yesterday and restarted the zotonic server. Shortly after, I could see a beam process running at 100% again, so I disabled the twitter module in case it was causing any issues. After restarting zotonic, I no longer had any processes using 100% CPU but this morning I checked the server and all beam/heart process had vanished. I couldn't see anything obvious in the logs and no crash file had been created. So, I tried running zotonic using the start.sh file to give me a shell to see if anything happened. After about 30 mins, the shell died with the following output.

/home/zotonic/src/scripts/zotonic-debug: line 22: 24462 Killed                  $ERL -env ERL_MAX_PORTS $MAX_PORTS +P 10000000 +K true -pa $PA -name $SNAME@$HOSTNAME -boot start_sasl -config $ZOTONIC/priv/erlang.config -sasl errlog_type error -s zotonic

I'm running version 0.9 of zotonic from Sat Oct 6 19:46:05

commit ca6e9c55901ee2352c08dd5a507f13a5f2ab8a0c

Thanks

Tom

[tpbradley]

A quick question... when you install a new version of erlang, do you have to rebuild zotonic by running make?

Cheers

[Arjan Scherpenisse]

Yep, that's best.
$ make clean
and then
$ make

[Marc Worrell]

On 7 nov. 2012, at 10:40, tpbradley wrote:
> /home/zotonic/src/scripts/zotonic-debug: line 22: 24462 Killed $ERL -env ERL_MAX_PORTS $MAX_PORTS +P 10000000 +K true -pa $PA -name $SNAME@$HOSTNAME -boot start_sasl -config $ZOTONIC/priv/erlang.config -sasl errlog_type error -s zotonic

Strange, when there is nothing in the logs then it makes me think that either:

- something is killing it
- hardware failure; is core dump turned off?

Is heart running?
If so, try without heart, maybe it is killing beam because it thinks that beam got stuck.

- Marc

[tpbradley]

Cheers guys... I've grabbed the latest zotonic from github and recompiled everything. So far everything is running fine with the twitter module enabled again. I don't think there is a hardware failure as we also run an apache server on the same machine without issue. The machine is hosted by linode.com. I'll turn core dump on to see if it picks up anything.

Heart is not currently running. I've started zotonic with the shell to monitor console output. Is that why heart is not running? Are there other ways to start zotonic without the heart process?

Thanks

[Marc Worrell]

Did you also do a separate git-pull-make-clean on the ua_classifier?

Just to be sure :)  I had some strange crashes in that one before I fixed a Heisenberg bug in there.

- Marc

[tpbradley]

ooo, where is the ua_classifier? I should be ok because I've created a whole new directory and re-checked out the zotonic source (git clone https://github.com/zotonic/zotonic.git).

Cheers

[Marc Worrell]

It is in 'deps', but when you did a clean checkout then it will be ok.

- Marc

[tpbradley]

Ah, excellent!

I think also that some of my crashes were down to memory problems. We only had 256mb of virtual memory available and from scanning through the kernel logs I can see the kernel killing the beam process because it was running out of memory. I've increased the VM size on our box up to 1024mb so I'll keep an eye on things to see how it performs over the next few days.

Cheers!!

[tpbradley]

Ok, everything is running really well now... No more runaway or disappearing processes and the memory usage is down to about 6% ;)

Thanks for all the help!!