Core Dump (1.4.22 github)

[Casey Ryback]

Hi guys,
Upgraded my OLS to the 1.4.22 version that was added to github, and just noticed this error in my logs:
 [NOTICE] [AutoRestarter] child process with pid=24152 received signal=6, a core file is created!

Core dump:
(gdb) bt
#0  0x00007a06a4a03067 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007a06a4a04448 in __GI_abort () at abort.c:89
#2  0x00007a06a49fc266 in __assert_fail_base (fmt=0x7a06a4b35238 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x6d64ee "m_fd != -1", file=file@entry=0x6d64aa "staticfilecachedata.cpp", line=line@entry=251,
    function=function@entry=0x6d65c0 <FileCacheDataEx::getCacheData(long, long&, char*, long)::__PRETTY_FUNCTION__> "const char* FileCacheDataEx::getCacheData(off_t, off_t&, char*, long int)") at assert.c:92
#3  0x00007a06a49fc312 in __GI___assert_fail (assertion=0x6d64ee "m_fd != -1", file=0x6d64aa "staticfilecachedata.cpp", line=251,
    function=0x6d65c0 <FileCacheDataEx::getCacheData(long, long&, char*, long)::__PRETTY_FUNCTION__> "const char* FileCacheDataEx::getCacheData(off_t, off_t&, char*, long int)") at assert.c:101
#4  0x00000000004c297a in FileCacheDataEx::getCacheData (this=0x48b8ff0, offset=0, wanted=wanted@entry=@0x7c74ac8ed178: 3420,
    pBuf=0xa03500 <HttpResourceManager::g_aBuf> "/home/www/REDACTED/public/data/taigachat/messagesmini.html", len=len@entry=3420)
    at staticfilecachedata.cpp:251
#5  0x00000000004e994f in HttpSession::sendStaticFileEx (this=this@entry=0x4844e90, pData=pData@entry=0x4845398) at httpsession.cpp:3701
#6  0x00000000004e9d1a in HttpSession::sendStaticFile (this=0x4844e90, pData=0x4845398) at httpsession.cpp:3736
#7  0x00000000004ea0e9 in HttpSession::flushBody (this=this@entry=0x4844e90) at httpsession.cpp:3072
#8  0x00000000004eaaf8 in HttpSession::flush (this=0x4844e90) at httpsession.cpp:3154
#9  0x00000000004cca93 in StaticFileHandler::process (this=0x5e58, pSession=0x4844e90, pHandler=0x6) at staticfilehandler.cpp:598
#10 0x00000000004e62c9 in HttpSession::handlerProcess (this=this@entry=0x4844e90, pHandler=<optimized out>) at httpsession.cpp:1658
#11 0x00000000004ec2f9 in HttpSession::smProcessReq (this=this@entry=0x4844e90) at httpsession.cpp:4155
#12 0x00000000004ed5d3 in HttpSession::onReadEx (this=0x4844e90) at httpsession.cpp:1970
#13 0x00000000004d7855 in NtwkIOLink::handleEvents (this=0x4849d10, evt=<optimized out>) at ntwkiolink.cpp:390
#14 0x0000000000537da8 in processAllEvents (this=0x4768be0) at ../../src/edio/pollfdreactor.h:150
#15 Poller::waitAndProcessEvents (this=0x4768bd0, iTimeoutMilliSec=<optimized out>) at poller.cpp:77
---Type <return> to continue, or q <return> to quit---
#16 0x00000000004cb701 in EventDispatcher::run (this=this@entry=0x4732ad8) at eventdispatcher.cpp:235
#17 0x00000000004ad370 in HttpServerImpl::start (this=0x4732ab0) at httpserver.cpp:472
#18 0x00000000004b6279 in HttpServer::start (this=<optimized out>) at httpserver.cpp:3693
#19 0x000000000048b860 in LshttpdMain::main (this=this@entry=0x4732860, argc=argc@entry=1, argv=argv@entry=0x7c74ac8ed598) at lshttpdmain.cpp:936
#20 0x000000000048b7b2 in main (argc=1, argv=0x7c74ac8ed598) at main.cpp:109

Also had bunch of errors about real time reports, but I solved that by doing chown nobody:nogroup /dev/shm/ols

Casey

[David]

Hi Casey,

Are you using the lastest 1.4.22 which was uploaded yesterday which should have fixed the permission of /dev/shm/ols?
About the bug which cause core dump, we will debug it.

Thanks for your feedback.
David

--
You received this message because you are subscribed to the Google Groups "OpenLiteSpeed Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlitespeed-deve...@googlegroups.com.
To post to this group, send email to openlitespee...@googlegroups.com.
Visit this group at https://groups.google.com/group/openlitespeed-development.
For more options, visit https://groups.google.com/d/optout.

[Casey Ryback]

Yes, I am using the latest build that was added to github yesterday. I was using 1.4.21 before that and can't remember seeing the same error on that one. (Did see report from other OLS user(s) mentioning the error tho).

To unsubscribe from this group and stop receiving emails from it, send an email to openlitespeed-development+unsub...@googlegroups.com.

[David]

Hi Casey,

I am debugging that error which cause core dump, but I can not reproduce that issue.

The below is a bug I found but maybe not the reason cause that core dump.
Can you apply that and re-make the binary and overwrite the binary file, then check if the core dump still exists?

diff --git src/http/staticfilecachedata.cpp src/http/staticfilecachedata.cpp
index 7e904b4..c0f7cd0 100644
--- src/http/staticfilecachedata.cpp
+++ src/http/staticfilecachedata.cpp
@@ -786,7 +786,11 @@ int StaticFileCacheData::readyCacheData(
     if ((m_fileData.isCached() ||
          (m_fileData.getfd() != -1)))
         return 0;
-    return m_fileData.readyData(pFileName);
+
+    ret = m_fileData.readyData(pFileName);
+    if (ret)
+        pECache->decRef();
+    return ret;
 }

And if still get a core file and worker Auto Restart, can you also test the bellow patch, if this make a difference or not.

diff --git src/http/httpvhost.cpp src/http/httpvhost.cpp
index 82e1e25..f79b87f 100644
--- src/http/httpvhost.cpp
+++ src/http/httpvhost.cpp
@@ -2621,7 +2621,7 @@ void HttpVHost::urlStaticFileHashClean()
     TPointerList<static_file_data_t>::iterator it;
     for (it = m_UrlStxFileDirtyList.begin(); it != m_UrlStxFileDirtyList.end();)
     {
-        if ((*it)->pData->getRef() <= 0)
+        if ((*it)->pData->getFileData()->getRef() <= 0)
         {
             LS_DBG_L("[VHost:%s][static file cache] Clean list.", getName());
             delete *it;
@@ -2635,11 +2635,11 @@ void HttpVHost::urlStaticFileHashClean()
     for (itt = m_pUrlStxFileHash->begin(); itt != m_pUrlStxFileHash->end();)
     {
         static_file_data_t *data = (static_file_data_t *)itt.second();
-        if (data->pData->getRef() <= 1)
+        if (data->pData->getFileData()->getRef() <= 1)
         {
             LS_DBG_L("[VHost:%s][static file cache] Clean HashT.", getName());
-            data->pData->reset();
-            data->pData->getFileData()->reset();
+            data->pData->decRef();
+            data->pData->getFileData()->decRef();
             delete data;
             m_pUrlStxFileHash->erase(itt);
         }

Again, I am still debugging that issue, please let me know anything relative with that issue.

Thanks.

David



On 8/12/2016 10:33 PM, Casey Ryback wrote:

--
You received this message because you are subscribed to the Google Groups "OpenLiteSpeed Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openlitespeed-deve...@googlegroups.com.

[Casey Ryback]

Hi,

I haven't seen that error occur again since I saw it for the first time, so whatever it was it doesn't seem to happen often at all. I did apply the (first) patch you posted, and will keep monitoring the logs for any weird behavior.

Casey

To unsubscribe from this group and stop receiving emails from it, send an email to openlitespeed-development+unsub...@googlegroups.com.

[David]

Good to know. When get that issue happens again, please let me know.
Thanks.
David

To unsubscribe from this group and stop receiving emails from it, send an email to openlitespeed-deve...@googlegroups.com.

[Casey Ryback]

Just checked the logs and same error had occured today three times within a three hour period. Was running fine without autorestarts until today.
Attached bt of the latest dump, but to me it looked almost identical to the first one.

I have now upgraded my OLS to the latest github version, which seems to include all the patches mentioned here. Will keep yall posted it the error occurs again.

Casey

[David]

Good. Please notice me immediately when it happens again.
Thanks.
David

To unsubscribe from this group and stop receiving emails from it, send an email to openlitespeed-deve...@googlegroups.com.

[Casey Ryback]

No errors so far, but had a quesion about one other weird behavior that's been happening and didn't want to start new topic...
Anyhow, I have got one dedicated server and two dev/backups sites (openvz) all running OLS on Debian 8, and all of them get these errors logged to stderr.log every night (stderr.log has last modified time 02:59 on all servers):
Failed to open temp file for swapping: Permission denied

Any idea what might trigger it? I kept debug logging running on one of the test sites to try and catch the reason, but nothing at all in the log. (No users/traffic during said time). Not sure exactly when this behavior started, but IIRC at least 1.4.16 didn't log these.

[Casey Ryback]

Error just happened again. Core dump looks almost identical to the first one, see attached file.
stderr.log "openlitespeed (lshttpd - #01): staticfilecachedata.cpp:251: const char* FileCacheDataEx::getCacheData(off_t, off_t&, char*, long int): Assertion `m_fd != -1' failed."

Casey

[David]

Sorry to hear that. I do know that the bug fix may not fix that issue because I can not reproduce it.
Still need time to review the code and find out.
Thanks.
David

On 8/20/2016 7:56 AM, Casey Ryback wrote:

Error just happened again. Core dump looks almost identical to the first one, see attached file.

To unsubscribe from this group and stop receiving emails from it, send an email to openlitespeed-deve...@googlegroups.com.

[David]

HI Casey ,

Can you update the code a little bit and rebuild the binary, then test if the crash still exist.
Again, I can not reproduce that crash, so I just view the code and try to find out which may cause that problem.

diff --git src/http/httpsession.cpp src/http/httpsession.cpp
index 8fda53c..31d8f47 100644
--- src/http/httpsession.cpp
+++ src/http/httpsession.cpp
@@ -321,7 +321,7 @@ inline void HttpSession::releaseStaticFileCacheData(StaticFileCacheData
 inline void HttpSession::releaseFileCacheDataEx(FileCacheDataEx *&pECache)
 {
     if (pECache && pECache->decRef() <= 0)
-        pECache->release();
+        pECache->closefd();

 }

Thanks.
David




On 8/20/2016 7:56 AM, Casey Ryback wrote:

Error just happened again. Core dump looks almost identical to the first one, see attached file.

To unsubscribe from this group and stop receiving emails from it, send an email to openlitespeed-deve...@googlegroups.com.

[David]

Hi Casey,

So did you test the core dump fixed or not with the patch?

Thanks.
David