Could not restart a new dotcms container after first initialization

[peterLee]

Hi.

I'm trying to build dotcms CE docker image for my own usage.

For now, I just have to docker container linked : one for postgresql database and the other for docker

There is no issue during the first start.

But while i'm removing the dotcms container and create another one, the cms is not accessible anymore (blank page)

I have these informations inside container logs :

dotcms_1             | 16-Nov-2018 05:07:13.257 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 45957 ms
dotcms_1             | 05:07:17.086  WARN  business.HostAPIImpl - Content Index is fouled up, need to try db: [working_20181116045606] IndexNotFoundException[no such index]
dotcms_1             | 05:07:17.144  INFO  caffine.CaffineCache - ***    Building Cache : hostaliascache, size:1000,Concurrency:32
dotcms_1             | 05:07:17.164  ERROR business.HostAPIImpl - [working_20181116045606] IndexNotFoundException[no such index]
dotcms_1             | java.lang.RuntimeException: [working_20181116045606] IndexNotFoundException[no such index]
dotcms_1             |  at com.dotcms.content.elasticsearch.business.ESContentFactoryImpl.indexSearch(ESContentFactoryImpl.java:1424) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at com.dotcms.content.elasticsearch.business.ESContentletAPIImpl.searchIndex(ESContentletAPIImpl.java:772) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at com.dotcms.content.elasticsearch.business.ESContentletAPIImpl.search(ESContentletAPIImpl.java:633) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at com.dotcms.content.elasticsearch.business.ESContentletAPIImpl.search(ESContentletAPIImpl.java:624) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at com.dotmarketing.portlets.contentlet.business.ContentletAPIInterceptor.search(ContentletAPIInterceptor.java:1501) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at com.dotmarketing.portlets.contentlet.business.HostAPIImpl.findAll_aroundBody8(HostAPIImpl.java:340) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at com.dotmarketing.portlets.contentlet.business.HostAPIImpl$AjcClosure9.run(HostAPIImpl.java:1) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjrt-1.8.10.jar:?]
dotcms_1             |  at com.dotcms.aspects.aspectj.AspectJDelegateMethodInvocation.proceed(AspectJDelegateMethodInvocation.java:43) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at com.dotcms.aspects.interceptors.CloseDBIfOpenedMethodInterceptor.invoke(CloseDBIfOpenedMethodInterceptor.java:29) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at com.dotcms.aspects.aspectj.CloseDBIfOpenedAspect.invoke(CloseDBIfOpenedAspect.java:41) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at com.dotmarketing.portlets.contentlet.business.HostAPIImpl.findAll(HostAPIImpl.java:334) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at com.dotmarketing.servlets.InitServlet$InitThread.run_aroundBody0(InitServlet.java:333) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at com.dotmarketing.servlets.InitServlet$InitThread$AjcClosure1.run(InitServlet.java:1) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[aspectjrt-1.8.10.jar:?]
dotcms_1             |  at com.dotcms.aspects.aspectj.AspectJDelegateMethodInvocation.proceed(AspectJDelegateMethodInvocation.java:43) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at com.dotcms.aspects.interceptors.CloseDBIfOpenedMethodInterceptor.invoke(CloseDBIfOpenedMethodInterceptor.java:29) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at com.dotcms.aspects.aspectj.CloseDBIfOpenedAspect.invoke(CloseDBIfOpenedAspect.java:41) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  at com.dotmarketing.servlets.InitServlet$InitThread.run(InitServlet.java:306) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             | Caused by: org.elasticsearch.index.IndexNotFoundException: no such index
dotcms_1             |  at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:678) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:630) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:586) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:164) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:139) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:294) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.action.search.TransportSearchAction.lambda$doExecute$4(TransportSearchAction.java:193) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:113) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:86) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:215) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:68) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46) ~[elasticsearch-6.1.3.jar:6.1.3]
dotcms_1             |  at com.dotcms.content.elasticsearch.business.ESContentFactoryImpl.indexSearch(ESContentFactoryImpl.java:1414) ~[dotcms_5.0.2_999999.jar:?]
dotcms_1             |  ... 18 more
dotcms_1             | 05:07:18.196  INFO  reindex.ReindexThread - Reindex Thread started with a sleep of 500
dotcms_1             | 05:08:00.143  INFO  job.FreeServerFromClusterJob - Server e9c56217-af4c-404b-925c-85989cf8ce09 with license null was Removed
dotcms_1             | 05:08:00.318  INFO  caffine.CaffineCache - ***    Building Cache : publishingendpointcache, size:100,Concurrency:32
dotcms_1             | 05:09:49.355  INFO  caffine.CaffineCache - ***    Building Cache : hostaliascache, size:1000,Concurrency:32
dotcms_1             | 05:09:49.365  WARN  web.HostWebAPIImpl - Content Index is fouled up, need to try db: [working_20181116045606] IndexNotFoundException[no such index]
dotcms_1             | 05:09:49.391  INFO  caffine.CaffineCache - ***    Building Cache : vanityurlcache, size:1000,Concurrency:32
dotcms_1             | 05:09:49.393  INFO  caffine.CaffineCache - ***    Building Cache : cachedvanityurlgroup, size:1000,Concurrency:32
dotcms_1             | 05:09:49.395  ERROR business.VanityUrlAPIImpl - Error when initializing Vanity URLs, no index found
dotcms_1             | 05:09:49.412  INFO  caffine.CaffineCache - ***    Building Cache : hostsgroup, size:1000,Concurrency:32
dotcms_1             | 05:09:49.560  INFO  caffine.CaffineCache - ***    Building Cache : versioninfocache, size:50000,Concurrency:32

Each new request to dotcms website produce other logs :

dotcms_1             | 05:18:21.520  ERROR business.VanityUrlAPIImpl - Error when initializing Vanity URLs, no index found
dotcms_1             | 05:18:21.543  ERROR business.VanityUrlAPIImpl - Error when initializing Vanity URLs, no index found

Here are attachments for dockerfile and container logs (2nd start)

[Brent Griffin]

It is really difficult for us to support custom containers since we have no idea how things are installed, etc.  On the surface, it appears that your ElasticSearch index is not being persisted.

I would encourage you to use images that we have created for general usage.  Those images are available here:  https://hub.docker.com/u/dotcms/dashboard/  and you can refer to these reference implementations for examples on how they can be used and configured:  https://github.com/dotCMS/docker/tree/master/referenceImplementations

All of the reference implementations persist the ES index and work properly after restarts.

Brent

[peterLee]

Hi,

I'm now using the official docker image for dotcms.

There is another issue.

While I'm restarting the container, hosted on AWS, i'm loosing all demo files inside admin browser (cf picture in attachment). There is only folder which are displayed.

Where the files are located (database or filesystem)?

[Brent Griffin]

The persistent file location is going to depend on your configuration.  How are you running the containers?  Please send you config file (docker-compose.yml or kubernetes yaml, etc)

For example, in this file, the persistence is handled but the volumes - https://github.com/dotCMS/docker/blob/master/referenceImplementations/02-single_node/docker-compose.yml

Here is an example of how persistence can be handled in a Kubernetes environment:  https://github.com/brentgriffin/2018BootcampK8sDemo

Please send the details of how you are configuring and starting your containers and can help you from there.

Brent