problems with a new installation

[Helena Bermúdez Sabel]

Hello!

I recently installed INCEpTION in a new server (RHEL 8) and it went well, but this morning the service stopped. This was the log message:

Mar 11 08:20:35 xx.unine.ch systemd[1]: Stopping INCEpTION...
Mar 11 08:21:05 xx.unine.ch inception.jar[5614]: 2021-03-11 08:21:05 WARN [SYSTEM] SqlExceptionHelper - SQL Error: 0, SQLState: 08000
Mar 11 08:21:05 xx.unine.ch inception.jar[5614]: 2021-03-11 08:21:05 ERROR [SYSTEM] SqlExceptionHelper - HikariPool-1 - Connection is not available, request timed out after 30001ms.
Mar 11 08:21:05 xx.unine.ch inception.jar[5614]: 2021-03-11 08:21:05 WARN [SYSTEM] SqlExceptionHelper - SQL Error: -1, SQLState: 08000
Mar 11 08:21:05 xx.unine.ch inception.jar[5614]: 2021-03-11 08:21:05 ERROR [SYSTEM] SqlExceptionHelper - Could not connect to address=(host=localhost)(port=3306)(type=master) : Socket fail to connect to host:localhost, port:3306. Connection refused (Connection refused)
Mar 11 08:21:05 xx.unine.ch inception.jar[5614]: 2021-03-11 08:21:05 WARN [SYSTEM] DisposableBeanAdapter - Invocation of destroy method failed on bean with name 'eventLoggingListener': org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
Mar 11 08:21:05 xx.unine.ch inception.jar[5614]: 2021-03-11 08:21:05 INFO [SYSTEM] SearchServiceImpl - Shutting down search service!
Mar 11 08:21:05 xx.unine.ch inception.jar[5614]: 2021-03-11 08:21:05 INFO [SYSTEM] SchedulingService - Shutting down scheduling service!
Mar 11 08:21:13 xx.unine.ch systemd[1]: inception.service: Succeeded.
Mar 11 08:21:13 xx.unine.ch systemd[1]: Stopped INCEpTION.
-- Reboot --
Mar 11 08:24:32 xx.unine.ch systemd[1]: Started INCEpTION.
Mar 11 08:24:32 xx.unine.ch systemd[1]: inception.service: Main process exited, code=exited, status=203/EXEC
Mar 11 08:24:32 xx.unine.ch systemd[1]: inception.service: Failed with result 'exit-code'.

Since the problem seemed to be related to the SQL database and I have been having problems with some packages of MariaDB, I uninstalled it and installed MySQL instead (I haven’t used that new installation of INCEpTION yet, so there was nothing to back-up).

1. I downloaded the new .jar file (version 0.18.3) and changed permits (the owner is the user apache, the www-data equivalent in RHEL).

2. The database url of settings.properties was updated:

database.url=jdbc:mysql://localhost:3306/inception?useSSL=false&serverTimezone=UTC&useUnicode=true&characterEncoding=UTF-8&allowPublicKeyRetrieval=true

3. The database inception exists and so does the user with the granted privileges. The password is correctly declared in the settings.properties file.

I don’t know what I might be missing. In /var/log I have no specific log files for INCEpTION. And service inception status only says:

Redirecting to /bin/systemctl status inception.service
● inception.service - INCEpTION
   Loaded: loaded (/etc/systemd/system/inception.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2021-03-11 14:56:50 CET; 6min ago
  Process: 4731 ExecStart=/srv/inception/inception.jar (code=exited, status=203/EXEC)
 Main PID: 4731 (code=exited, status=203/EXEC)

Mar 11 14:56:50 xx.unine.ch systemd[1]: Started INCEpTION.
Mar 11 14:56:50 xx.unine.ch systemd[1]: inception.service: Main process exited, code=exited, status=203/EXEC
Mar 11 14:56:50 xx.unine.ch systemd[1]: inception.service: Failed with result 'exit-code'.

The version I had previously installed was 18.02 but I doubt this is a bug of 18.03: I think it is more likely that something went wrong with my configuration. Any ideas where the problem might be?

Thank you in advance for your time!

Helena

[Richard Eckart de Castilho]

Hi,

> On 11. Mar 2021, at 15:12, Helena Bermúdez Sabel <helena....@gmail.com> wrote:
>
> Mar 11 08:21:05 xx.unine.ch inception.jar[5614]: 2021-03-11 08:21:05 ERROR [SYSTEM] SqlExceptionHelper - Could not connect to address=(host=localhost)(port=3306)(type=master) : Socket fail to connect to host:localhost, port:3306. Connection refused (Connection refused)

Is the DB actually running? Can you "connect" to it using the "mysql" command on the command line?

Is the DB configured to accept TCP/IP network connections? It may be configured to permit only Unix Socket connections.

Does the user you try to log in with actually have the permission to connect to the host?

-- Richard

[Helena Bermúdez Sabel]

Thank you for your answer, Richard!

- Is the DB actually running? Can you "connect" to it using the "mysql" command on the command line?

yes

- Is the DB configured to accept TCP/IP network connections? It may be configured to permit only Unix Socket connections.

On the mysql conf file I added the lines:

port=3306
bind-address=0.0.0.0

And I opened the port 3306

firewall-cmd --zone=public --add-service=mysql --permanent

firewall-cmd --zone=public --add-port=3306/tcp --permanent

Thus, I think that it is now configured to accept TCP/IP connections, but the problem persists.

- Does the user you try to log in with actually have the permission to connect to the host?

Yes:

mysql> SHOW GRANTS FOR 'inception'@'localhost';
+------------------------------------------------------------------+
| Grants for inception@localhost                                   |
+------------------------------------------------------------------+
| GRANT USAGE ON *.* TO `inception`@`localhost`                    |
| GRANT ALL PRIVILEGES ON `inception`.* TO `inception`@`localhost` |
+------------------------------------------------------------------+

And in the settings.properties file:

database.url=jdbc:mysql://localhost:3306/inception?useSSL=false&serverTimezone=UTC&useUnicode=true&characterEncoding=UTF-8&allowPublicKeyRetrieval=true
database.username=inception
database.password=the_password_I_gave_to_user_inception

I am going to redo the installation from scratch to see if there is something I am missing!

Thank you again for these suggestions and if you can think of anything else, please let me know.

All the best,

Helena

--
You received this message because you are subscribed to the Google Groups "inception-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to inception-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/inception-users/A9EA4FB9-895F-40D8-94B9-61C0C81B7274%40gmail.com.

[Helena Bermúdez Sabel]

Hello all,

I am sorry for persisting with this, but I did some troubleshooting and maybe some of you have additional suggestions.

If I run the jar file in the server, INCEpTION starts. I opened the port where I have it running, and indeed I can log in, create projects and so on. Thus it seems that the problem is when I try to run it as a service. However, I cannot see what might be wrong. I added the debug option in systemd/system.conf to get additional information but the error message is still not very informative:

Redirecting to /bin/systemctl status inception.service
● inception.service - INCEpTION
   Loaded: loaded (/etc/systemd/system/inception.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Fri 2021-03-12 22:23:22 CET; 15s ago
  Process: 14698 ExecStart=/srv/inception/inception.jar (code=exited, status=203/EXEC)
 Main PID: 14698 (code=exited, status=203/EXEC)

Mar 12 22:23:22 xx.unine.ch systemd[1]: inception.service: Changed failed -> running
Mar 12 22:23:22 xx.unine.ch systemd[1]: inception.service: Job inception.service/start finished, result=done
Mar 12 22:23:22 xx.unine.ch systemd[1]: Started INCEpTION.
Mar 12 22:23:22 xx.unine.ch systemd[1]: inception.service: User lookup succeeded: uid=48 gid=48
Mar 12 22:23:22 xx.unine.ch systemd[1]: inception.service: Child 14698 belongs to inception.service.
Mar 12 22:23:22 xx.unine.ch systemd[1]: inception.service: Main process exited, code=exited, status=203/EXEC
Mar 12 22:23:22 xx.unine.ch systemd[1]: inception.service: Failed with result 'exit-code'.
Mar 12 22:23:22 xx.unine.ch systemd[1]: inception.service: Changed running -> failed
Mar 12 22:23:22 xx.unine.ch systemd[1]: inception.service: Unit entered failed state.

If anybody can suggest additional tests, I will be more than happy to try them.

Thank you and all the best,

H.

[Jan-Christoph Klie]

Hi,

can you post the content of the inception.service file, the output of "journalctl -u inception" and the permission of the INCEpTION jar file ("ls -lha" in the folder where your INCEpTION is, ideally also the permissions of the settings.properties)?

Best,

Jan

[Helena Bermúdez Sabel]

Hello, Jan!

Thank you for you message!

The permissions of /srv/inception (I am in RHEL 8, that’s why the user is apache instead of www-data):

[root@vxx inception]# ls -lha
total 259M
drwxr-xr-x. 2 apache apache   97 Mar 12 23:28 .
drwxr-xr-x. 3 root   root     23 Mar 11 19:17 ..
-rw-r--r--. 1 apache apache   69 Mar 12 19:32 inception.conf
-rwxr-xr-x. 1 apache apache 259M Mar  9 20:33 inception.jar
-rw-r--r--. 1 apache apache  486 Mar 12 21:20 settings.properties

The contents of /etc/systemd/system/inception.service

[Unit]
Description=INCEpTION

[Service]
ExecStart=/srv/inception/inception.jar
User=apache

[Install]
WantedBy=multi-user.target

And the contents of journalctl -u inception are attached. As you can see in the log, I could run INCEpTION at first. Then it stopped working. The log seemed to indicate that the issue was with the database and I had a small problem with a mariadb package, so I decided to uninstall and reinstall the database. It never worked back again and the only message I get is the unhelpful (for me) “inception.service: Main process exited, code=exited, status=203/EXEC”.

Any suggestions that might shed some light on the issue would be greatly appreciated!

Best,

Helena

To view this discussion on the web visit https://groups.google.com/d/msgid/inception-users/df24cf29-d0f1-4982-b9f4-16b4573fa24bn%40googlegroups.com.

[Jan-Christoph Klie]

Hi,

from the information you sent me I see no obvious error, you set it up by the books. What I would check first is whether it is a problem with the db or something else. Can you move/rename the config files so that INCEPTION starts with systemd and runs without any custom config, that is no settings.properties and inception.conf? Then it should use the embedded db. What is the content of the inception.conf and settings.properties (you can redact passwords)? You say it runs when you only start the jar, does it then also uses the config of yours + the maria/mysql db or embedded if you start it by hand?

Best,

[Helena Bermúdez Sabel]

Hello, Jan!

Thank you again for all your help with this. I now know what was the source of the problem: SELinux was enabled without the proper configuration.

All the best,

H.

To view this discussion on the web visit https://groups.google.com/d/msgid/inception-users/8f230fd1-f520-4799-b6fb-fd8a35e799a3n%40googlegroups.com.

[Jan-Christoph Klie]

I am happy that you found out what the error was. Good find! I would never have thought of that. We will add this to the setup documentation to disable SELinux.