All of a sudden we ran into an issue yesterday with our gerrit. We are using gerrit 2.5 and mysql 5.0.77. No one did any changes on server and no patches applied on server.
But we can not see all security groups any more. I am an administrator and I used to see all groups and used to create new groups and add users. BUt not any more. SInce yesterday even administrators do not see all groups or we lost access to create new groups or add new permissions on projects.
We only see groups we are part of and no one are able to push changes any more. Its impacting almost 200 developers. Please help. When i do /etc/init.d/mysqld status its says "mysql dead but subsys locked". I am not sure if its misleading us. I connected to mysql and reviewed tables. All tables have data intact.SO what is happening? Why we lost permissions?
1.Stopped mysql, gerrit, gitweb and restored complete file system including mysql to a good back up day
3. stopped mysqld , gerrit, git web and Deleted "mysqld" file from /var/lock/subsys/mysqld file and removed mysql.sock file and started the mysql. No use.
We did a complete restore to 2 days back. Still no use.
What is happening? NO idea? We are pretty big organization and a part of development wants to use gerrit because of benefits it provides. BUt not able to understand or fix this issue is little unbalancing.
Please help. No errors in application logs or database logs.
Mar 27, 2015 9:50:12 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/jdk1.6.0_25/jre/lib/amd64/server:/usr/java/jdk1.6.0_25/jre/lib/amd64:/usr/java/jdk1.6.0_25/jre/../lib/amd64:/gerrit/software/zlib-1.2.7/lib:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Mar 27, 2015 9:50:12 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'debug' to '0' did not find a matching property.
Mar 27, 2015 9:50:12 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-
Mar 27, 2015 9:50:12 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 473 ms
Mar 27, 2015 9:50:12 PM org.apache.catalina.realm.JAASRealm setContainer
INFO: Set JAAS app name Catalina
Mar 27, 2015 9:50:12 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Mar 27, 2015 9:50:12 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.37
Mar 27, 2015 9:50:12 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/gerrit/gerrit/config/WEB-INF/lib/servlet-api-6.0.29.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
[2015-03-27 21:50:14,770] INFO com.google.gerrit.server.cache.h2.H2CacheFactory : Enabling disk cache /gerrit/gerrit/config/cache
[2015-03-27 21:50:16,923] WARN com.google.gerrit.sshd.SshDaemon : Disabling cipher aes192-cbc: Illegal key size; try installing unlimited cryptography extension
[2015-03-27 21:50:16,923] WARN com.google.gerrit.sshd.SshDaemon : Disabling cipher aes256-cbc: Illegal key size; try installing unlimited cryptography extension
[2015-03-27 21:50:17,436] INFO com.google.gerrit.server.plugins.PluginLoader : Loading plugins from /gerrit/gerrit/config/plugins
[2015-03-27 21:50:17,536] INFO com.google.gerrit.sshd.SshDaemon : Started Gerrit SSHD on *:29418
Mar 27, 2015 9:50:18 PM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor manager.xml
Mar 27, 2015 9:50:18 PM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor host-manager.xml
Mar 27, 2015 9:50:18 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory docs
Mar 27, 2015 9:50:18 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory examples
Mar 27, 2015 9:50:18 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-
Mar 27, 2015 9:50:18 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 6025 ms