Jenkins version : Jenkins ver. 2.32.1 Google Login Plugin version : 1.3
I got this error after I restart my jenkins, can anyone suggest me how to solve it ?
The thing is :
-I already make sure that in plugins directory I had the google-login plugin enabled
-I didn't change anything on the plugins, just straight restart
-It's been so long since I restart my jenkins
hudson.util.HudsonFailedToLoad: org.jvnet.hudson.reactor.ReactorException: java.io.IOException: Unable to read /var/lib/jenkins/config.xml at hudson.WebAppMain$3.run(WebAppMain.java:248)
Caused by: org.jvnet.hudson.reactor.ReactorException: java.io.IOException: Unable to read /var/lib/jenkins/config.xml at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:269) at jenkins.InitReactorRunner.run(InitReactorRunner.java:47) at jenkins.model.Jenkins.executeReactor(Jenkins.java:1110) at jenkins.model.Jenkins.<init>(Jenkins.java:926) at hudson.model.Hudson.<init>(Hudson.java:85) at hudson.model.Hudson.<init>(Hudson.java:81) at hudson.WebAppMain$3.run(WebAppMain.java:231)
Caused by: java.io.IOException: Unable to read /var/lib/jenkins/config.xml at hudson.XmlFile.unmarshal(XmlFile.java:161) at jenkins.model.Jenkins.loadConfig(Jenkins.java:3015) at jenkins.model.Jenkins.access$1100(Jenkins.java:326) at jenkins.model.Jenkins$16.run(Jenkins.java:3033) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:1086) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) at java.lang.Thread.run(Thread.java:748)
Caused by: jenkins.util.xstream.CriticalXStreamException: org.jenkinsci.plugins.googlelogin.GoogleOAuth2SecurityRealm : org.jenkinsci.plugins.googlelogin.GoogleOAuth2SecurityRealm
---- Debugging information ----
message : org.jenkinsci.plugins.googlelogin.GoogleOAuth2SecurityRealm
cause-exception : com.thoughtworks.xstream.mapper.CannotResolveClassException
cause-message : org.jenkinsci.plugins.googlelogin.GoogleOAuth2SecurityRealm
class : hudson.model.Hudson
required-type : hudson.model.Hudson
converter-type : hudson.util.RobustReflectionConverter
path : /hudson/securityRealm
line number : 485
version : not available
------------------------------- at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:356) at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66) at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50) at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189) at hudson.util.XStream2.unmarshal(XStream2.java:114) at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173) at hudson.XmlFile.unmarshal(XmlFile.java:159) ... 11 more
Caused by: com.thoughtworks.xstream.mapper.CannotResolveClassException: org.jenkinsci.plugins.googlelogin.GoogleOAuth2SecurityRealm at com.thoughtworks.xstream.mapper.DefaultMapper.realClass(DefaultMapper.java:79) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.DynamicProxyMapper.realClass(DynamicProxyMapper.java:55) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.PackageAliasingMapper.realClass(PackageAliasingMapper.java:88) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.ClassAliasingMapper.realClass(ClassAliasingMapper.java:79) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.ArrayMapper.realClass(ArrayMapper.java:74) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.SecurityMapper.realClass(SecurityMapper.java:71) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at hudson.util.XStream2$CompatibilityMapper.realClass(XStream2.java:282) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at hudson.util.xstream.MapperDelegate.realClass(MapperDelegate.java:43) at com.thoughtworks.xstream.mapper.MapperWrapper.realClass(MapperWrapper.java:30) at com.thoughtworks.xstream.mapper.CachingMapper.realClass(CachingMapper.java:48) at hudson.util.RobustReflectionConverter.determineType(RobustReflectionConverter.java:461) at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:327) ... 22 more5 Answers
I ran into a similar problem today, while upgrading Jenkins from 2.176.1 to 2.176.2. The upgrade itself wasn't the problem though; it was plugins.
Summary
Your post only contains a subset of the log file, but in my case, I found some entries that pointed to plugins as the cause of the problem. The important piece of data in the log was:
[...]
Jul 17, 2019 10:13:53 AM jenkins.InitReactorRunner$1 onTaskFailed
SEVERE: Failed Loading plugin SAML Plugin v1.1.2 (saml)
java.io.IOException: SAML Plugin version 1.1.2 failed to load. - bouncycastle-api version 2.16.0 is older than required. To fix, install version 2.16.1 or later.
[...]And any other entries that look similar in format and communicate similar info.
Root-Cause
My system had a few plugins with pending updates. When upgrading services (e.g. apt upgrade jenkins, etc), the service is usually restarted as part of the process. The restart is what brings the issue to light, showing that some plugins were failing to load because they now required more up-to-date dependencies.
These plugins don't simply cause errors to get logged: they throw exceptions. Jenkins does not appear to be ready to handle them, and crashes. Visiting the main Jenkins page will show a big fat stack-trace and nothing else.
The catch is that, in order to properly upgrade the plugins, you need Jenkins to be up-and-running, but the offending plugins prevent Jenkins from getting to the needed up-and-running state in the first place.
(My) Solution
There were several goals:
- Get Jenkins to a point where it can initialize and remain in a running state;
- Get access to the admin controls to upgrade the offending plugins;
- Restart the Jenkins service and return to normal operations.
Notes Before Proceeding
A few things to either keep in mind or try out before moving onto the next sections:
- By default, Jenkins gets installed under
/var/lib/jenkins/and itsconfig.xmlfile is under this directory. Create a backup of thisconfig.xmlfile before proceeding! - You should have a good-enough understanding and be reasonably convinced you have a similar-enough issue to justify trying this.
- The log messages about not being able to read
config.xmlare bogus; what it really seems to be trying to say is that it failed when it was trying to load the SAML Java class that was specified there. - My setup uses SAML 2.0 for Okta integration, which is important b/c it manages logins and might mean that just launching the service may not be enough.
- Downgrading did not fix the problem. If the upgrade had been the cause, then downgrading with
sudo apt-get install jenkins=2.176.1would've fixed the issue after restarting the service, but it didn't. - Manual plugin installation had failed.
Booting Jenkins
In my case, the main offender was the SAML plugin shown above because it controls my ability to sign in as the admin. Since I'm using SAML 2.0 to integrate auth with Okta, my config.xml file has the following node:
<securityRealm plugin="[email protected]"> [...]
</securityRealm>Removing this section from the file (remember the backup?) causes Jenkins to stop trying to load the SAML-related Java class, which allowed it to boot in my case. However, due to the (now missing) Okta integration, there's no way to sign in after restarting the service.
Accessing Admin Controls + Plugin Upgrades
SECURITY WARNING: This works in my case because I'm behind a corporate firewall and this tool cannot be accessed from outside the company. If your Jenkins instance is accessible to the Internet, do NOT do this and find some other way! Consider temporarily blocking Internet access to your server at a minimum (e.g. disable port-forwarding rules, etc) while you fix the issue. (This assumes you're in the same LAN as the server, so that you don't cut off your own access.)
In my case, auth is a combination between Okta integration (i.e. for access) and role-based auths in Jenkins, stored in the config.xml file. If this sounds like you, you should have something similar to:
<authorizationStrategy> [...] <role name="admin" pattern=".*"> <permissions> <permission>hudson.model.View.Delete</permission> <permission>hudson.model.Computer.Connect</permission> [...] </permissions> <assignedSIDs> <sid>anonymous</sid> [...] </assignedSIDs> </role> [...]
</authorizationStrategy>As shown above, I've temporarily added the <sid>anonymous</sid> entry under the admin role. This will allow you (well, anyone really, hence the security warning above) to temporarily access your Jenkins instance as an Anonymous Administrator.
Restart the Jenkins service after this (e.g. sudo systemctl restart jenkins.service). You can now enter the Manage Jenkins >> Manage Plugins section and force the updates. Check the box to restart Jenkins after the plugins are downloaded to make sure they install.
Cleaning Up
At this point you should restore the original config.xml (e.g. sudo mv config.xml.backup config.xml) and then restart the Jenkins service again. This undoes the changes specified in the previous sections and you should be back to your original configuration, whatever that may've been.
I encountered the very same error on the jenkins web pages instead of the login page. I got that because I wanted to update all plugins at once on my Jenkins version 2.73.3.
And just like you, a system reboot did not help.
So I had a look at the logs and they seemed similar :
oct. 22, 2019 9:41:14 AM jenkins.InitReactorRunner$1 onTaskFailed
GRAVE: Failed Loading plugin Credentials Plugin v2.3.0 (credentials)
java.io.IOException: Credentials Plugin v2.3.0 failed to load. - You must update Jenkins from v2.73.3 to v2.138.4 or later to run this plugin. - Structs Plugin v1.20 failed to load. Fix this plugin first. at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:626)To one small detail: for me the plugins could not update because Jenkins version was too old. That exactly the opposite of the logs in the accepted solution.
So I decided to update jenkins first
sudo apt-get update sudo apt-get install jenkins # Say N to overwrite the config.And I got Jenkins back, but not all plugins : some monitoring jobs were still in errors although the servers were up & running.
So I decided to relaunch the plugins update (in order to be sync with the version of jenkins) in the Administration page, I relaunched the search for updates and took all compatible ones. After this installation and a restart all was finally back to normal.
I thought it was important to add this to complete the accepted solution because you don't have to perform dangerous modification of config.xml.
Such bugs blocking access to Jenkins web console may result from authorization plugin errors an your in Jenkins config file, caused by manual editing or disrupted plugin update.
As a quick and dirty solution, when more delicate ones fail (including backups being too old or incompatible) and only if you ensured that potential hackers won't have access to your unprotected Jenkins master,
deleting Jenkins config.xml
will re-create it with defaults and give you (but also all others!) quick anonymous access to Jenkins web interface. Please don't treat it as a hacking advice.
So the first thing to do after access to Jenkins console is restored, is to set up proper access controls at:
The other setting you may need to restore is node/cloud label(s) that prevent pipeline builds from starting on the slave or master node(s), which can be done at:
In my case it is usually caused by a breaking change in Jenkins container, e.g. a Jenkins downgrade, breaking GitHub Authorization Plugin.
The least destructive solution in my experience is to backup Jenkins config file (config.xml) and remove from it both security-related sections (authorizationStrategy and securityRealm), then restart Jenkins and quickly restore authentication and authorization.
Might be worth checking the umask of /var/lib/jenkins/config.xml
