Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-53399

Lost all saved managed files after upgrade to 3.0

    Details

    • Similar Issues:

      Description

      Upgraded Jenkins from 2.140 to 2.141 and in the same time config-file-provider-plugin from 2.18 to 3.0 

      After Jenkins restart, all configured files were gone.

      Jenkins shows warning about unreadable data:

      The first one is generic:

      org.jenkinsci.plugins.configfiles.GlobalConfigFiles Managed files ConversionException: org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1 : org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1 ---- Debugging information ---- message : org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1 cause-exception : com.thoughtworks.xstream.mapper.CannotResolveClassException cause-message : org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1 class : java.util.TreeSet required-type : java.util.TreeSet converter-type : com.thoughtworks.xstream.converters.collections.TreeSetConverter path : /org.jenkinsci.plugins.configfiles.GlobalConfigFiles/configs/comparator line number : 4 -------------------------------, MissingFieldException: No field 'org.jenkinsci.plugins.configfiles.maven.MavenSettingsConfig' found in class 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles', MissingFieldException: No field 'org.jenkinsci.plugins.configfiles.maven.MavenSettingsConfig' found in class 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles', MissingFieldException: No field 'org.jenkinsci.plugins.configfiles.maven.MavenToolchainsConfig' found in class 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles', MissingFieldException: No field 'org.jenkinsci.plugins.configfiles.maven.MavenToolchainsConfig' found in class 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles'

       

      The second one is specific to one Jenkins Job folder, where there should be no such configuration at all.

      com.cloudbees.hudson.plugins.folder.Folder releng+target ConversionException: org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty$1 : org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty$1 ---- Debugging information ---- message : org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty$1 cause-exception : com.thoughtworks.xstream.mapper.CannotResolveClassException cause-message : org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty$1 class : java.util.TreeSet required-type : java.util.TreeSet converter-type : com.thoughtworks.xstream.converters.collections.TreeSetConverter path : /com.cloudbees.hudson.plugins.folder.Folder/properties/org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty/configs/comparator line number : 13 -------------------------------, ConversionException: Could not call org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty.readResolve() : null : Could not call org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty.readResolve() : null ---- Debugging information ---- message : Could not call org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty.readResolve() : null cause-exception : com.thoughtworks.xstream.converters.reflection.ObjectAccessException cause-message : Could not call org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty.readResolve() : null class : org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty required-type : java.util.TreeSet converter-type : hudson.util.RobustReflectionConverter path : /com.cloudbees.hudson.plugins.folder.Folder/properties/org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty/configs line number : 14 -------------------------------

       

      Now I don't know if this version is expected to break all stored provided files (the version number change could suggest it). If it is, then there should be a strong warning at the Jenkins plugin page stating that it is incompatible with the old version.

      Otherwise, please fix the problem  Thanks.

        Attachments

          Issue Links

            Activity

            beyerj Jens Beyer created issue -
            Show
            oleg_nenashev Oleg Nenashev added a comment - Yes, pom.xml refers 2.15 in compatibleSince: https://github.com/jenkinsci/config-file-provider-plugin/blob/master/pom.xml#L275 . CC Dominik Bartholdi Stefan Rempfer
            Hide
            beyerj Jens Beyer added a comment -

            Just for completeness' sake: After downgrading the plugin back to 2.18, the configured managed files are back.

            Show
            beyerj Jens Beyer added a comment - Just for completeness' sake: After downgrading the plugin back to 2.18, the configured managed files are back.
            oleg_nenashev Oleg Nenashev made changes -
            Field Original Value New Value
            Labels regression
            Hide
            mnagni Maurizio Nagni added a comment -

            I confirm Beyer's note

            Show
            mnagni Maurizio Nagni added a comment - I confirm Beyer's note
            Hide
            pwolfart Patrick Wolfart added a comment -

            Hi, I downgraded the plugin to 2.18, but the error remains. 

            Additionally Jenkins complains about not readable data now. 

            Show
            pwolfart Patrick Wolfart added a comment - Hi, I downgraded the plugin to 2.18, but the error remains.  Additionally Jenkins complains about not readable data now. 
            Hide
            jhack Giacomo Boccardo added a comment -

            If you used the "Manage Old Data" functionality and removed the unreadable configuration downgrading is useless. So I kept the latest version and I recreated the configuration using the interface and the data I found in a backup in the org.jenkinsci.plugins.configfiles.GlobalConfigFiles.xml file. Than I had to reselect by hand the new configuration name for each job!

            Moreover I found that NPM configuration's id field is read only. I don't know if it's related to this release, but I surely set it by hand the first time I created the NPM configuration.

            Show
            jhack Giacomo Boccardo added a comment - If you used the "Manage Old Data" functionality and removed the unreadable configuration downgrading is useless. So I kept the latest version and I recreated the configuration using the interface and the data I found in a backup in the  org.jenkinsci.plugins.configfiles.GlobalConfigFiles.xml file. Than I had to reselect by hand the new configuration name for each job! Moreover I found that NPM configuration's id field is read only. I don't know if it's related to this release, but I surely set it by hand the first time I created the NPM configuration.
            Hide
            patope Tomi Pakarinen added a comment - - edited

            I got old configuration to load by changing comparator class

            <comparator class="org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1"/>

            to

            <comparator class="org.jenkinsci.plugins.configfiles.ConfigComparator"/>

             and per project config

            <comparator class="org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty$1"/>

             to

            <comparator class="org.jenkinsci.plugins.configfiles.ConfigComparator"/>

             

            Show
            patope Tomi Pakarinen added a comment - - edited I got old configuration to load by changing comparator class <comparator class="org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1"/> to <comparator class="org.jenkinsci.plugins.configfiles.ConfigComparator"/>  and per project config <comparator class="org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty$1"/>  to <comparator class="org.jenkinsci.plugins.configfiles.ConfigComparator"/>  
            Hide
            oeuftete oeuftete added a comment -

            I can open a new issue, but assuming the same root cause is responsible for the Organization Folder plugin not being able to load its config either.  A downgrade of the Config File Provider plugin back to 2.18 was a successful workaround.

             

            2018-09-04T12:56:53.932168699Z SEVERE: Failed Loading item MyOrg
            2018-09-04T12:56:53.932184399Z java.lang.NullPointerException
            2018-09-04T12:56:53.932195699Z  at jenkins.branch.OrganizationFolder.onLoad(OrganizationFolder.java:199)
            2018-09-04T12:56:53.932207199Z  at hudson.model.Items.load(Items.java:372)
            2018-09-04T12:56:53.932218199Z  at jenkins.model.Jenkins$15.run(Jenkins.java:3125)
            2018-09-04T12:56:53.932229399Z  at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
            2018-09-04T12:56:53.932240499Z  at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
            2018-09-04T12:56:53.932251499Z  at jenkins.model.Jenkins$5.runTask(Jenkins.java:1068)
            2018-09-04T12:56:53.932262499Z  at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
            2018-09-04T12:56:53.932273499Z  at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
            2018-09-04T12:56:53.932284499Z  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            2018-09-04T12:56:53.932295599Z  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            2018-09-04T12:56:53.932306799Z  at java.lang.Thread.run(Thread.java:748)
            
            Show
            oeuftete oeuftete added a comment - I can open a new issue, but assuming the same root cause is responsible for the Organization Folder plugin not being able to load its config either.  A downgrade of the Config File Provider plugin back to 2.18 was a successful workaround.   2018-09-04T12:56:53.932168699Z SEVERE: Failed Loading item MyOrg 2018-09-04T12:56:53.932184399Z java.lang.NullPointerException 2018-09-04T12:56:53.932195699Z at jenkins.branch.OrganizationFolder.onLoad(OrganizationFolder.java:199) 2018-09-04T12:56:53.932207199Z at hudson.model.Items.load(Items.java:372) 2018-09-04T12:56:53.932218199Z at jenkins.model.Jenkins$15.run(Jenkins.java:3125) 2018-09-04T12:56:53.932229399Z at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) 2018-09-04T12:56:53.932240499Z at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296) 2018-09-04T12:56:53.932251499Z at jenkins.model.Jenkins$5.runTask(Jenkins.java:1068) 2018-09-04T12:56:53.932262499Z at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214) 2018-09-04T12:56:53.932273499Z at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) 2018-09-04T12:56:53.932284499Z at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 2018-09-04T12:56:53.932295599Z at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 2018-09-04T12:56:53.932306799Z at java.lang.Thread.run(Thread.java:748)
            Hide
            bizdelnick Dmitry Mikhirev added a comment -

            Same for me. On Jenkins 2.121.3 after upgrading the config-file-provider-plugin from 2.18 to 3.0 folders' configurations are lost. Downgrade to 2.18 solves the problem.

            Show
            bizdelnick Dmitry Mikhirev added a comment - Same for me. On Jenkins 2.121.3 after upgrading the config-file-provider-plugin from 2.18 to 3.0 folders' configurations are lost. Downgrade to 2.18 solves the problem.
            abayer Andrew Bayer made changes -
            Priority Critical [ 2 ] Blocker [ 1 ]
            Hide
            abayer Andrew Bayer added a comment -

            I've got at least one report of the upgrade breaking shared libraries and per-project permissions on folders as well - Daniel Beck, I think we should be seriously considering removing 3.0 from the update center.

            Show
            abayer Andrew Bayer added a comment - I've got at least one report of the upgrade breaking shared libraries and per-project permissions on folders as well - Daniel Beck , I think we should be seriously considering removing 3.0 from the update center.
            Hide
            scottschreckengaust Scott Schreckengaust added a comment -

            I second Jens Beyer's comment for a compatibility warning prior to upgrade.  How to upgrade correctly (Tomi Pakarinen's comment)?

            Show
            scottschreckengaust Scott Schreckengaust added a comment - I second Jens Beyer 's comment for a compatibility warning prior to upgrade.  How to upgrade correctly ( Tomi Pakarinen 's comment)?
            Hide
            abayer Andrew Bayer added a comment -

            Opened https://github.com/jenkins-infra/update-center2/pull/236 to remove 3.0 from the update center.

            Show
            abayer Andrew Bayer added a comment - Opened https://github.com/jenkins-infra/update-center2/pull/236 to remove 3.0 from the update center.
            Hide
            patope Tomi Pakarinen added a comment -

            Why comparator is serialized into config file? It is static constant anyway.. 

             

            https://github.com/jenkinsci/config-file-provider-plugin/commit/2366eaee6980564e59771897274f38d462bb0abc

            Can it be made transient?

            Show
            patope Tomi Pakarinen added a comment - Why comparator is serialized into config file? It is static constant anyway..    https://github.com/jenkinsci/config-file-provider-plugin/commit/2366eaee6980564e59771897274f38d462bb0abc Can it be made transient?
            Hide
            hsantana29 Hector Santana added a comment -
            org.jenkinsci.plugins.configfiles.GlobalConfigFiles
            
            Managed files
            
            ConversionException: org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1 : org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1 ---- Debugging information ---- message : org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1 cause-exception : com.thoughtworks.xstream.mapper.CannotResolveClassException cause-message : org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1 class : java.util.TreeSet required-type : java.util.TreeSet converter-type : com.thoughtworks.xstream.converters.collections.TreeSetConverter path : /org.jenkinsci.plugins.configfiles.GlobalConfigFiles/configs/comparator line number : 4 -------------------------------, MissingFieldException: No field 'org.jenkinsci.plugins.configfiles.xml.XmlConfig' found in class 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles', MissingFieldException: No field 'org.jenkinsci.plugins.managedscripts.PowerShellConfig' found in class 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles', MissingFieldException: No field 'org.jenkinsci.plugins.managedscripts.PowerShellConfig' found in class 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles', MissingFieldException: No field 'org.jenkinsci.plugins.managedscripts.PowerShellConfig' found in class 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles'
            

            We also had the same problem when updated from 2.18 to 3.0

            Show
            hsantana29 Hector Santana added a comment - org.jenkinsci.plugins.configfiles.GlobalConfigFiles Managed files ConversionException: org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1 : org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1 ---- Debugging information ---- message : org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1 cause-exception : com.thoughtworks.xstream.mapper.CannotResolveClassException cause-message : org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1 class : java.util.TreeSet required-type : java.util.TreeSet converter-type : com.thoughtworks.xstream.converters.collections.TreeSetConverter path : /org.jenkinsci.plugins.configfiles.GlobalConfigFiles/configs/comparator line number : 4 -------------------------------, MissingFieldException: No field 'org.jenkinsci.plugins.configfiles.xml.XmlConfig' found in class 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles' , MissingFieldException: No field 'org.jenkinsci.plugins.managedscripts.PowerShellConfig' found in class 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles' , MissingFieldException: No field 'org.jenkinsci.plugins.managedscripts.PowerShellConfig' found in class 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles' , MissingFieldException: No field 'org.jenkinsci.plugins.managedscripts.PowerShellConfig' found in class 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles' We also had the same problem when updated from 2.18 to 3.0
            Hide
            imod Dominik Bartholdi added a comment - - edited

            hmm, oh damn seems the easiest looking change of all broke the whole thing... and yes sure the idea never was to have the comparator beeing serialized into the config... 

            anyway, Andrew Bayer and Oleg Nenashev many thanks for the quick reaction, I will need to check how I can fix this without lifting the 'compatibleSince'

            sorry to everyone I caused issues with this!!!!

            Show
            imod Dominik Bartholdi added a comment - - edited hmm, oh damn seems the easiest looking change of all broke the whole thing... and yes sure the idea never was to have the comparator beeing serialized into the config...  anyway, Andrew Bayer and Oleg Nenashev many thanks for the quick reaction, I will need to check how I can fix this without lifting the 'compatibleSince' sorry to everyone I caused issues with this!!!!
            Hide
            abayer Andrew Bayer added a comment -

            FYI, 3.0 has been removed from the update center. I've had multiple reports that downgrading config-files-provider to 2.18 fixes all symptoms those reports encountered (including folder permissions and shared libraries being broken), so that's a clear path forward for now.

            Show
            abayer Andrew Bayer added a comment - FYI, 3.0 has been removed from the update center. I've had multiple reports that downgrading config-files-provider to 2.18 fixes all symptoms those reports encountered (including folder permissions and shared libraries being broken), so that's a clear path forward for now.
            Hide
            jhack Giacomo Boccardo added a comment -

            I've installed 3.0 and reconfigured everything.

            Please, consider that the next update should not break all the configurations which are still using 3.0.

            Show
            jhack Giacomo Boccardo added a comment - I've installed 3.0 and reconfigured everything. Please, consider that the next update should not break all the configurations which are still using 3.0.
            Hide
            srempfer Stefan Rempfer added a comment -
            Show
            srempfer Stefan Rempfer added a comment - Dominik Bartholdi : May this could help you https://wiki.jenkins.io/display/JENKINS/Serialization+of+anonymous+classes (see chapter Usages in XStream)
            abayer Andrew Bayer made changes -
            Link This issue is duplicated by JENKINS-53423 [ JENKINS-53423 ]
            Hide
            pavenova Pavel Novak added a comment -

            Hi, all, just for being in touch, 
            so its the problem of config provider? 

            I guess the issue is not possible to revert, right? 

            We reverted from backups about 1TB of data (jenk. home, jobs and plugins folders) and then it was for sure back in the origin state, we did not found any other option :/

             

            Show
            pavenova Pavel Novak added a comment - Hi, all, just for being in touch,  so its the problem of config provider?  I guess the issue is not possible to revert, right?  We reverted from backups about 1TB of data (jenk. home, jobs and plugins folders) and then it was for sure back in the origin state, we did not found any other option :/  
            Hide
            imod Dominik Bartholdi added a comment -

            Giacomo Boccardo Stefan Rempfer not sure this will be possible - the problem is that the serialized field is not part of my code, but a member of TreeMap -> java.util.TreeMap#comparator

            Oleg Nenashev do you have any idea how this could be solved? I currently only see the possibility to lift 'compatibleSince' or roll back and therefore screw the users who upgraded to 3.0 again...

            Show
            imod Dominik Bartholdi added a comment - Giacomo Boccardo Stefan Rempfer not sure this will be possible - the problem is that the serialized field is not part of my code, but a member of TreeMap -> java.util.TreeMap#comparator Oleg Nenashev do you have any idea how this could be solved? I currently only see the possibility to lift 'compatibleSince' or roll back and therefore screw the users who upgraded to 3.0 again...
            Hide
            pavenova Pavel Novak added a comment -

            Dominik Bartholdi 

            Hi, maybe override the default comparator, and implement custom one will be kind of workaround, how to fix the issue? 

            look eg. at following link:

            http://www.java2s.com/Tutorials/Java/Collection_How_to/Map/Create_custom_Comparator_for_TreeMap.htm 

             

            And that I understood it well, there is a bug in java implementation ? 
            Then, the issue should be reported into Java provider  

            Btw I can remember I was facing similar issue in the past, there was missing implementation of some method in OpenJDK, which was available in Oracle JDK, what I can remember it was also some kind of comparator, but on hashmap or something like these classes., or another abstract map.

             

            Giacomo Boccardo, Oleg Nenashev FYI

            Show
            pavenova Pavel Novak added a comment - Dominik Bartholdi   Hi, maybe override the default comparator, and implement custom one will be kind of workaround, how to fix the issue?  look eg. at following link: http://www.java2s.com/Tutorials/Java/Collection_How_to/Map/Create_custom_Comparator_for_TreeMap.htm     And that I understood it well, there is a bug in java implementation ?  Then, the issue should be reported into Java provider   Btw I can remember I was facing similar issue in the past, there was missing implementation of some method in OpenJDK, which was available in Oracle JDK, what I can remember it was also some kind of comparator, but on hashmap or something like these classes., or another abstract map.   Giacomo Boccardo , Oleg Nenashev FYI
            Hide
            imod Dominik Bartholdi added a comment -

            thanks Pavel Novak its not a bug, but my code uses a java.util.TreeSet, which internally uses a java.util.TreeMap which in turn contains the comparator as a member field

            Show
            imod Dominik Bartholdi added a comment - thanks Pavel Novak its not a bug, but my code uses a java.util.TreeSet, which internally uses a java.util.TreeMap which in turn contains the comparator as a member field
            Hide
            pavenova Pavel Novak added a comment -

            Dominik Bartholdi

            ok, thanks for clarification, then simply ignore the comment  

            Godo luck, with that 

            Show
            pavenova Pavel Novak added a comment - Dominik Bartholdi ok, thanks for clarification, then simply ignore the comment   Godo luck, with that 
            Hide
            srempfer Stefan Rempfer added a comment -

            I created a test to ensure that the descriptor data could be read. See https://github.com/srempfer/config-file-provider-plugin/commit/2ffe8695f734b48b695e4bd44c35de33e0c24919
            If someone else will contribute there anonymized 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles.xml' too, we will have a better test base to prevent breaking thinks in further versions.
            Don't forget information about the Jenkins and plugin version.

            Show
            srempfer Stefan Rempfer added a comment - I created a test to ensure that the descriptor data could be read. See https://github.com/srempfer/config-file-provider-plugin/commit/2ffe8695f734b48b695e4bd44c35de33e0c24919 If someone else will contribute there anonymized 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles.xml' too, we will have a better test base to prevent breaking thinks in further versions. Don't forget information about the Jenkins and plugin version.
            Hide
            poundkeys chad ruppert added a comment -

            Can you at least tell us WHERE these files are stored, so I can try to recover the scripts from a partial backup and enter them in the new system?

            We have a ton of builds on this machine, some don't use the scripted stuff. rolling back jenkins would be... awful.

            Show
            poundkeys chad ruppert added a comment - Can you at least tell us WHERE these files are stored, so I can try to recover the scripts from a partial backup and enter them in the new system? We have a ton of builds on this machine, some don't use the scripted stuff. rolling back jenkins would be... awful.
            Hide
            abayer Andrew Bayer added a comment -

            chad ruppert - if you didn't wipe the "old data" from "Manage Jenkins", you should be able to just roll back to version 2.18.

            Show
            abayer Andrew Bayer added a comment - chad ruppert - if you didn't wipe the "old data" from "Manage Jenkins", you should be able to just roll back to version 2.18.
            poundkeys chad ruppert made changes -
            Comment [ Just to be clear, you just broke all my production builds and there's no recovery except for a backup restore?


            you have got to be kidding me? ]
            Hide
            poundkeys chad ruppert added a comment - - edited

            Andrew, Rolling back sucks too. I found the old configs in 

            Jenkins/config-history/org.jenkinsci.plugins.configfiles.GlobalConfigFiles/<dates>

            Thankfully the entry system allows us to specify the ID, so that's wonderful. I was able to strip them out of the xml and make it work, i think. I'll be finding out shortly.

             

            Edit: Yup, worked a treat! 

            Show
            poundkeys chad ruppert added a comment - - edited Andrew, Rolling back sucks too. I found the old configs in  Jenkins/config-history/org.jenkinsci.plugins.configfiles.GlobalConfigFiles/<dates> Thankfully the entry system allows us to specify the ID, so that's wonderful. I was able to strip them out of the xml and make it work, i think. I'll be finding out shortly.   Edit: Yup, worked a treat! 
            Hide
            beyerj Jens Beyer added a comment -

            chad ruppert Sorry, but why does rolling back suck? It is easy, quick and foolproof (thanks @Jenkins team).

            This is basically the first thing I usually try before even thinking about looking into our backups. Upgrade, something looks fishy, do not overwrite any configuration, especially do not delete "unreadable data" from Jenkins. Try to keep the executors offline, try to isolate the culprit by rolling back upgraded versions one after another or by divide and conquer - whatever fits your style and the situation best.

            Of course, having a backup keeps blood pressure down while being at it.

            And having a test machine to test upgrades before going into production would be the best, but I know from own experience that this is a wish not easily fulfilled, especially if Ops or bosses say "you have a backup if something goes wrong".

            @all who come here now because of this issue (you shouldn't anymore because the 3.0 version has been pulled back) and see this big list of comments and feel lost - here is your way out: do not delete "unreadable data" and simply revert the config-file-provider-plugin back from version 3.0 to 2.18, then restart Jenkins, and everything is back to normal.

            Show
            beyerj Jens Beyer added a comment - chad ruppert Sorry, but why does rolling back suck? It is easy, quick and foolproof (thanks @Jenkins team). This is basically the first thing I usually try before even thinking about looking into our backups. Upgrade, something looks fishy, do not overwrite any configuration, especially do not delete "unreadable data" from Jenkins. Try to keep the executors offline, try to isolate the culprit by rolling back upgraded versions one after another or by divide and conquer - whatever fits your style and the situation best. Of course, having a backup keeps blood pressure down while being at it. And having a test machine to test upgrades before going into production would be the best, but I know from own experience that this is a wish not easily fulfilled, especially if Ops or bosses say "you have a backup if something goes wrong". @all who come here now because of this issue (you shouldn't anymore because the 3.0 version has been pulled back) and see this big list of comments and feel lost - here is your way out: do not delete "unreadable data" and simply revert the config-file-provider-plugin back from version 3.0 to 2.18, then restart Jenkins, and everything is back to normal.
            Hide
            poundkeys chad ruppert added a comment -

            That's the thing. I didn't see Jenkins warn me about the unreadable data. Either I had to restart the service manually after a not great upgrade, or I just flat out missed it.

            A lot of it is probably bad practice by upgrading a ton of plugins at the same time, as well as a jenkins upgrade right before.

            This is, I think, the first time I've had a real issue with an upgrade. Backups help a lot, and that's where I was able to pull the config from. After looking through my backups I found the same folder on the build server.  It's been a week- we were caught up in that South Central Azure outage and a few other things, so the BP was already high.

            In the future I may consider a rollback of Jenkins. I always get worried about that failing, simply because its another feature I haven't tried. Not a fulltime Ops engineer by any stretch of the word. Sorry for the initial panic.

            Show
            poundkeys chad ruppert added a comment - That's the thing. I didn't see Jenkins warn me about the unreadable data. Either I had to restart the service manually after a not great upgrade, or I just flat out missed it. A lot of it is probably bad practice by upgrading a ton of plugins at the same time, as well as a jenkins upgrade right before. This is, I think, the first time I've had a real issue with an upgrade. Backups help a lot, and that's where I was able to pull the config from. After looking through my backups I found the same folder on the build server.  It's been a week- we were caught up in that South Central Azure outage and a few other things, so the BP was already high. In the future I may consider a rollback of Jenkins. I always get worried about that failing, simply because its another feature I haven't tried. Not a fulltime Ops engineer by any stretch of the word. Sorry for the initial panic.
            Hide
            imod Dominik Bartholdi added a comment -

            given the mess I created... I rolled back the culprit change which caused this issue and will release a version 3.1 - 3.1 will then be compatible with 2.18 again. Sorry to everyone who has 3.0 installed and needs to change there configs back to what it was in 2.18.

            basically this means in the file 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles.xml' (only one file) you have to change

            <comparator class="org.jenkinsci.plugins.configfiles.ConfigComparator"/> back to: <comparator class="org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1"/>

            and in the folder config.xml's (potentially multiple files) you have to change 

            <comparator class="org.jenkinsci.plugins.configfiles.ConfigComparator"/> back to: <comparator class="org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty$1"/>

            Show
            imod Dominik Bartholdi added a comment - given the mess I created... I rolled back the culprit change which caused this issue and will release a version 3.1 - 3.1 will then be compatible with 2.18 again. Sorry to everyone who has 3.0 installed and needs to change there configs back to what it was in 2.18. basically this means in the file 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles.xml' (only one file) you have to change <comparator class="org.jenkinsci.plugins.configfiles.ConfigComparator"/> back to: <comparator class="org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1"/> and in the folder config.xml's (potentially multiple files) you have to change  <comparator class="org.jenkinsci.plugins.configfiles.ConfigComparator"/>  back to: <comparator class="org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty$1"/>
            Hide
            imod Dominik Bartholdi added a comment -

            just released 3.1 - please let me know if you find its not working out for you

             

            ...and sorry again!!!!

            Show
            imod Dominik Bartholdi added a comment - just released 3.1 - please let me know if you find its not working out for you   ...and sorry again!!!!
            Hide
            srempfer Stefan Rempfer added a comment -

            Updated from 2.18 to 3.1 without problems...

            Show
            srempfer Stefan Rempfer added a comment - Updated from 2.18 to 3.1 without problems...
            Hide
            jhack Giacomo Boccardo added a comment -

            Is it safe to update from 3.0 to 3.1? What's the exact procedure to avoid another disaster?

            Update and change 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles.xml' or viceversa?

            Show
            jhack Giacomo Boccardo added a comment - Is it safe to update from 3.0 to 3.1? What's the exact procedure to avoid another disaster? Update and change 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles.xml' or viceversa?
            Show
            imod Dominik Bartholdi added a comment - Giacomo Boccardo please see my comment above...  https://issues.jenkins-ci.org/browse/JENKINS-53399?focusedCommentId=348653&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-348653
            Hide
            jhack Giacomo Boccardo added a comment -

            Dominik Bartholdi I read it, so I asked those questions.

            Show
            jhack Giacomo Boccardo added a comment - Dominik Bartholdi I read it, so I asked those questions.
            Hide
            imod Dominik Bartholdi added a comment -

            if you have 3.0 installed (and it is working), then do this:

            upgrade to 3.1 and in the file 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles.xml' (only one file) you have to change

            <comparator class="org.jenkinsci.plugins.configfiles.ConfigComparator"/> back to: <comparator class="org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1"/>

            and in the folder config.xml's (potentially multiple files) you have to change 

            <comparator class="org.jenkinsci.plugins.configfiles.ConfigComparator"/> back to: <comparator class="org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty$1"/>

            Show
            imod Dominik Bartholdi added a comment - if you have 3.0 installed (and it is working), then do this: upgrade to 3.1 and in the file 'org.jenkinsci.plugins.configfiles.GlobalConfigFiles.xml' (only one file) you have to change <comparator class="org.jenkinsci.plugins.configfiles.ConfigComparator"/>  back to: <comparator  class="org.jenkinsci.plugins.configfiles.GlobalConfigFiles$1"/> and in the folder config.xml's (potentially multiple files) you have to change  <comparator class="org.jenkinsci.plugins.configfiles.ConfigComparator"/>  back to: <comparator  class="org.jenkinsci.plugins.configfiles.folder.FolderConfigFileProperty$1"/>
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            It would be possible to retain the class introduced in 3.0 and then do some migration to it via readResolve() on the Tree level (replace the Tree instance when pre-3.0 configuration is being read). It would keep pre-3.0 instances compatible and allow migrating to the new data structure as it was intended.

             

             

            Show
            oleg_nenashev Oleg Nenashev added a comment - It would be possible to retain the class introduced in 3.0 and then do some migration to it via readResolve() on the Tree level (replace the Tree instance when pre-3.0 configuration is being read). It would keep pre-3.0 instances compatible and allow migrating to the new data structure as it was intended.    
            Hide
            imod Dominik Bartholdi added a comment -

            thanks Oleg, I can take look at this in the future, but for now this is released again with the 2.18 implementation

            Show
            imod Dominik Bartholdi added a comment - thanks Oleg, I can take look at this in the future, but for now this is released again with the 2.18 implementation
            Hide
            cyprien Cyprien Devillez added a comment -

            3.1 fix the issue for me, thanks!

            Show
            cyprien Cyprien Devillez added a comment - 3.1 fix the issue for me, thanks!
            Hide
            beyerj Jens Beyer added a comment -

            Upgraded from 2.18 to 3.1, had no issues so far. No unreadable data reported. Builds running. Thanks for your quick work.

            What's the workflow now, do I have to set the status of the issue to something or are you devs doing it?

            Show
            beyerj Jens Beyer added a comment - Upgraded from 2.18 to 3.1, had no issues so far. No unreadable data reported. Builds running. Thanks for your quick work. What's the workflow now, do I have to set the status of the issue to something or are you devs doing it?
            Hide
            imod Dominik Bartholdi added a comment - - edited

            thanks Jens Beyer - i'll resolve it... fixed with 3.1

            Show
            imod Dominik Bartholdi added a comment - - edited thanks Jens Beyer - i'll resolve it... fixed with 3.1
            imod Dominik Bartholdi made changes -
            Status Open [ 1 ] Resolved [ 5 ]
            Resolution Fixed [ 1 ]
            renescheibe René Scheibe made changes -
            Remote Link This issue links to "Revert commit #fab44849a2b944cece2e79704eed9f523a65a648 (Web Link)" [ 23520 ]
            renescheibe René Scheibe made changes -
            Remote Link This issue links to "PR#69 (Web Link)" [ 23526 ]
            Hide
            renescheibe René Scheibe added a comment -

            With pull-request #69 the anonymous class serialization warnings are now fixed while maintaining XStream backward compatibility.

            Show
            renescheibe René Scheibe added a comment - With pull-request #69 the anonymous class serialization warnings are now fixed while maintaining XStream backward compatibility.

              People

              • Assignee:
                domi Dominik Bartholdi
                Reporter:
                beyerj Jens Beyer
              • Votes:
                8 Vote for this issue
                Watchers:
                23 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: