Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-52357

After upgrade 2.100 --> 2.129, publish-over-SSH config disappeared

    Details

    • Similar Issues:

      Description

      My Jenkins implementation uses 'publish-over-ssh' plugin to do various simple tasks on hosts that are deployment targets, such as stop/start the app server, clean out log files etc. The configurations for that plugin were merely sets of UNIX commands such as :

      service foo start

      or 

      cd log_dir
      rm -rf *

      My Jenkins worked fine until I was to upgrade from v2.100 to v2.101. When I upgraded to 2.101, I encountered that the upgrade broke a lot of jobs, in that when I would go to the job config, I would get error messages (can't remember which type of jobs). I reverted to 2.100 and they all worked again. So, for a couple of months, I was scared to upgrade Jenkins because I thought your software upgrades were breaking my implementation, basically by not being reverse compatible and no longer supporting configurations created under previous versions (how else to describe my situation?).

      I finally decided to bite the bullet and upgrade from 2.100 to 2.129 hoping you fixed the previous incompatibility. However, I turned out that all the jobs that were utilizing the publish-over-ssh plugin lost their remote command scripts under "Exec command" field and that the SSH Server name defaults to the first SSH server I have defined in my Jenkins configuration, of which there are more than 50. So essentially, I upgraded to 2.129 and a significant portion of the jobs lost their configuration.

      Not only did the configuration disappear in the front end but I dug into the JENKINS_HOME directory substructure where the job configs are in XML files (config.xml) and it appears that the upgrade wiped out the SSH commands there as well as though they never had existed. I can luckily find them under logs but not in the same format as they were in the job (so there is some hope of recovery). It looks like the upgrade changed config.xml to remove any trace of the SSH commands and did not back up the old version that does have the commands, at least I can't find any version history.

      When I went in the problem jobs to re-enter the lost config, upon saving I got the attached error message, which looks like Java stack trace. That problem went away once I upgraded the SSH publish plugin from 1.17 to 1.19 but the configs were still gone.

      This was a sizable disaster for us and a huge disappointment in Jenkins. I urge you to fix this issue ASAP.

        Attachments

          Issue Links

            Activity

            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            > This was a sizable disaster for us and a huge disappointment in Jenkins. I urge you to fix this issue ASAP.
             
            Maybe you want to read the https://jenkins.io/redirect/class-filter/ page as suggested in the error message. Note that this issue is described in upgrade guidelines and weekly/LTS changelogs
             

            Show
            oleg_nenashev Oleg Nenashev added a comment - > This was a sizable disaster for us and a huge disappointment in Jenkins. I urge you to fix this issue ASAP.   Maybe you want to read the https://jenkins.io/redirect/class-filter/  page as suggested in the error message. Note that this issue is described in upgrade guidelines and weekly/LTS changelogs   The page references https://wiki.jenkins.io/display/JENKINS/Plugins+affected+by+fix+for+JEP-200 There is a number of Publish Over plugins referenced in the beginning of the page The issue points to JENKINS-49124  which has exactly the same stacktrace as you reported The issue is resolved in newer plugin versions   I highly  recommend to read  https://jenkins.io/redirect/class-filter/   and follow the upgrade guidelines there.     
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            Resolving since it looks like a full duplicate of JENKINS-49124 .

            If the upgrade of plugins does not resolve the issue, please reopen the ticket

            Show
            oleg_nenashev Oleg Nenashev added a comment - Resolving since it looks like a full duplicate of  JENKINS-49124  . If the upgrade of plugins does not resolve the issue, please reopen the ticket
            Hide
            creekwise Asad Makarevic added a comment - - edited

            I upgraded the plugin but that still doesn't change the fact that the configuration for that plugin disappeared when I upgraded Jenkins – not when I upgraded the plugin. You may argue that is because the condition to have the plugin upgraded when Jenkins was being upgraded wasn't met – however, it is a Jenkins problem because the event that wiped out the configs was the Jenkins upgrade.

            The issue was only partly resolved by the plugin upgrade – It enabled re-entry of the SSH configs, however, it did not restore the configs that were deleted when Jenkins was upgraded.

            Please do not regard this as a plugin/user problem and not a Jenkins problem. The user should not have to be responsible to make sure all of their plugins are up to date when they upgrade Jenkins. You should have a mechanism to not do damage in case of this use case scenario (outdated plugins).

            Show
            creekwise Asad Makarevic added a comment - - edited I upgraded the plugin but that still doesn't change the fact that the configuration for that plugin disappeared when I upgraded Jenkins – not when I upgraded the plugin . You may argue that is because the condition to have the plugin upgraded when Jenkins was being upgraded wasn't met – however, it is a Jenkins problem because the event that wiped out the configs was the Jenkins upgrade. The issue was only partly resolved by the plugin upgrade – It enabled re-entry of the SSH configs, however, it did not restore the configs that were deleted when Jenkins was upgraded. Please do not regard this as a plugin/user problem and not a Jenkins problem. The user should not have to be responsible to make sure all of their plugins are up to date when they upgrade Jenkins. You should have a mechanism to not do damage in case of this use case scenario (outdated plugins).
            Hide
            creekwise Asad Makarevic added a comment -

            I understand your "recommendation" but it is not realistic to expect the user community to read fine print disclaimers before performing potentially dangerous actions – if that were the industry standard, we would spend entire work days reading disclaimers. Disclaimers are not a substitute for carefully designed software that assumes stupidity on the part of the user (I am sorry but I have a thousand other things to focus on and things like managing Jenkins need not be usurping mental faculties, i.e. needs to be so soft around the edge as to be impossible to make damage unless some documentation of the size of Anna Karenina was read).

            So yeah – this is not a user carelessness problem, it is not a plugin problem, it is a Jenkins problem that you all need to fix by making your app provide reasonable safety nets for user and plugin stupidity.

            Show
            creekwise Asad Makarevic added a comment - I understand your "recommendation" but it is not realistic to expect the user community to read fine print disclaimers before performing potentially dangerous actions – if that were the industry standard, we would spend entire work days reading disclaimers. Disclaimers are not a substitute for carefully designed software that assumes stupidity on the part of the user (I am sorry but I have a thousand other things to focus on and things like managing Jenkins need not be usurping mental faculties, i.e. needs to be so soft around the edge as to be impossible to make damage unless some documentation of the size of Anna Karenina was read). So yeah – this is not a user carelessness problem, it is not a plugin problem, it is a Jenkins problem that you all need to fix by making your app provide reasonable safety nets for user and plugin stupidity.
            Hide
            oleg_nenashev Oleg Nenashev added a comment -

            > The issue was only partly resolved by the plugin upgrade – It enabled re-entry of the SSH configs, however, it did not restore the configs that were deleted when Jenkins was upgraded.

            This is strange. The dismissed configs should land in Old Data monitor.

             

            > So yeah – this is not a user carelessness problem, it is not a plugin problem, it is a Jenkins problem that you all need to fix by making your app provide reasonable safety nets for user and plugin stupidity.

            Here I agree with you. We used all available channels as a part of this story, including blogposts, mailing lists and upgrade guidelines. With 100 affected plugins, we have got less than 200 issues reported, so my assumption is that we pretty effectively reached out to users. But we can do better for sure.

            If you have any particular proposals, please put them in the retrospective document: https://docs.google.com/document/d/1KCCIxWh-c44GJbW_AwKooOd7wD4vthWp_KC0r2OJQl0/edit

             

             

             

             

             

            Show
            oleg_nenashev Oleg Nenashev added a comment - > The issue was only partly resolved by the plugin upgrade – It enabled re-entry of the SSH configs, however, it did not restore the configs that were deleted when Jenkins was upgraded. This is strange. The dismissed configs should land in Old Data monitor.   > So yeah – this is not a user carelessness problem, it is not a plugin problem, it is a Jenkins problem that you all need to fix by making your app provide reasonable safety nets for user and plugin stupidity. Here I agree with you. We used all available channels as a part of this story, including blogposts, mailing lists and upgrade guidelines. With 100 affected plugins, we have got less than 200 issues reported, so my assumption is that we pretty effectively reached out to users. But we can do better for sure. If you have any particular proposals, please put them in the retrospective document: https://docs.google.com/document/d/1KCCIxWh-c44GJbW_AwKooOd7wD4vthWp_KC0r2OJQl0/edit          
            Hide
            creekwise Asad Makarevic added a comment -

            > The dismissed configs should land in Old Data monitor.

             

            Can I find them somewhere under my $JENKINS_HOME directory on the server filesystem where the Jenkins runs?

            Show
            creekwise Asad Makarevic added a comment - > The dismissed configs should land in Old Data monitor.   Can I find them somewhere under my $JENKINS_HOME directory on the server filesystem where the Jenkins runs?

              People

              • Assignee:
                Unassigned
                Reporter:
                creekwise Asad Makarevic
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: