Zimbra – upgrade ubuntu 18.04 to 20.04 and update zcs-8.8.15_GA_3869 to zcs-8.8.15_GA_4179

Ubuntu 18.04 is going EOL in April 2023. If you are like me and you are running your zimbra email server, it is time to upgrade your instance to Ubuntu 20.04. Note that you will also need to upgrade your zimbra installation from zcs-8.8.15_GA_3869 to zcs-8.8.15_GA_4179.

After several unsuccesfull trials I was able to complete the upgrade and have a working server again, and since this was not done on the first atempt I decided to write this post to help others that will try the same.

The upgrade steps are:

  1. BACKUP! If you are like me and run zimbra on a VM make sure you take a snapshot before you do anything. You might want to go back if things don’t go well.
  2. Gather server information: Run the following and save the output somewhere. You will need it later.
    sudo su
    su - zimbra
    zmlocalconfig -s | grep -i password
  3. Take backup of some important files:
    sudo su
    mkdir -p /opt/{zimbra_backup/.saveconfig,ldap_backup}  
    chown zimbra:zimbra /opt/{zimbra_backup,ldap_backup} -R
    cp -pvr /opt/zimbra/.install_history /opt/zimbra_backup/.saveconfig/.install_history
    cp -pvr /opt/zimbra/conf/localconfig.xml /opt/zimbra_backup/.saveconfig/localconfig.xml
    apt list --installed | grep -i 'zimbra-' | cut -d'/' -f1 > /opt/zimbra_backup/.saveconfig/.installedPackages
    su - zimbra
    /opt/zimbra/libexec/zmslapcat -c /opt/ldap_backup/
    /opt/zimbra/libexec/zmslapcat -a /opt/ldap_backup/
    /opt/zimbra/libexec/zmslapcat /opt/ldap_backup/
  4.  Update the current packages:
    apt-get update && apt-get dist-upgrade
    reboot
  5. Upgrade ubuntu (providing the default answer to each upgrade question is your best bet) – Note that your DNS resolution might have been done by zimbra, and zimbra is now stopped. Make sure you add your DNS server in /etc/resolv.confand after that:
    sudo su
    do-release-upgrade
    reboot
  6.  Download the zimbra tgz from https://www.zimbra.com/downloads/zimbra-collaboration-open-source/ and copy it in a folder on your server.
  7. Untar the zimbra package:
    tar -xvzf zcs-8.8.15_GA_4179.UBUNTU20_64.20211118033954.tgz
  8. Reinstall the zimbra package (Remember to add a DNS server that can resolve your zimbra host to /etc/resolv.conf or the following will fail):
    sudo su
    cd zcs-8.8.15_GA_4179.UBUNTU20_64.20211118033954
    ./install.sh -s --skip-activation-check
  9. Make sure during the install you answer NO to the following question:
    The Zimbra Collaboration Server does not appear to be installed,
    yet there appears to be a ZCS directory structure in /opt/zimbra.
    
    Would you like to delete /opt/zimbra before installing? [N] N
  10. Copy the files you have saved in the begining:
    cp -pvr /opt/zimbra_backup/.saveconfig/.install_history /opt/zimbra/.install_history  
    cp -pvr /opt/zimbra_backup/.saveconfig/localconfig.xml /opt/zimbra/conf/localconfig.xml
  11. Run the zimbra setup:
    sudo su
    /opt/zimbra/libexec/zmsetup.pl
  12. Before the install is complete you need to put back the correct passwords (use the passwords you have saved in point 2 – Told you this was important ;P ):
    ******* +Ldap Admin password:                  Not Verified
    ******* +Ldap postfix password:                Not Verified
    ******* +Ldap amavis password:                 Not Verified
    ******* +Ldap nginx password:                  Not Verified
    ******* +Ldap Bes Searcher password:           Not Verified
    ******* +Bind password for postfix ldap user:  Not Verified
    ******* +Bind password for amavis ldap user:   Not Verified
    ******* +Admin Password                        UNSET
  13. Once you have a clean menu like the following press a to apply:
    Main menu   
       1) Common Configuration:                                                  
       2) zimbra-ldap:                             Enabled                       
       3) zimbra-logger:                           Enabled                       
       4) zimbra-mta:                              Enabled                       
       5) zimbra-dnscache:                         Enabled                       
       6) zimbra-snmp:                             Enabled                       
       7) zimbra-store:                            Enabled                       
       8) zimbra-spell:                            Enabled                       
       9) zimbra-proxy:                            Enabled                       
      10) Default Class of Service Configuration:                                
       s) Save config to file                                                    
       x) Expand menu                                                            
       q) Quit                                    
    
    *** CONFIGURATION COMPLETE - press 'a' to apply
    Select from menu, or press 'a' to apply config (? - help)
  14.  Wait for the process to complete, reboot your server and get ready for the fun part. As zimbra run zmcontrol status and check if all services are running. Chances are you will get:
    Host mail.procentric.tv
    amavis                  Running
    antispam                Running
    antivirus               Running
    dnscache                Running
    ldap                    Running
    logger                  Stopped
    zmlogswatchctl is not running
    mailbox                 Stopped
    zmmailboxdctl is not running.
    memcached               Running
    mta                     Running
    opendkim                Running
    proxy                   Running
    service webapp          Stopped
    zmmailboxdctl is not running.
    snmp                    Running
    spell                   Running
    stats                   Running
    zimbra webapp           Stopped
    zmmailboxdctl is not running.
    zimbraAdmin webapp      Stopped
    zmmailboxdctl is not running.
    zimlet webapp           Stopped
    zmmailboxdctl is not running.
    zmconfigd               Running
  15. To fix this do the following:
    sudo su
    cp -r /opt/zimbra/ssl/zimbra/commercial ~/ssl-backup
    mv /opt/zimbra/mailboxd/etc/keystore /root/keystore.old
    su - zimbra
    /opt/zimbra/bin/zmcertmgr createca -new
    /opt/zimbra/bin/zmcertmgr createcrt -new -days 3650
    /opt/zimbra/bin/zmcertmgr deployca
    /opt/zimbra/bin/zmcertmgr deploycrt self
  16. If you got an error on the last line delete /opt/zimbra/ssl/zimbra/jetty.pkcs12 and start over:
    rm /opt/zimbra/ssl/zimbra/jetty.pkcs12
    /opt/zimbra/bin/zmcertmgr createca -new
    /opt/zimbra/bin/zmcertmgr createcrt -new -days 3650
    /opt/zimbra/bin/zmcertmgr deployca
    /opt/zimbra/bin/zmcertmgr deploycrt self
    exit
    reboot
  17. After the reboot, as zimbra user check what processes are running:
    sudo su
    su - zimbra
    zmcontrol status
  18. You should probably get something like this:
    amavis                  Running
    antispam                Running
    antivirus               Running
    dnscache                Running
    ldap                    Running
    logger                  Stopped
    zmlogswatchctl is not running
    mailbox                 Running
    memcached               Running
    mta                     Running
    opendkim                Running
    proxy                   Running
    service webapp          Running
    snmp                    Running
    spell                   Running
    stats                   Running
    zimbra webapp           Running
    zimbraAdmin webapp      Running
    zimlet webapp           Running
    zmconfigd               Running
  19. as root update and upgrade all packages (This should fix some perl package missmatch):
    apt-get update; apt-get -y upgrade
  20. Reboot and check again the services. They should all be running. If they are not restart from point 14. Also, if you had commercial ssl certificates we need to put them back. For this run as root:
    cp -Rf ~/ssl-backup /tmp
    chown -R zimbra:zimbra /tmp/ssl-backup
    cd /tmp/ssl-backup
    /opt/zimbra/bin/zmcertmgr deploycrt comm commercial.crt commercial_ca.crt
  21.  Reboot and check your services they should all be running. login admin interface on port 7071 and check services also.
  22. Please let me know if you run into problems, and we can maybe figure it out.
Posted in Linux, Uncategorized.

17 Comments

  1. Hi, I’m updating ubuntu 18.04 to 20.04, with Zimbra zcs-8.8.15_GA_3869 installed and updated to GA_4179. Everything ok until the creation of the new certificates, where I get this error: /opt/zimbra/bin/zmcertmgr deploycrt self
    ** Saving config key ‘zimbraSSLCertificate’ via zmprov modifyServer XXXXX.it…failed (rc=1)
    ** Installing imapd certificate ‘/opt/zimbra/conf/imapd.crt’ and key ‘/opt/zimbra/conf/imapd.key’
    ** Copying ‘/opt/zimbra/ssl/zimbra/server/server.crt’ to ‘/opt/zimbra/conf/imapd.crt’
    ** Copying ‘/opt/zimbra/ssl/zimbra/server/server.key’ to ‘/opt/zimbra/conf/imapd.key’
    ERROR: imapd keytool(-delete -alias jetty) returned non-zero(1):
    keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect
    ** Creating file ‘/opt/zimbra/ssl/zimbra/jetty.pkcs12’
    ** Creating keystore ‘/opt/zimbra/conf/imapd.keystore’
    ERROR: com.zimbra.cert.MyPKCS12Import to ‘/opt/zimbra/ssl/zimbra/jetty.pkcs12’ returned non-zero(1):
    Exception in thread “main” java.io.IOException: Keystore was tampered with, or password was incorrect
    at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:813)
    at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:221)
    at java.base/java.security.KeyStore.load(KeyStore.java:1473)
    at com.zimbra.cert.MyPKCS12Import.main(MyPKCS12Import.java:104)
    Caused by: java.security.UnrecoverableKeyException: Password verification failed
    at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:811)
    … 3 more
    I also tried to delete the file /opt/zimbra/ssl/zimbra/jetty.pkcs12, but the problem persists.
    Do you have any suggestions for me?
    Thank you

    • Hi Alessandro.
      I completely forgot to approve and answer your comment. Is this still an issue? Were you able to complete the upgrade?

      Thank you,
      Paul

    • Hi,

      I hat the same problem. For me it helped to set the mailboxd_keystore_password from the zmlocalconfig backup.

      zmlocalconfig -e mailboxd_keystore_password=”PASSWORD”

      After that it was possible to do step 15.

      I upgraded from the latest version (8.8.15 P45). There is also a warning in the wiki regarding OpenJDK. Maybe that was also part of the problem.

      https://wiki.zimbra.com/wiki/Zimbra_Releases/8.8.15/P45

      Regards

  2. I just read your manual for upgrading Ubuntu 18.04 to 20.04 on our Zimbra server. I am currently running zcs-8.8.15_GA_4464 on 18.04? Do your instructions also work on that combination?
    Do I have to install the “8.8.15_GA_4177” after upgrading to 20.04? This is the only downloadable version for 20.04 I can find on the Zimbra website…

  3. Hi Paul,

    Huge thanks for putting these instructions together – it was exactly what I needed and saved me a lot of heartache!
    To (hopefully) answer Alessandro’s question above, I got a similar error and needed to delete /opt/zimbra/conf/imapd.keystore as well as /opt/zimbra/ssl/zimbra/jetty.pkcs12 at step 16.

    Thanks again
    C

    • Ingo, Zimbra 9 is not available for the community only paid. I meant that you will need to update your zimbra 8.x.x to work with the libraries included with Ubuntu 20.04. If you upgrade your OS the old zimbra installation will stop working.

      • Hi Paul!
        I did the Update yesterday on a copy of my live VM. And everything worked as expected. I also did the update of Zimbra with the repositories of apt, but that was also a very easy step with “apt-get upgrade”.
        I am using the paid Network Edition, not the free one. However I also had to delete the file /opt/zimbra/conf/imapd.keystore, that C mentioned above.

        Thanks a lot for your manual!

  4. Hi Paul!

    I have upgrade ubuntu to 20.04 and updated zimbra, but at the end I have this situation:
    Host
    amavis Running
    antispam Running
    antivirus Running
    dnscache Running
    ldap Running
    logger Running
    mailbox Stopped
    zmmailboxdctl is not running.
    memcached Running
    mta Running
    opendkim Running
    proxy Stopped
    proxy is not running.
    service webapp Stopped
    zmmailboxdctl is not running.
    snmp Stopped
    zmswatch is not running.
    spell Running
    stats Running
    zimbra webapp Stopped
    zmmailboxdctl is not running.
    zimbraAdmin webapp Stopped
    zmmailboxdctl is not running.
    zimlet webapp Stopped
    zmmailboxdctl is not running.
    zmconfigd Running

    In particular if I try to start the proxy I have this error:
    Starting proxy…nginx: [emerg] a duplicate default server for 0.0.0.0:443 in /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default:34

    Do you have any tips on how to resolve this issue?

    • Hi Andrea,

      I would look in /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default.

      It looks like you have 2 enteries for default server for 0.0.0.0:443

      I would make a backup of that file and remove everything related to one of the enteries.

      Please let me know if that helps.

      Thank you,
      Paul

      • Hi Paul,
        thanks for your fast reply.

        In fact the file nginx.conf.web.https.default there are two “server” block with listen port 443.

        (this is the file /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default after the upgrade)

        server {
        listen 443 default_server ssl;
        server_name _;

        }
        server
        {
        listen 443 default_server ssl http2; # <– line 34 reported by the error message

        server_name server.name.tld; # add aliases and perhaps public

        }

        Apparently "default_server" is also present in the second "server" block, I've looked into the same files before the upgrade and there "default_server" was not present in the second block:

        (this is the file /opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default before the upgrade)

        server {
        listen 443 default_server ssl;
        server_name _;

        }
        server
        {
        listen 443 ssl http2;

        server_name server.name.tld;

        }

        I've tried to modify it, but it's always regenerated so it keeps to give the error.
        Maybe some configuration in zibra recreate the file?
        How may I change this setting?

        • Andrea,

          can you check if there are differences between before and after upgrade for :
          /opt/zimbra/conf/nginx/templates/nginx.conf.web.http.default.template

          It is just a guess but that is probably the file that is used to create./opt/zimbra/conf/nginx/includes/nginx.conf.web.https.default.

          Can you try to use the file from before the upgrade and see if the server starts?

          Please let me know how it goes.

          Thank you,
          Paul

    • The command

      zmlocalconfig -s | grep -i password

      would give you all the set passwrds.

      Than you can use them in step 12

Leave a Reply

Your email address will not be published. Required fields are marked *