Ooops – Not A Smooth Rollout For Registry Support

Well, starting a new feature announcement with an apology and instructions to work around a bug is not the way this was planned. But as the saying goes in German

Erstens kommt es anders und zweitens als man denkt.

Loosely translated into: Murphy’s law strikes again.

With this out of the way let’s start at the beginning. SUSE Manager 5.0 will be in a container and as such updates will be delivered from a container registry rather than a package repository. For BYOS users of the upcoming SUMa 5.0 images the upstream registry will be registry.suse.com and the registration code provides access to the SUMa 5.0 container. PAYG users on the other hand have no SUSE registration code and updated containers need to be delivered in a different way. Logically this is the SUSE update infrastructure that already delivers repositories for all the PAYG listings of SUSE products. Therefore container registry support was added to the SUSE update infrastructure.

Announcement

The SUSE update infrastructure in AWS, Azure and GCE now supports a container registry that can be accessed by PAYG instances or BYOS instances that use the update infrastructure as proxy to SCC.

Back to the container registry and our Ooops

The registry that is part of the update infrastructure is not a general registry, i.e. you cannot push your own container images to this registry. This registry is setup to distribute container images that SUSE publishes and that require authenticated access. All container images that SUSE publishes that do not require authentication will continue to be delivered by registry.suse.com

Great news and so far so good. Unfortunately a bug slipped through multiple gates and worse we published images that have the buggy code. Unfortunately, due to an image refresh necessitated to fix a security bug the number of images affected is rather large. The list of affected images is at the end of this blog.

Getting to the heart of the issue

If you do not have cloud-regionsrv-client-10.3.0-150300.13.3.1 or cloud-regionsrv-client-10.3.0-150500.2.1 on your instance you have nothing to worry about and you do not need to read the rest of this article.

Two package versions same bug, yes, very bad. The cloud-regionsrv-client-10.3.0-150300.13.3.1 version of the package would have arrived on your instance via an update, i.e. zypper up, while the cloud-regionsrv-client-10.3.0-150500.2.1 package version is only in your instance if you started an instance from one of the images listed below. Both packages deliver the same code and that code breaks one of the docker configuration files. If you do not run docker on your instances you also do not need to read the rest of this article.

If you start a new PAYG instance the running system will automatically get registered to the update infrastructure. The new client code also automatically configures the registry setup for use with podman and docker. You can check this via a simple grep command

# grep susecloud /etc/docker/daemon.json

If the file is not there or the grep returns empty the bug does not effect your instance and you can skip the rest of this blog.

In this iteration of the code the registry will not be set up automatically upon reboot. This feature will be part of the next release of cloud-regionsrv-client when we will support LTSS (Long Term Service-Pack Support) as an add-on to PAYG.

Well you are still reading and as such we messed up your system, sorry again. The easiest and most straight forward fix is the following sed expression, which needs to be run as root

On Azure

sed -i 's/registry-azure/https:\/\/registry-azure/' /etc/docker/daemon.json

On EC2

sed -i 's/registry-ec2/https:\/\/registry-ec2/' /etc/docker/daemon.json

On GCE

sed -i 's/registry-gce/https:\/\/registry-gce/' /etc/docker/daemon.json

Now when you start the docker daemon it will run, rather than error out.

Recommendation

We have released an updated package cloud-regionsrv-client-10.3.0-150300.13.6.1. It is recommended that you install this package if you have an instance with one of the buggy packages.

For those that picked up version 10.3.0 via and update, i.e. zypper up, meaning you have cloud-regionsrv-client-10.3.0-150300.13.3.1 on your system another zypper up will do the trick. The updated package is expected to be available on all update servers by end of day on August 22, 2024. For those that have instances from the affected images getting the fixed package onto your system will appear as a downgrade and as such you have to run

zypper in --oldpackage cloud-regionsrv-client-10.3.0-150300.13.6.1

This is an artifact triggered by how the package got into the affected images in the first place. The package that ended up in the affected images was built in a project that builds SLES 15 SP5, thus the number 150500 in the package name, while the package that is part of the release for repositories is built in a project for SLES 15 SP3, thus the 150300 in the package name.

Once you have the fixed package on your system and with the docker configuration already fixed via the above sed expressions you are all set.

Tell me more about the registry

Container registry support in the update infrastructure is currently primarily there to support SUSE Manager 5.0. As I already mentioned SUMa 5.0 runs in a container. The listings in the CSPs will still be VM listings as the SUMa container is only supported when running on a SUSE System, SLE Micro 5.5 and SLE Micro 6.0 support on the way. While SUMa 5.0 is already released the Public Cloud images are a little behind since we needed to finish up some Public Cloud specific code and it was simply not ready on release date. All code has been merged into the first maintenance update for SUMa and once this is ready for release we will publish SUMa 5.0 images. But I digress, back to the registry. As mentioned the registry as part of the update infrastructure is there to provide access to containers published by SUSE that require authentication. Currently those are SUMa 5 containers only. In the not too distant future we will support Long Term Service Pack Support (LTSS) for PAYG instances as an add-on, more on this by mid-October, and then access to LTSS Base Container Images will also be supported through this registry.

Leaves me with a rather embarrassingly long list of affected images:

Amazon:

• suse-sle-micro-5-5-v20240808-hvm-ssd-arm64-ltd
• suse-sle-micro-5-5-v20240808-hvm-ssd-x86_64-ltd
• suse-sle-micro-5-5-v20240808-hvm-ssd-x86_64-llc
• suse-sle-micro-5-5-byos-v20240808-hvm-ssd-x86_64
• suse-sle-micro-5-5-byos-v20240808-hvm-ssd-arm64
• suse-sle-micro-5-5-v20240808-hvm-ssd-arm64-llc
• suse-sles-sap-15-sp5-v20240430-hvm-ssd-x86_64
• suse-sle-hpc-15-sp5-byos-v20240808-hvm-ssd-x86_64
• suse-sles-sap-15-sp5-hardened-byos-v20240808-hvm-ssd-x86_64
• suse-sles-15-sp5-byos-v20240808-hvm-ssd-x86_64
• suse-sles-15-sp5-v20240808-hvm-ssd-arm64
• suse-sles-15-sp5-v20240808-ecs-hvm-ssd-x86_64
• suse-sles-15-sp5-sapcal-v20240808-hvm-ssd-x86_64
• suse-sles-15-sp5-v20240808-hvm-ssd-x86_64
• suse-sles-15-sp5-byos-v20240808-hvm-ssd-arm64
• suse-sles-sap-15-sp5-byos-v20240808-hvm-ssd-x86_64

Azure:

• suse-sle-micro-5-5-v20240810-arm64-llc
• suse-sle-micro-5-5-v20240809-arm64-ltd
• suse-sles-sap-15-sp5-hardened-byos-v20240812-x86_64
• suse-sles-sap-15-sp5-hardened-v20240812-x86_64
• suse-sle-hpc-15-sp5-v20240809-x86_64
• suse-sles-15-sp5-sapcal-v20240809-x86_64
• suse-sles-sap-15-sp5-v20240809-x86_64
• suse-sles-15-sp5-byos-v20240809-x86_64
• suse-sle-hpc-15-sp5-byos-v20240809-x86_64
• suse-sles-15-sp5-byos-v20240809-arm64
• suse-sles-sap-15-sp5-byos-v20240809-x86_64
• suse-sles-15-sp5-basic-v20240809-x86_64
• suse-sles-15-sp5-v20240809-x86_64
• suse-sles-15-sp5-hardened-byos-v20240809-x86_64
• suse-sles-15-sp5-v20240809-arm64

Google:

• sle-micro-5-5-byos-v20240807-arm64
• sle-micro-5-5-byos-v20240807-x86-64
• sles-15-sp5-sap-v20240808-x86-64
• sles-sap-15-sp5-hardened-byos-v20240808-x86-64
• sles-15-sp5-sapcal-v20240808-x86-64
• sles-15-sp5-byos-v20240808-arm64
• sles-15-sp5-byos-v20240808-x86-64
• sles-15-sp5-hardened-byos-v20240808-x86-64
• sle-hpc-15-sp5-byos-v20240808-x86-64
• sles-15-sp5-v20240808-arm64
• sles-15-sp5-v20240808-x86-64
• sles-sap-15-sp5-hardened-v20240808-x86-64
• sles-15-sp5-sap-byos-v20240808-x86-64

The affected images are being refreshed as quickly as we can get them published and the new images contain the fixed cloud-regionsrv-client package.

Apologies for the inconvenience, testing has been improved.