Problem
Once fedora:latest moves to Fedora 45, all four images will ship an invalid
/etc/containers/storage.conf, and every command inside them will exit 1:
$ podman run --rm <image> buildah --version
level=error msg="decode configuration \"/etc/containers/storage.conf\": toml: line 50 (last key \"storage.options\"): expected '.' or '=', but got ',' instead"
$ echo $?
1
This is not yet visible in the published images, because they are built from
Fedora stable. It becomes visible as soon as the base rolls to F45. I hit it
building the recipe against registry.fedoraproject.org/fedora:rawhide.
Cause
Each Containerfile inserts the shared image store with:
-e '/additionalimage.*/a "/var/lib/shared",'
That assumes storage.conf ships an open additionalimagestores = [
array to append into. containers-common changed that key to a commented
one-liner:
# containers-common-0.67.0-1.fc44
54:additionalimagestores = [
# containers-common-0.69.0-1.fc45
49:# additionalimagestores = []
So on F45 the appended value lands at top level inside [storage.options]
rather than inside an array:
[storage.options]
# AdditionalImageStores is used to pass paths to additional Read/Only image stores
# Must be comma separated list.
# additionalimagestores = []
"/var/lib/shared", <-- no longer inside an array; invalid TOML
The sed is present in all four recipes, so buildah, podman, skopeo
and aio are all affected:
$ for d in buildah podman skopeo aio; do
printf "%-8s " "$d"
grep -c 'additionalimage.*/a' "$d/Containerfile"
done
buildah 1
podman 1
skopeo 1
aio 1
The second sed, which derives /home/build/.config/containers/storage.conf
from /etc/containers/storage.conf, copies the broken line through, so the
rootless config is invalid too.
Why CI would not catch this
The build itself succeeds. During a build, buildah reads the builder's
config, not the one it is writing into the image, so nothing fails until
somebody runs the finished image. This seems like a concrete argument for
#39 — a post-build buildah --version in the built image would have caught
it outright.
Reproducer
$ git clone https://github.com/podman-container-tools/image_build
$ cd image_build
$ buildah build --build-arg FLAVOR=stable \
--from registry.fedoraproject.org/fedora:rawhide -t bud-test ./buildah
$ podman run --rm bud-test buildah --version
level=error msg="decode configuration \"/etc/containers/storage.conf\": toml: ..."
$ echo $?
1
Suggested fix
Set the key rather than appending to it, so the result does not depend on
whether the shipped file has it commented out. Something along the lines of:
-e 's|^#*[[:space:]]*additionalimagestores.*|additionalimagestores = ["/var/lib/shared"]|'
replacing the current /additionalimage.*/a append. That is idempotent
across both layouts.
Note that F44's file also carries "/usr/lib/containers/storage" as a
default additional store, which F45 drops; keeping it out looks preferable
anyway, since that store is what #56 reports as unreadable for the
non-root build user.
I am working around it downstream with an overlay layer that rewrites the
key, guarded on it being commented out, so I have not sent a PR — happy to
if the approach above looks right to you.
Problem
Once
fedora:latestmoves to Fedora 45, all four images will ship an invalid/etc/containers/storage.conf, and every command inside them will exit 1:This is not yet visible in the published images, because they are built from
Fedora stable. It becomes visible as soon as the base rolls to F45. I hit it
building the recipe against
registry.fedoraproject.org/fedora:rawhide.Cause
Each
Containerfileinserts the shared image store with:That assumes
storage.confships an openadditionalimagestores = [array to append into.
containers-commonchanged that key to a commentedone-liner:
So on F45 the appended value lands at top level inside
[storage.options]rather than inside an array:
The
sedis present in all four recipes, sobuildah,podman,skopeoand
aioare all affected:The second
sed, which derives/home/build/.config/containers/storage.conffrom
/etc/containers/storage.conf, copies the broken line through, so therootless config is invalid too.
Why CI would not catch this
The build itself succeeds. During a build, buildah reads the builder's
config, not the one it is writing into the image, so nothing fails until
somebody runs the finished image. This seems like a concrete argument for
#39 — a post-build
buildah --versionin the built image would have caughtit outright.
Reproducer
Suggested fix
Set the key rather than appending to it, so the result does not depend on
whether the shipped file has it commented out. Something along the lines of:
replacing the current
/additionalimage.*/aappend. That is idempotentacross both layouts.
Note that F44's file also carries
"/usr/lib/containers/storage"as adefault additional store, which F45 drops; keeping it out looks preferable
anyway, since that store is what #56 reports as unreadable for the
non-root
builduser.I am working around it downstream with an overlay layer that rewrites the
key, guarded on it being commented out, so I have not sent a PR — happy to
if the approach above looks right to you.