first problem: docker image with wrong arch

➜ helloworld-go git:(main) kubectl describe pod helloworld-go

Name:             helloworld-go-586857689f-7dtxb

Namespace:        default

...

Containers:

  helloworld-go:

    Container ID:   containerd://2a812fef1c5961108c94b15aa7232d459c783b6fa35d697491d13e3097116e3e

    Image:          docker.io/tjenwellens/helloworld-go:latest

    Image ID:       docker.io/tjenwellens/helloworld-go@sha256:21f452f905d1453d914641ad98fd23d36282ccab688a1ea123689584699051ce

...

Events:

  Type     Reason     Age               From               Message

  ----     ------     ----              ----               -------

  Normal   Scheduled  30s               default-scheduler  Successfully assigned default/helloworld-go-586857689f-7dtxb to homeserver

  Normal   Pulled     5s (x3 over 30s)  kubelet            Container image "docker.io/tjenwellens/helloworld-go:latest" already present on machine

  Normal   Created    5s (x3 over 30s)  kubelet            Created container helloworld-go

  Normal   Started    5s (x3 over 30s)  kubelet            Started container helloworld-go

  Warning  BackOff    0s (x4 over 29s)  kubelet            Back-off restarting failed container helloworld-go in pod helloworld-go-586857689f-7dtxb_default(67ed7a01-671a-4a64-a64f-9fc04fb0c65d)

originally there was something in the events which made me google k3s readyness probe failedconnection refused

something about the readyness

➜ helloworld-go git:(main) kubectl logs helloworld-go-586857689f-54t28

exec /server: exec format error

There could be many reasons why this error occurs while working with containers:

  1. Using the wrong script header such as adding a space in between.
  1. Using some incompatible character encodings while writing the scripts
  1. Non-matching CPU architecture
  1. Missing file permissions

I assume non-matching CPU arch, since I did a plain docker build on my mac

src: https://stackoverflow.com/questions/73285601/docker-exec-usr-bin-sh-exec-format-error

dockerhub shows which os/arch are available (used to be only linux/arm64)

![_new/blog/resources/dockerx build for multi arch builds 20240524152050.md/2104f755-8500-47ea-9772-20be01b37a6e.png](new/blog/resources/dockerx build for multi arch builds 20240524152050.md/2104f755-8500-47ea-9772-20be01b37a6e.png)

solution docker buildx multi arch

➜ helloworld-go git:(main) docker buildx build --platform linux/amd64,linux/arm64 --push -t tjenwellens/helloworld-go:latest --progress=plain .

src: https://www.baeldung.com/ops/docker-buildx

second problem: got weird error when building amd64:

ERROR: failed to solve: process “/bin/sh -c CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build -mod=readonly -v -o server” did not complete successfully: exit code: 1

(docker hub was down so could be that)

tried to log in the docker build step

RUN echo “TARGETOS=${TARGETOS} GOARCH=${TARGETARCH} xxx” #13 0.059 TARGETOS=linux GOARCH=amd64 xxx

docker buildx build –platform linux/amd64 –push -t tjenwellens/helloworld-go:latest –progress=plain .

// outputs all instead of disappearing

src: https://stackoverflow.com/questions/64804749/why-is-docker-build-not-showing-any-output-from-commands

tried to do the build myself inside the base image:

➜ helloworld-go git:(main) docker run -it -v .:/app -w /app -e TARGETOS=linux -e TARGETARCH=amd64 golang:latest /bin/bash

and then running the install myself

CGO_ENABLED=0 GOOS=${TARGETOS} GOARCH=${TARGETARCH} go build -mod=readonly -v -o server

solution: solved itself, probably just a temporary thing because docker hub was down?

which magically seemed to work, then I tried the multibuild again, and that worked!

so probably was just dockerhub that was down, and then up again?

docker buildx build --platform linux/amd64,linux/arm64 --push -t tjenwellens/helloworld-go:latest --progress=plain .

--platform linux/amd64,linux/arm64

don’t forget the --push as it seems like you cannot do a multistep (first build, then push) like with regular docker build

how do you know which platform?

tjen@homeserver:~$ uname -m

x86_64

tjen@homeserver:~$ arch

x86_64

If the output is “arm”, it means the processor architecture is ARM-based. If the output is “x86_64”, it means the processor architecture is AMD-based (also known as x86-64 or Intel 64).

src: https://stackoverflow.com/questions/48678152/how-to-detect-386-amd64-arm-or-arm64-os-architecture-via-shell-bash

problem 3: old image cached on cluster

helm chart is configured with pullPolicy: IfNotPresent

//values.yaml

image:
  repository: docker.io/tjenwellens/helloworld-go
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: "latest"

don’t really wanna change this

how to go to node and remove image?

  • ubuntu server with k3s, nothing else
  • docker? no docker installed on machine
  • (only a single master, no agents)

k3s tool?

tjen@homeserver:~$ k3s crictl images

WARN[0000] Failed to stat /var/lib/rancher/k3s/agent/etc/crictl.yaml: permission denied

FATA[0000] load config file: stat /var/lib/rancher/k3s/data/b159f6e26663d8c92285e7bc4a6881d85bd8c81efc55eb2cf191c54100387fbb/bin/crictl.yaml: no such file or directory

tjen@homeserver:~$ sudo k3s crictl images

IMAGE                                        TAG                     IMAGE ID            SIZE

docker.io/bitnami/mariadb                    10.3.22-debian-10-r27   86f8545f6c096       94.2MB

docker.io/bitnami/wordpress                  5.3.2-debian-10-r32     287dfba5a5b4b       156MB

docker.io/library/nginx                      latest                  e784f4560448b       71MB

docker.io/rancher/klipper-helm               v0.8.3-build20240228    0929b4140ada6       91.2MB

docker.io/rancher/klipper-lb                 v0.4.7                  edc812b8e25d0       4.78MB

docker.io/rancher/local-path-provisioner     v0.0.26                 c54dcef6214cb       17.2MB

docker.io/rancher/mirrored-coredns-coredns   1.10.1                  ead0a4a53df89       16.2MB

docker.io/rancher/mirrored-library-busybox   1.36.1                  65ad0d468eb1c       2.16MB

docker.io/rancher/mirrored-library-traefik   2.10.7                  ee69e8120b64a       43.2MB

docker.io/rancher/mirrored-metrics-server    v0.7.0                  b9a5a1927366a       19.4MB

docker.io/rancher/mirrored-pause             3.6                     6270bb605e12e       301kB

docker.io/tjenwellens/helloworld-go          latest                  5746bb87c6212       7.43MB

tjen@homeserver:~$ sudo crictl images

IMAGE                                        TAG                     IMAGE ID            SIZE

docker.io/bitnami/mariadb                    10.3.22-debian-10-r27   86f8545f6c096       94.2MB

docker.io/bitnami/wordpress                  5.3.2-debian-10-r32     287dfba5a5b4b       156MB

docker.io/library/nginx                      latest                  e784f4560448b       71MB

docker.io/rancher/klipper-helm               v0.8.3-build20240228    0929b4140ada6       91.2MB

docker.io/rancher/klipper-lb                 v0.4.7                  edc812b8e25d0       4.78MB

docker.io/rancher/local-path-provisioner     v0.0.26                 c54dcef6214cb       17.2MB

docker.io/rancher/mirrored-coredns-coredns   1.10.1                  ead0a4a53df89       16.2MB

docker.io/rancher/mirrored-library-busybox   1.36.1                  65ad0d468eb1c       2.16MB

docker.io/rancher/mirrored-library-traefik   2.10.7                  ee69e8120b64a       43.2MB

docker.io/rancher/mirrored-metrics-server    v0.7.0                  b9a5a1927366a       19.4MB

docker.io/rancher/mirrored-pause             3.6                     6270bb605e12e       301kB

docker.io/tjenwellens/helloworld-go          latest                  5746bb87c6212       7.43MB

tjen@homeserver:~$ sudo crictl rmi docker.io/tjenwellens/helloworld-go:latest

Deleted: docker.io/tjenwellens/helloworld-go:latest

tjen@homeserver:~$ sudo crictl images

IMAGE                                        TAG                     IMAGE ID            SIZE

docker.io/bitnami/mariadb                    10.3.22-debian-10-r27   86f8545f6c096       94.2MB

docker.io/bitnami/wordpress                  5.3.2-debian-10-r32     287dfba5a5b4b       156MB

docker.io/library/nginx                      latest                  e784f4560448b       71MB

docker.io/rancher/klipper-helm               v0.8.3-build20240228    0929b4140ada6       91.2MB

docker.io/rancher/klipper-lb                 v0.4.7                  edc812b8e25d0       4.78MB

docker.io/rancher/local-path-provisioner     v0.0.26                 c54dcef6214cb       17.2MB

docker.io/rancher/mirrored-coredns-coredns   1.10.1                  ead0a4a53df89       16.2MB

docker.io/rancher/mirrored-library-busybox   1.36.1                  65ad0d468eb1c       2.16MB

docker.io/rancher/mirrored-library-traefik   2.10.7                  ee69e8120b64a       43.2MB

docker.io/rancher/mirrored-metrics-server    v0.7.0                  b9a5a1927366a       19.4MB

docker.io/rancher/mirrored-pause             3.6                     6270bb605e12e       301kB

solution: delete old image from k3s node with ‘sudo crictl rmi’

sudo crictl images

sudo crictl rmi docker.io/tjenwellens/helloworld-go:latest

(or maybe sudo k3s crictl images, probably works too)

inspiration: