https://microk8s.io/docs/registry-images
So images. A container is based on an image. You can pull these images in from the outside world, e.g. from a Public, Private registry, or from the built-in registry.
For this copy the “hello-python” directory we created in Part 2 into one called “hello-python3”, you can delete the .tar file in there for now, its not needed.
Push Image into Kubernetes Image Repository
So in the earlier steps, we basically injected the image straight into the Kubernetes image cache, so not using the built-in registry.
Now we’ll do the same but this time working with the Kubernetes registry, so firstly lets enable it if not already done:
microk8s enable registryRight, so the registry is now available at localhost:32000, when we upload an image we need to tag it accordingly with localhost:32000/your-image-name.
So let’s inject the image we used before, but this time tag with another name:
docker build . -t localhost:32000/hello-python:registryBuild the image:
root@k8s-master:/home/ubuntu/application/hello-python/app# docker build . -t localhost:32000/hello-python:registry
Sending build context to Docker daemon 99.71MB (updates….)Wait for the image to upload into the docker images library.
docker images
root@k8s-master:/home/ubuntu/application/hello-python/app# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost:32000/hello-python registry 03e7eaf7b301 13 seconds ago 1.77GB
hello-python local a5aba8ae8f7d 38 hours ago 874MB
python 3.7 42cd7c61ce0e 2 days ago 863MBNow we want to push this into the Kubernetes registry:
docker push localhost:32000/hello-python
root@k8s-master:/home/ubuntu/application/hello-python/app# docker push localhost:32000/hello-python
The push refers to repository [localhost:32000/hello-python]
32963f3bbf70: Pushing [==================> ] 3.899MB/10.74MB
0059fe46108e: Pushing [> ] 9.997MB/897.8MB
8c51df724705: Pushed
fc5072a676ca: Pushing [==========================> ] 3.773MB/7.235MB
bfc730b5d6a1: Pushed
c2047c5c6a3d: Pushing [===> ] 6.636MB/86.97MB
ad87fe9fbf86: Pushing [===============> ] 5.625MB/18.08MB
de08a49275ea: Waiting
e6eb43d220d2: Waiting
a76bedae40bf: Waiting
9184b9a70c9e: Waiting
9276caf83dc1: WaitingAt this point the image is then within the Kubernetes image repository and ready to be used in your application deployments.
Deploy Container using Registry Image
So lets create a cheeky application, create a deployment.yaml and service.yaml as follows:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-python3
spec:
selector:
matchLabels:
app: hello-python3
replicas: 1
template:
metadata:
labels:
app: hello-python3
spec:
containers:
- name: hello-python3
image: localhost:32000/hello-python:registry
ports:
- containerPort: 5000And an service.yaml:
apiVersion: v1
kind: Service
metadata:
name: hello-python3-service
spec:
selector:
app: hello-python3
ports:
- port: 8080
targetPort: 5000
type: LoadBalancerThen deploy the application and service with:
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
root@k8s-master:/home/ubuntu/application/hello-python3/app# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-python-6bfc96894d-qhv8h 1/1 Running 1 38h 10.1.54.21 k8s-master <none> <none>
hello-python-redux-7849b9b844-55sxb 1/1 Running 0 19h 10.1.54.40 k8s-master <none> <none>
hello-python3-5fdcb55ccd-dvpnx 0/1 ImagePullBackOff 0 47s 10.1.49.2 k8s-worker-02 <none> <none>Oh look, we have an error, excellent. So let’s look at why this happened.
So so far, we’ve deployed the image registry on the master node, the worker nodes have no configuration for them to reach this registry at the moment. Now if we were to have built and pushed this registry in when we would have only had one node, then the only place the pod could have run would have been on the master node, which can get to the Kubernetes image registry. To fix this we need to complete the next section, as you can see the “ImagePullBackOff” basically means that the pods has been scheduled to a node which is unable to download the image.
If we look at the log for the pod: “hello-python3-5fdcb55ccd-dvpnx” we see:
root@k8s-master:/home/ubuntu/application/hello-python3/app# kubectl logs hello-python3-7b6df7676b-sp2mf
Error from server (BadRequest): container "hello-python3" in pod "hello-python3-7b6df7676b-sp2mf" is waiting to start: image can't be pulledSo we make the change to the deployment.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-python3
spec:
selector:
matchLabels:
app: hello-python3
replicas: 1
template:
metadata:
labels:
app: hello-python3
spec:
containers:
- name: hello-python3
image: 192.168.1.164:32000/hello-python:registry
ports:
- containerPort: 5000And then deploy again with:
kubectl apply -f deployment.yamlWe still are having problems.
If we run this on the node where the pod is trying to start:
tail -f /var/log/syslog | grep helloWe see:
Aug 2 14:48:49 k8s-worker-02 microk8s.daemon-kubelet[1324]: E0802 14:48:49.023084 1324 pod_workers.go:191] Error syncing pod 9490b872-87cd-48c9-90b6-952c7dc7b452 ("hello-python3-7b6df7676b-vt4v5_default(9490b872-87cd-48c9-90b6-952c7dc7b452)"), skipping: failed to "StartContainer" for "hello-python3" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"192.168.1.164:32000/hello-python:registry\": failed to resolve reference \"192.168.1.164:32000/hello-python:registry\": failed to do request: Head \"https://192.168.1.164:32000/v2/hello-python/manifests/registry\": http: server gave HTTP response to HTTPS client"So this exact error is described here: https://microk8s.io/docs/registry-private under the section “Configuring MicroK8s”. We need to make some changes to the configuration of the MicroK8s nodes.
Allowing Insecure Registry
Edit the file: /var/snap/microk8s/current/args/containerd-template.toml on ONLY the worker nodes, so in our case k8s-worker-01 and k8s-worker-02, right at the bottom of the file look for the following section:
[plugins] -> [plugins.”io.containerd.grpc.v1.cri”.registry] -> [plugins.”io.containerd.grpc.v1.cri”.registry.mirrors]:
Create a new section underneath to match the IP address of your master node where the image repository lies for example the below, in my case the k8s-master node is on 192.168.1.164.
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.1.164:32000"]
endpoint = ["http://192.168.1.164:32000"]The Microk8s documentation then talks about just restarting microk8s with a “microk8s stop” followed by a “microk8s start”, however I found this was not enough, so rebooted the worker and master nodes, once restarted, i attempted the deployment again:
kubectl apply -f deployment.yamlThen a quick check on the progress:
root@k8s-master:/home/ubuntu/application/hello-python3/app# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-python-6bfc96894d-qhv8h 1/1 Running 4 40h 10.1.54.63 k8s-master <none> <none>
hello-python-redux-7849b9b844-55sxb 1/1 Running 3 21h 10.1.54.59 k8s-master <none> <none>
hello-python3-7b6df7676b-qk2mq 1/1 Running 0 8m11s 10.1.81.4 k8s-worker-01 <none> <none>As you can see it is there, the pod is running on the k8s-worker-01 node, lets have a quick look at the logs:
cat /var/log/syslog | grep hello-python3Now this time we see all is well:
Aug 2 15:11:21 k8s-worker-01 microk8s.daemon-containerd[1303]: time="2020-08-02T15:11:21.254616306+01:00" level=info msg="CreateContainer within sandbox \"8f4012258d385007f7a6523b627ca1fa27f4ea21d01e3ac6bf3024b4ffe1fca0\" for container &ContainerMetadata{Name:hello-python3,Attempt:0,}"
Aug 2 15:11:25 k8s-worker-01 microk8s.daemon-containerd[1303]: time="2020-08-02T15:11:25.561390309+01:00" level=info msg="CreateContainer within sandbox \"8f4012258d385007f7a6523b627ca1fa27f4ea21d01e3ac6bf3024b4ffe1fca0\" for &ContainerMetadata{Name:hello-python3,Attempt:0,} returns container id \"bc93c29f081139990e98558d065e6b459fe26b34e3f17e49b7c7ecbc3768785e\""Now lets create the service to access the application, so back on the Master node, we run:
kubectl apply -f service.yamlAnd have a quick check:
root@k8s-master:/home/ubuntu/application/hello-python3/app# kubectl get services -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
hello-python3-service LoadBalancer 10.152.183.149 192.168.1.22 8080:31272/TCP 16s app=hello-python3
kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 40h <none>Yes, there it is: http://192.168.1.22:8080, so lets try to access it:
Good stuff, we’ve done it.
Doing it All Again! Well Why Not!?
Lets run this command:
microk8s kubectl create deployment hello-python4 --image=192.168.1.164:32000/hello-python:registryThat will just create a deployment of that image as a new container and then schedule it on one of the worker nodes:
root@k8s-master:/home/ubuntu/application/hello-python3/app# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-python-6bfc96894d-qhv8h 1/1 Running 4 40h 10.1.54.63 k8s-master <none> <none>
hello-python-redux-7849b9b844-55sxb 1/1 Running 3 21h 10.1.54.59 k8s-master <none> <none>
hello-python3-7b6df7676b-qk2mq 1/1 Running 0 13m 10.1.81.4 k8s-worker-01 <none> <none>
hello-python4-55bdb58666-sk8wg 1/1 Running 0 5s 10.1.81.5 k8s-worker-01 <none> <none>Cool, there it is “hello-python4-55bdb58666-sk8wg” working as expected on the k8s-worker-01 node, thats good that proves that the registry is working.
Verifying the Registry Image is Used
Okay so lets copy the existing directory:
cp -r hello-python hello-python5Now within there, if there is an image .tar file delete it, we don’t need it anymore.
Let’s edit the application to show we are getting a new image.
cd hello-python5/appEdit the “main.py”, and add something to the line saying “Hello from Python”, e.g. “This is a test page using Hello-python5 application!”
Build the image, and tag it as the image name for the Kubernetes image registry when put into the docker image library.
docker build . -t localhost:32000/hello-python5:registry
docker images
root@k8s-master:/home/ubuntu/application/hello-python5/app# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost:32000/hello-python5 registry e699191159f2 6 seconds ago 874MBOkay so now get docker to push it into the Kubernetes image library.
docker push localhost:32000/hello-python5Now edit the deployment.yaml and service.yaml, and where you have “hello-python3…” change it to “hello-python5”, including the image location in the deployment.yaml file, so we make sure we use our new image!
So now deploy the service and deployment with:
kubectl apply -f deployment.yaml
kubectl apply -f service.yamlIf we run a quick: “tail -f /var/log/syslog | grep hello-python5” we can see the container image being deployed and started.
Aug 2 15:44:32 k8s-worker-02 microk8s.daemon-containerd[1300]: time="2020-08-02T15:44:32.322560841+01:00" level=info msg="PullImage \"192.168.1.164:32000/hello-python5:registry\" returns image reference \"sha256:e699191159f229e8e6aeae167a32c37a830d7c238182ee3f8825e8d5000f0d3d\""
Aug 2 15:44:32 k8s-worker-02 microk8s.daemon-containerd[1300]: time="2020-08-02T15:44:32.328968944+01:00" level=info msg="CreateContainer within sandbox \"35c4ee9ebdd4b90744f276d066ba7c75abe1f856cfe6e2e795100c39411789f4\" for container &ContainerMetadata{Name:hello-python5,Attempt:0,}"
Aug 2 15:44:33 k8s-worker-02 microk8s.daemon-containerd[1300]: time="2020-08-02T15:44:33.337282018+01:00" level=info msg="CreateContainer within sandbox \"35c4ee9ebdd4b90744f276d066ba7c75abe1f856cfe6e2e795100c39411789f4\" for &ContainerMetadata{Name:hello-python5,Attempt:0,} returns container id \"59ff7a8553651222ed6f3e8a13801c2ae850f995d133fa5e79bb37da5b297db9\""
kubectl get servicesGives us:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-python5-service LoadBalancer 10.152.183.154 192.168.1.23 8080:32278/TCP 23sThere it is running at http://192.168.1.23:8080, so let’s give it a connect in a browser:
And you can see we really are running the expected image, because its the hello-python5 one as we wanted.
Using an External Public Registry (e.g. DockerHub)
https://microk8s.io/docs/registry-public
Create an account at Docker Hub.
You’ll also need to create a repository. On your master node run:
Docker loginEnter your logon details.
Now lets create a new image and push it up into the Docker Hub.
First make a copy of the existing application image definition.
cp -r hello-python5 hello-python6
cd hello-python6/appMake a change to the image’s main.py, so you know its something different for when you run it. Build the image locally, then we’ll push it up.
docker build . -t geekmungus/private:hello-python6Where “geekmungus” is my DockerHub username, “private” is the repository name and “hello-python6” is the tag (in this case the name of the application).
root@k8s-master:/home/ubuntu/application/hello-python6/app# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
geekmungus/private hello-python6 1dfc3b05a3ca 5 seconds ago 874MBOkay now we can push that repository up.
docker push geekmungus/privateCool, and there it is:
Let’s delete the local copy to ensure we only ever use the pulled version, when we now do a deployment.
docker images
root@k8s-master:/home/ubuntu/application/hello-python6/app# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
geekmungus/private hello-python6 1dfc3b05a3ca 4 minutes ago 874MB
docker image rm 1dfc3b05a3ca So now that image has gone and is not available locally, so it should pull from DockerHub now when we refer to it.
Change the deployment.yaml and service.yaml in the hello-python6/app folder, so everywhere where it did say “hello-python5” it now says “hello-python6”.
Then ensure you put the image name in as: geekmungus/private:hello-python6
kubectl apply -f deployment.yaml
kubectl apply -f service.yamlWatching the logs I got this error:
Aug 2 20:59:45 k8s-worker-01 microk8s.daemon-containerd[1303]: time="2020-08-02T20:59:45.554847607+01:00" level=info msg="PullImage \"geekmungus/private:hello-python6\""
Aug 2 20:59:47 k8s-worker-01 microk8s.daemon-containerd[1303]: time="2020-08-02T20:59:47.366927939+01:00" level=error msg="PullImage \"geekmungus/private:hello-python6\" failed" error="failed to pull and unpack image \"docker.io/geekmungus/private:hello-python6\": failed to resolve reference \"docker.io/geekmungus/private:hello-python \": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed"https://stackoverflow.com/questions/56642311/microk8s-cannot-pull-from-private-registry
https://github.com/ubuntu/microk8s/issues/512
So run this to create a secret to be used when you connect to the external public repository.
kubectl create secret docker-registry regcred --docker-username=<yourdockerhubusername> --docker-password=<password> --docker-email=<youremailaddress>And then we see:
root@k8s-master:/home/ubuntu/application/hello-python6/app# kubectl create secret docker-registry regcred --docker-username=<yourdockerhubusername> --docker-password=<password> --docker-email=<youremailaddress> --docker-server=https://index.docker.io/v1/
secret/regcred createdWe now have a secret called “regcred” we can use in our deployments.
Now within the deployment.yaml, you need to refer to it so your deployment.yaml will end up looking like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-python6
spec:
selector:
matchLabels:
app: hello-python6
replicas: 1
template:
metadata:
labels:
app: hello-python6
spec:
containers:
- name: hello-python6
image: geekmungus/private:hello-python6
ports:
- containerPort: 5000
imagePullSecrets:
- name: regcredNow redeploy with:
Kubectl apply -f deployment.yamlIt might take a little while because it will be downloading the image from dockerhub, but then:
root@k8s-master:/home/ubuntu/application/hello-python6/app# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-python6-7ccb5c4bf5-4z9sf 0/1 ContainerCreating 0 8s <none> k8s-worker-01 <none> <none>
root@k8s-master:/home/ubuntu/application/hello-python6/app# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-python6-7ccb5c4bf5-4z9sf 1/1 Running 0 67s 10.1.81.11 k8s-worker-01 <none> <none>Finally deploy the service:
kubectl apply -f service.yaml
root@k8s-master:/home/ubuntu/application/hello-python6/app# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-python6-service LoadBalancer 10.152.183.229 192.168.1.24 8080:32475/TCP 6s
kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 46h
And we are done! We can see the application running on that port in our browser and it is the image we expected to see.
