Container images are stored in image registries (e.g. Docker Hub, Quay.io) from where they can be pulled and run manually or in a Kubernetes cluster. Vanilla Kubernetes doesn’t have an internal image registry which is typically needed in an enterprise environment. Red Hat OpenShift intends to fill this gap by providing an integrated registry out-of-the-box and introducing a new Kubernetes resource called ImageStream to manage images in a “Kubernetes-native way”.

ImageStreams are designed to support different use cases, but the impact of certain parameters can be confusing at the beginning. In this post we’ll take a look at some of these parameters and how they change the behavior of the deployments created from different ImageStreams. For the basic concept of image management in OpenShift take a look at these links first:

Managing imagestreams in OpenShift 4 documentation

Managing Images in OpenShift 3 documentation

Related blogs here, here and here.

In OpenShift 4 the functionality of ImageStreams is mostly the same as it was in OpenShift 3. For the tests in this blog post we used OpenShift v4.1.6.

Technically an ImageStream is just a Kubernetes resource containing metadata. Beside managing internal images — that are typically built within an OpenShift cluster — ImageStreams can point to images in external registries. In this case they are only an extra abstraction layer providing a few additional features (e.g. polling updates, caching). Because an ImageStream can behave in two distinct ways, it’s easier to understand them if we differentiate whether they point to an internal or an external image.

Similarly to a repository in a registry an ImageStream has multiple tags pointing to an image, so an ImageStream is actually a collection of ImageStreamTags. Strictly speaking the container is created from an ImageStreamTag (not an ImageStream), but for simplicity we use these terms interchangeably in this post.

Internal vs external images

First, a few notes about images that are stored in the internal registry:

For internal images the ImageStreams and the internal registry are tightly coupled. For example myimage-internal ImageStream in the myproject namespace is mapped to image-registry.openshift-image-registry.svc:5000/myproject/myimage-internal repository.

ImageStream in the namespace is mapped to repository. If we run a new build where the output is an ImageStream, the image will be stored in the internal registry.

If we manually push to the registry (see howto), a matching ImageStream is automatically created. The url must refer a valid namespace where we have edit permission.

If we delete the ImageStream, the repository is also deleted in the internal registry and we loose our images.

To maintain space used by the registry, you have to occasionally prune images. Internal images that are used — or were recently used — in a build or deployment are kept automatically during the pruning process.

How about ImageStreams for external images?:

We can create an ImageStream pointing to an external image with the oc import-image or oc tag commands — or directly with a yaml.

or commands — or directly with a yaml. Within a single ImageStream you can have tags pointed to images from completely different registries and repositories. This is an effective way to couple external images that are related, but being stored in different repositories.

Being an abstraction layer ImageStreams hide where the images are actually coming from. When builds and deployments reference an ImageStream, we can change the image being used by modifying the url within the ImageStream instead of editing each build and deployment one-by-one.

An ImageStream refers images by their unique sha256 ID — instead of the tag — when they are created or updated. This provides stability against tags moving in the external registry.

ImageStreams can periodically (15 mins) monitor if the external tag changes (via --scheduled=true ). Sometimes this is a useful feature — for security updates,— sometimes we want the opposite and use ImageStreams to make sure we use the same image even if the external tag moves — for stability.

). Sometimes this is a useful feature — for security updates,— sometimes we want the opposite and use ImageStreams to make sure we use the same image even if the external tag moves — for stability. We can cache the external image in the internal registry (via --reference-policy=local ). This is really useful as it makes deployments quicker and independent from the availability of the external registry.

Another important feature — both for internal an external images — is the ability to automatically trigger related BuildConfigs and DeploymentConfigs if their ImageStream is updated. This behaviour can be set on those resources (see imageChangeParams.automatic=true ) and is usually enabled.

Variations

Now let’s have a look at the details of the different types of ImageStreams:

Pointing to an external image

Pointing to an external image with pullthrough (cached)

Direct reference to a external image url

Pointing to an internal image

Pointing to another ImageStream

External image

Create an ImageStream pointing to an external image like this:

oc import-image myproject/myimage-ref-source:mytag --from="docker.io/balazsszeti/hello:sleeper" --confirm

Alternatively we could use oc tag having the same result:

oc tag docker.io/balazsszeti/hello:sleeper myproject/myimage-ref-source:mytag

See the yaml resource created by the previous commands:

$ oc get is myimage-ref-source -oyaml

apiVersion: image.openshift.io/v1

kind: ImageStream

metadata:

annotations:

openshift.io/image.dockerRepositoryCheck: "2019-08-08T17:50:46Z"

creationTimestamp: "2019-08-08T17:50:45Z"

generation: 2

name: myimage-ref-source

namespace: myproject

resourceVersion: "741320"

selfLink: /apis/image.openshift.io/v1/namespaces/myproject/imagestreams/myimage-ref-source

uid: 0c8c4396-ba05-11e9-b920-0a580a800116

spec:

lookupPolicy:

local: false

tags:

- annotations: null

from:

kind: DockerImage

name: docker.io/balazsszeti/hello:sleeper

generation: 2

importPolicy: {}

name: mytag

referencePolicy:

type: Source

status:

dockerImageRepository: image-registry.openshift-image-registry.svc:5000/myproject/myimage-ref-source

publicDockerImageRepository: default-route-openshift-image-registry.apps-crc.testing/myproject/myimage-ref-source

tags:

- items:

- created: "2019-08-08T17:50:46Z"

dockerImageReference: docker.io/balazsszeti/hello@sha256:42957024...

generation: 2

image: sha256:42957024...

tag: mytag

In this example we use the name “myimage-ref-source” because of the setting referencePolicy.type being set to Source by default. See “External image with pullthrough” below how we can change this setting for different behaviour.

Let’s discuss how this ImageStream works:

What image url is used by a Pod when it’s created by a build or deployment referring this ImageStream?

The image is pulled by the sha256 ID where the external tag pointed at the moment the ImageStream was created: docker.io/balazsszeti/hello@sha256:42957024b43e121a210a1b3a8a44f497233a2385f7ef48227a6866afdb9b8e1b Does it work if the external registry is unreachable? E.g. during a temporary network outage or when the external registry is down?

No, because the Pod uses an external url. What happens if the tag in the external registry moves and we have a new rollout for the DeploymentConfig?

Since the ImageStream tag points to the sha256 ID of the image, the container will be created from that version of the image even if the external tag moves. You can manually update the reference in the ImageStream by running the oc tag or oc import-image again or by enabling automatic update via --scheduled=true (that adds importPolicy: {scheduled: true} in the yaml). Can this ImageStream be used in a Pod (or other vanilla Kubernetes resource)?

Yes, but not by default. You need lookupPolicy: {local: true} set and it only works within the same namespace.

oc set image-lookup myimage-ref-source Is DeploymentConfig rollback effective to switch back to previous version after a rollout?

Yes. Rolling back with oc rollback dc/myimage-ref-source creates a new deployment using the previous image url of the DeploymentConfig. As it refers to the external image url by the previous sha256 ID — Note: This will only work as long as the old image is still available in the external registry.

Our container for testing is a simple “Hello World” with a Dockerfile:

Also a DeploymentConfig example for reference to test deployments with an ImageStream:

External image with pullthrough

To create an ImageStream pointing to an external image and enable pullthrough modify the reference-policy flag to be —-reference-policy=local :

oc import-image myproject/myimage-ref-local:mytag --from="docker.io/balazsszeti/hello:sleeper" --confirm --reference-policy=local

or

oc tag docker.io/balazsszeti/hello:sleeper myproject/myimage-ref-local:mytag --reference-policy=local

Here we show only the interesting parts of the yaml:

$ oc get is -oyaml myimage-ref-local

apiVersion: image.openshift.io/v1

kind: ImageStream

metadata:

name: myimage-ref-local

spec:

tags:

- from:

kind: DockerImage

name: docker.io/balazsszeti/hello:sleeper

name: mytag

referencePolicy:

type: Local

status:

dockerImageRepository: image-registry.openshift-image-registry.svc:5000/myproject/myimage-ref-local

tags:

- items:

- dockerImageReference: docker.io/balazsszeti/hello@sha256:42957024...

image: sha256:42957024...

tag: mytag

The same questions as above:

What is the image url in a Pod?

The image is pulled through into the internal registry, so the Pod refers an internal image url: image-registry.openshift-image-registry.svc:5000/myproject/myimage-ref-local@sha256:42957024b43e121a210a1b3a8a44f497233a2385f7ef48227a6866afdb9b8e1b Does it work when the external registry is unreachable?

The image is cached when it’s used for the first time. So it wouldn’t work offline during the first deployment, but once cached, the external registry never needs to be contacted again. Even if the image is deleted from the external registry it will remain in the cache. What happens if the tag moves?

As before, the ImageStream tag points to the sha256 ID of the image, so the container is created from the same image even if the external tag moves. Using --scheduled=true or manually refreshing the ImageStream will update the sha256 ID it refers, but the new image will be cached only when it’s needed for a container. Can this ImageStream be used in a Pod?

Yes, after setting lookupPolicy: {local: true} . Is rollback effective?

Yes. The pod points to the internal url by sha256 ID, so the old image is being used after the rollback as expected. Obviously this only works if the old image was not pruned from the internal registry — it’s available if you can still see it in the ImageStream tag items history.

Reference to an external image

We can create an ImageStream referring an external tag directly via the --reference=true flag:

oc tag docker.io/balazsszeti/hello:sleeper myproject/myimage-reference:mytag --reference=true

The related yaml:

$ oc get is myimage-reference -oyaml

apiVersion: image.openshift.io/v1

kind: ImageStream

metadata:

name: myimage-reference

spec:

tags:

- from:

kind: DockerImage

name: docker.io/balazsszeti/hello:sleeper

name: mytag

reference: true

referencePolicy:

type: Source

status:

dockerImageRepository: image-registry.openshift-image-registry.svc:5000/myproject/myimage-reference

tags:

- items:

- dockerImageReference: docker.io/balazsszeti/hello:sleeper

image: ""

tag: mytag

The same questions as above:

What is the image url in a Pod?

The Pod is pulling the image by tag (instead of sha256 ID) from the external registry: docker.io/balazsszeti/hello:sleeper Does it work offline?

No, the image is pulled from the external registry. Adding --reference-policy=local makes no difference. What happens if the tag moves?

The container is created from the external tag url, so the new image behind the tag is always used. Pod parameter imagePullPolicy: Always is important in this case otherwise the old image may be cached on the node by the container runtime. Can this ImageStream be used in a Pod?

No. Even lookupPolicy: {local: true} doesn’t help. If that flag is set, the Pod will try to pull from a non existing internal url: image: image-registry.openshift-image-registry.svc:5000/myproject/myimage-reference:mytag Is rollback effective?

No, because the Pod is referencing the external floating tag (instead of sha256 ID). So after a rollback the container will pull the same tagged version of the image (e.g. docker.io/balazsszeti/hello:sleeper ) from the external registry, just like if we used the external image tag in the DeploymentConfig instead of the ImageStream.

In Quay.io public registry when a new image is pushed to an existing tag, the previous sha256 ID becomes invalid right away and the previous image can’t be pulled anymore — this doesn’t seem to be the case with Docker Hub. This is a good example where the pullthrough feature can help to cache the images we deploy.

Internal image

Let’s create an image by running a Docker build on the cluster — see the example Dockerfile above:

oc new-build --to='myimage-internal:mytag' --strategy=docker --binary=true --name=myimage-internal oc start-build myimage-internal --from-dir=. --follow

The yaml looks a bit different, note: there is no spec.tags :

$ oc get is myimage-internal -oyaml

apiVersion: image.openshift.io/v1

kind: ImageStream

metadata:

labels:

build: myimage-internal

name: myimage-internal

spec:

lookupPolicy:

local: false

status:

dockerImageRepository: image-registry.openshift-image-registry.svc:5000/myproject/myimage-internal

tags:

- items:

- dockerImageReference: image-registry.openshift-image-registry.svc:5000/myproject/myimage-internal@sha256:ff3b3c5e...

image: sha256:ff3b3c5e...

tag: mytag

The usual questions again:

What is the image url in a Pod?

The image URL will be a reference to the internal registry by sha256 ID: image-registry.openshift-image-registry.svc:5000/myproject/myimage-internal@sha256:ff3b3c5e3d09bf93e2a05ece04235f4ea8212f36b617161d8f11f374a14aeb74 Does it work offline?

Yes. The image is in the internal registry. What happens if the tag moves?

The ImageStreams and the internal images are kept in sync, so a new build (or push) updates the tag as well. A rollout will always create a pod with the latest image. Can this ImageStream be used in a Pod?

Yes, after setting lookupPolicy: {local: true} . Is rollback effective?

Yes. As long as the image was not pruned from the internal registry.

Note: In the examples above we used a different ImageStream for each type, but the same could be achieved by using different tags within one ImageStream.

Referencing another ImageStream

An ImageStream can also be created from another ImageStream, the new tag’s image url is set where the source tag points at that moment. The new image stream is not following the old one automatically (and you can’t set --scheduled=true in this case). Also the other properties ( referencePolicy , reference , lookupPolicy , …) are not taken over. The new ImageStream takes the sha256 ID of the image of the source ImageStream when it’s created/updated and is independent afterwards.

The most common use case for creating a new ImageStream from another one is for internally stored images, as it makes less sense in case of external images. For example if we have an automatic build triggered by a git hook, tagging the ImageStream in a “Development”, “Test” or “Prod” namespace can trigger the DeploymentConfig and we don’t need to do much else to have a basic CI/CD process. But this topic is definitely out of scope of this — already too long — post.

Let’s check how the image url looks like based on the type of the source ImageStream and different parameters — when Pods are created from our new ImageStream.

This last section is only here to cover all scenarios. We typically don’t use these edge cases — other than creating simple “cloned” tags in another namespace.

Old ImageStream points to an external image:

oc tag myproject/myimage-ref-source:mytag myproject/fromis-ref-source-myimage-ref-source:mytag

--> Pod image: docker.io/balazsszeti/hello@sha256:42957024... oc tag myproject/myimage-ref-source:mytag myproject/fromis-reference-myimage-ref-source:mytag --reference=true

--> Pod image: docker.io/balazsszeti/hello@sha256:42957024... oc tag myproject/myimage-ref-source:mytag myproject/fromis-ref-local-myimage-ref-source:mytag --reference-policy=local

--> Pod image: image-registry.openshift-image-registry.svc:5000/myproject/fromis-ref-local-myimage-ref-source@sha256:42957024...

All three yaml variants look similar:

$ oc get is -o yaml fromis-ref-source-myimage-ref-source

apiVersion: image.openshift.io/v1

kind: ImageStream

metadata:

name: fromis-ref-source-myimage-ref-source

spec:

tags:

- from:

kind: ImageStreamImage

name: myimage-ref-source@sha256:42957024...

namespace: myproject

name: mytag

status:

dockerImageRepository: image-registry.openshift-image-registry.svc:5000/myproject/fromis-ref-source-myimage-ref-source

tags:

- items:

- dockerImageReference: docker.io/balazsszeti/hello@sha256:42957024...

image: sha256:42957024...

tag: mytag

Old ImageStream points to an external image with pullthrough:

oc tag myproject/myimage-ref-local:mytag myproject/fromis-ref-source-myimage-ref-local:mytag

--> Pod image: docker.io/balazsszeti/hello@sha256:42957024... oc tag myproject/myimage-ref-local:mytag myproject/fromis-reference-myimage-ref-local:mytag --reference=true

--> Pod image: docker.io/balazsszeti/hello@sha256:42957024... oc tag myproject/myimage-ref-local:mytag myproject/fromis-ref-local-myimage-ref-local:mytag --reference-policy=local

--> Pod image: image-registry.openshift-image-registry.svc:5000/myproject/fromis-ref-local-myimage-ref-local@sha256:42957024...

There is no difference compared to tagging from a non-pullthrough ImageStream.

Old ImageStream is an external reference:

oc tag myproject/myimage-reference:mytag myproject/fromis-ref-source-myimage-reference:mytag

--> Pod image: docker.io/balazsszeti/hello@sha256:42957024... oc tag myproject/myimage-reference:mytag myproject/fromis-reference-myimage-reference:mytag --reference=true

--> Pod image: docker.io/balazsszeti/hello:sleeper oc tag myproject/myimage-reference:mytag myproject/fromis-ref-local-myimage-reference:mytag --reference-policy=local

--> Pod image: image-registry.openshift-image-registry.svc:5000/myproject/fromis-ref-local-myimage-reference@sha256:42957024...

If the source ImageStream we tag from is --reference=true the new ImageStream definition looks different than in the other cases. Here the tag’s kind is DockerImage and there’s no sign that this was created from another ImageStream.

$ oc get is -oyaml fromis-ref-source-myimage-reference

apiVersion: image.openshift.io/v1

kind: ImageStream

metadata:

name: fromis-ref-source-myimage-reference

spec:

tags:

- from:

kind: DockerImage

name: docker.io/balazsszeti/hello:sleeper

name: mytag

status:

dockerImageRepository: image-registry.openshift-image-registry.svc:5000/myproject/fromis-ref-source-myimage-reference

tags:

- items:

- dockerImageReference: docker.io/balazsszeti/hello@sha256:42957024...

image: sha256:42957024...

tag: mytag

Old ImageStream points to an internal image:

As mentioned above tagging internal images is common between different namespaces. There seems to be some special logic to handle --reference-policy=source like --reference-policy=local in this case to avoid permission problems between namespaces:

oc tag myproject/myimage-internal:mytag myproject/fromis-ref-source-myimage-internal:mytag

--> Pod image: image-registry.openshift-image-registry.svc:5000/myproject/fromis-ref-source-myimage-internal@sha256:ff3b3c5e... oc tag myproject/myimage-internal:mytag myproject/fromis-reference-myimage-internal:mytag --reference=true

--> Pod image: image-registry.openshift-image-registry.svc:5000/myproject/myimage-internal@sha256:ff3b3c5e... oc tag myproject/myimage-internal:mytag myproject/fromis-ref-local-myimage-internal:mytag --reference-policy=local

--> Pod image: image-registry.openshift-image-registry.svc:5000/myproject/fromis-ref-local-myimage-internal@sha256:ff3b3c5e...

The ImageStream yaml looks like this:

$ oc get is -oyaml fromis-ref-source-myimage-internal

apiVersion: image.openshift.io/v1

kind: ImageStream

metadata:

name: fromis-ref-source-myimage-internal

spec:

tags:

- from:

kind: ImageStreamImage

name: myimage-internal@sha256:ff3b3c5e...

namespace: myproject

name: mytag

status:

dockerImageRepository: image-registry.openshift-image-registry.svc:5000/myproject/fromis-ref-source-myimage-internal

tags:

- items:

- dockerImageReference: image-registry.openshift-image-registry.svc:5000/myproject/fromis-ref-source-myimage-internal@sha256:ff3b3c5e...

image: sha256:ff3b3c5e...

tag: mytag

Summary

ImageStreams are a great way to manage images stored in OpenShift’s internal registry, but some parameters can be confusing when they point to external urls. The release and increasing popularity of OpenShift 4 gave us a good reason to do an extensive overview and examine how ImageStream tags are resolved to an actual container image url when they are used in a deployment. Hopefully this helps to answer some questions you may run into while designing your own image management strategy.