Verify Images

Sigstore is a Linux Foundation project focused on software signing and transparency log technologies to improve software supply chain security. Cosign is a sub-project that provides image signing, verification, and storage in an OCI registry.

The Kyverno verifyImages rule uses Cosign to verify container image signatures stored in an OCI registry. The rule matches an image reference (wildcards are supported) and specifies a public key to be used to verify the signed image. The policy rule check fails if the image signature is not found in the OCI registry, or if the image was not signed using the specified key.

The rule also mutates matching images to add the image digest if the digest is not already specified. Using an image digest has the benefit of making image references immutable. This helps ensure that the version of the deployed image does not change and, for example, is the same version that was scanned and verified by a vulnerability scanning and detection tool.

The imageVerify rule executes as part of the mutation webhook as the applying policy may insert the image digest. The imageVerify rules execute after other mutation rules are applied but before the validation webhook is invoked. This order allows other policy rules to first mutate the image reference if necessary, for example, to replace the registry address, before the image signature is verified.

The imageVerify rule can be combined with auto-gen so that policy rule checks are applied to Pod controllers.

Here is a sample image verification policy:

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: check-image
 5spec:
 6  validationFailureAction: enforce
 7  background: false
 8  rules:
 9    - name: check-image
10      match:
11        resources:
12          kinds:
13            - Pod
14      verifyImages:
15      - image: "ghcr.io/kyverno/test-verify-image:*"
16        key: |-
17          -----BEGIN PUBLIC KEY-----
18          MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8nXRh950IZbRj8Ra/N9sbqOPZrfM
19          5/KAQN0/KjHcorm/J5yctVd7iEcnessRQjU917hmKO6JWVGHpDguIyakZA==
20          -----END PUBLIC KEY-----          

This policy will validate that all images that match ghcr.io/kyverno/test-verify-image:* are signed with the specified key.

A signed image can be run as follows:

1kubectl run signed --image=ghcr.io/kyverno/test-verify-image:signed
2pod/signed created

The deployed Pod will be mutated to use the image digest.

Attempting to run an unsigned image will produce a policy error as follows:

1kubectl run unsigned --image=ghcr.io/kyverno/test-verify-image:unsigned
2Error from server: admission webhook "mutate.kyverno.svc" denied the request:
3
4resource Pod/default/unsigned was blocked due to the following policies
5
6check-image:
7  check-image: 'image verification failed for ghcr.io/kyverno/test-verify-image:unsigned:
8    signature not found'

Similarly, attempting to run an image which matches the specified rule but is signed with a different key will produce an error:

1kubectl run signed-other --image=ghcr.io/kyverno/test-verify-image:signed-by-someone-else
2Error from server: admission webhook "mutate.kyverno.svc" denied the request:
3
4resource Pod/default/signed-other was blocked due to the following policies
5
6check-image:
7  check-image: 'image verification failed for ghcr.io/kyverno/test-verify-image:signed-by-someone-else:
8    invalid signature'

Signing images

To sign images, install Cosign and generate a public-private key pair.

1cosign generate-key-pair

Next, use the cosign sign command and specifying the private key in the -key command line argument.

1# ${IMAGE} is REPOSITORY/PATH/NAME:TAG
2cosign sign -key cosign.key ${IMAGE}

This command will sign your image and publish the signature to the OCI registry. You can verify the signature using the cosign -verify command.

1cosign verify -key cosign.pub ${IMAGE}

Refer to the Cosign documentation for usage details and OCI registry support.

Using private registries

To use a private registry, you must create an image pull secret in the Kyverno namespace and specify the secret name as an argument for the Kyverno deployment:

1. Configure the image pull secret:

1kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-password> --docker-email=<your-email> 
2-n kyverno

2. Update the Kyverno deployment to add the --imagePullSecrets=regcred argument:

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  labels:
 5    app.kubernetes.io/component: kyverno
 6   ...
 7
 8
 9spec:
10  replicas: 1
11  selector:
12    matchLabels:
13      app: kyverno
14      app.kubernetes.io/name: kyverno
15  template:
16    spec:
17      containers:
18      - args:
19        ...
20        - --webhooktimeout=15
21        - --imagePullSecrets=regcred

Known Issues

1. Some registry calls can take a few seconds to complete. Hence, the webhook timeout should be set to a higher number such as 15 seconds.

2. Prometheus metrics and the Kyverno CLI are currently not supported. Check the Kyverno GitHub for a complete list of pending issues.