notation auth to CNCF ORAS registry

[jimmyraywv]

I am using the CNCF ORAS registry distribution in k8s. I have setup basic auth with htpasswd. The oras CLI can authenticate with the registry using -u ... -p ... flags. However, the notation 0.7.1-alpha.1 CLI fails auth with the same -u ... -p ... flags. Can someone suggest what I am missing? The command I am using is below:

This succeeds:

oras discover -o tree $IMAGE  -u <USERNAME> -p <PASSWORD>

This does not succeed:

notation sign $IMAGE -u <USERNAME> -p <PASSWORD>
2022/05/04 19:03:32 https://<HOST>:443/v2/<REPO>/manifests/<VERSION>: 401 Unauthorized

[jimmyraywv]

Looking into the ORAS logs, I see the following error: invalid authorization credential

According to the ORAS auth error, // ErrInvalidCredential is returned when the auth token does not authenticate correctly.

[ams0]

I have the same, tried 0.7.0-alpha.1 and it doesn't help. ACR in South Central with zone redundancy enabled.

[jimmyraywv]

Maybe it just doesn't work?

#119

[ams0]

but I would expect to work for GHCR?

notation sign ghcr.io/ams0/net-monitor:v1 -u ams0 -p $GITHUB_PAT

[qmuntal]

It works for me. Can you try setting the image path at the very end of the command?

notation sign -u ams0 -p $GITHUB_PAT ghcr.io/ams0/net-monitor:v1

@dtzar

[jimmyraywv]

I will test with the ORAS registry, again.

[jimmyraywv]

@qmuntal The results of my testing are below:

notation sign $IMAGE -u <USERNAME> -p <PASSWORD>
2022/05/31 10:39:43 https://<HOST>:443/v2/<IMAGE>/manifests/<TAG>: 401 Unauthorized

notation sign -u <USERNAME> -p <PASSWORD> $IMAGE
2022/05/31 10:41:31 fail to push signature to "<HOST>:443/<IMAGE>:<TAG>": sha256:...: push signature failure: failed to upload: 400 Bad Request

After the Bad Request failure, I see the invalid authorization credential error in the oras registry logs:

{
  "environment": "dev",
  "go.version": "go1.15.13",
  "http.request.contenttype": "application/octet-stream",
  "http.request.host": "<HOST>",
  "http.request.id": "482bc32a-7750-4644-a2fb-596e73424319",
  "http.request.method": "PUT",
  "http.request.remoteaddr": "<IP>",
  "http.request.uri": "/v2/<IMAGE>/blobs/uploads/...",
  "http.request.useragent": "Go-http-client/2.0",
  "instance.id": "aa4a4648-b277-4655-900a-7131794ea575",
  "level": "warning",
  "msg": "error authorizing context: basic authentication challenge for realm \"<REALM>\": invalid authorization credential",
  "service": "registry",
  "time": "2022-05-31T14:36:52.867871593Z",
  "vars.name": "<IMAGE>",
  "vars.uuid": "9c7d4b00-22fe-4ef9-b676-ef0b3b895490",
  "version": "v0.0.3-alpha"
}

The oras client and docker client logins work fine against the oras registry. I don't know what is different for the notation client:

oras discover -o tree $IMAGE  -u <USERNAME> -p <PASSWORD>

cat pfile | docker login <HOST> -u <USERNAME> --password-stdin

[gokarnm]

@jimmyraywv a new alpha release of notation is now available, that includes ORAS client integration. This would be a good version to test registry authentication using ORAS.

[dtzar]

At least in part there is an issue when you use $IMAGE before the positional parameters (i.e. -u -p). Please see this issue: #183

@jimmyraywv @ams0 agree with gokarnm - we made a number of improvements to the authentication in the new release which is out today. Can you please try with this?

[jimmyraywv]

@gokarnm @dtzar Please see my results below:

With oras registry basic auth enabled:
oras client version:

oras version
Version:        0.2.1-alpha.1
Go version:     go1.16.2
Git commit:     58fa344544b34571012d91f523cea54bf364549b
Git tree state: clean

oras call without creds (fails):

oras discover <HOST>/<IMAGE>:<IMAGE_TAG>
Error: unexpected status code [manifests <IMAGE_TAG>]: 401 Unauthorized

oras call with creds:

oras discover <HOST>/<IMAGE>:<IMAGE_TAG> -u <USERNAME> -p <PASSWORD>
Discovered 0 artifacts referencing <HOST>/<IMAGE>:<IMAGE_TAG>
Digest: sha256:0d91813...

notation client version:

notation --version
notation version 0.9.0-alpha.1

notation sign without creds (fails):

notation sign $IMAGE
2022/06/02 09:46:10 HEAD "https://<HOST>:443/v2/<IMAGE>/manifests/<IMAGE_TAG>": credential required for basic auth

notation sign with incorrect creds (fails):

notation sign -u <USERNAME> -p <WRONG_PASSWORD> $IMAGE
2022/06/02 10:11:13 HEAD "https://<HOST>:443/v2/<IMAGE>/manifests/<IMAGE_TAG>": unexpected status code 401: Unauthorized

notation sign with creds (fails):

notation sign -u <USERNAME> -p <PASSWORD> $IMAGE
2022/06/02 09:47:47 fail to push signature to "<HOST>:443/<IMAGE>:<IMAGE_TAG>": sha256:0d9181386b...: push signature failure: PUT "https://<HOST>/v2/<IMAGE>/blobs/uploads/b6e75f48-0258-4c32-8832-1f9f60934404?_state=28rHOzCLeKtW8v2rr6pCQ1U6rTj8xqIuO-l-PNv-nwp7Ik5hbWUiOiJnby1odHRwLXNlcnZlciIsIlVVSUQiOiJiNmU3NWY0OC0wMjU4LTRjMzItODgzMi0xZjlmNjA5MzQ0MDQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjItMDYtMDJUMTM6NDc6NDcuNTI4MTQxMjM2WiJ9&digest=sha256%3Ac57bd63dd69ea40a146287afc12441076ef5b4e17e748650fc09d84cee37184b": credential required for basic auth

error msg in oras registry logs:

{
  "environment": "dev",
  "go.version": "go1.15.13",
  "http.request.contenttype": "application/octet-stream",
  "http.request.host": "<HOST>",
  "http.request.id": "3eaf822f-0a98-416e-8ce8-a69f395e337c",
  "http.request.method": "PUT",
  "http.request.remoteaddr": "<IP>",
  "http.request.uri": "/v2/<IMAGE>/blobs/uploads/e6655723-7327-415e-997b-d0ec535701ee?_state=kzXFfj-vAdlXCQsOYV5GeMH_Siv6Ml1qa-mwCV46cNN7Ik5hbWUiOiJnby1odHRwLXNlcnZlciIsIlVVSUQiOiJlNjY1NTcyMy03MzI3LTQxNWUtOTk3Yi1kMGVjNTM1NzAxZWUiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjItMDYtMDJUMTM6MzY6MTYuNzA2MTA3ODg4WiJ9&digest=sha256%3Ae1111ba9566784439c012b7e63bc98488091928e0949617e533971df1cf53aeb",
  "http.request.useragent": "notation/0.9.0-alpha.1",
  "instance.id": "aa4a4648-b277-4655-900a-7131794ea575",
  "level": "warning",
  "msg": "error authorizing context: basic authentication challenge for realm \"everyone\": invalid authorization credential",
  "service": "registry",
  "time": "2022-06-02T13:36:16.752860072Z",
  "vars.name": "<IMAGE>",
  "vars.uuid": "e6655723-7327-415e-997b-d0ec535701ee",
  "version": "v0.0.3-alpha"
}

If I disable oras registry basic auth, I see this:

docker push <HOST>/<IMAGE>:<IMAGE_TAG>
The push refers to repository [<HOST>/<IMAGE>]
90ec0fff6942: Pushed
bb1aaf5ef092: Pushed
aa45eda38212: Pushed
<IMAGE_TAG>: digest: sha256:0d9181386b3fbcbcc01a04728e592d427a6e780fe80befedb7c170a78846d4c4 size: 945

oras discover <HOST>/<IMAGE>:<IMAGE_TAG>
Discovered 0 artifacts referencing <HOST>/<IMAGE>:<IMAGE_TAG>
Digest: sha256:0d9181386b3fbcbcc01a04728e592d427a6e780fe80befedb7c170a78846d4c4

notation sign $IMAGE
sha256:0d9181386b3fbcbcc01a04728e592d427a6e780fe80befedb7c170a78846d4c4

oras discover <HOST>/<IMAGE>:<IMAGE_TAG>
Discovered 1 artifacts referencing <HOST>/<IMAGE>:<IMAGE_TAG>
Digest: sha256:0d9181386b3fbcbcc01a04728e592d427a6e780fe80befedb7c170a78846d4c4

Artifact Type                              Digest
application/vnd.cncf.notary.v2.signature   sha256:86bcb6c714f0445e1c944bbebb56b5ab4d4e684f92561c0171c2ae37f3021ae9

[dtzar]

@jimmyraywv thanks for sharing your results!

If you run this and then try a notation sign $IMAGE does this work for you?

    export USER_NAME="00000000-0000-0000-0000-000000000000"
    export PASSWORD=$(az acr login --name $ACR_NAME --expose-token --output tsv --query accessToken)
    export NOTATION_USERNAME=$USER_NAME
    export NOTATION_PASSWORD=$PASSWORD

@shizhMSFT @FeynmanZhou

[shizhMSFT]

@jimmyraywv Could you share more details on the registry? Like what registry are you using so that we can test it out?

For ACR registries, I've tested various login methods and they works fine.

Admin access:

USER_NAME=$(az acr credential show -n $ACR_NAME --output tsv --query username)
PASSWORD=$(az acr credential show -n shizhnv2 --output tsv --query passwords[0].value)
notation sign -u $USER_NAME -p $PASSWORD $IMAGE

Token access:

TOKEN=$(az acr login --name $ACR_NAME --expose-token --output tsv --query accessToken)
notation sign -p $TOKEN $IMAGE

Token access via basic auth:

USER_NAME=00000000-0000-0000-0000-000000000000
PASSWORD=$(az acr login --name $ACR_NAME --expose-token --output tsv --query accessToken)
notation sign -u $USER_NAME -p $PASSWORD $IMAGE

[shizhMSFT]

Note: If you are using zsh, please make sure you are using USER_NAME instead of USERNAME for the environment variables as USERNAME is reserved and immutable in zsh.

[jimmyraywv]

I don't have an Azure account

[akcrisp]

hi, can I suggest that the above re azure token - gets properly added to the documentation. I was struggling to get notation working with azure acr - until i saw the above - there's no mention of it following oras login process (which I had already done). It would be useful if made clear as well that the USER_NAME is a value and not a placeholder to be replaced (as i initially thought) or can you even remove it entirely as oras do...

[shizhMSFT]

I'm able to reproduce the issue using ghcr.io/oras-project/registry:v1.0.0-rc with basic auth enabled. Investigating...

[shizhMSFT]

The error message is unexpected status code 500: unknown: unknown error but it is not related to auth as I missed distribution extensions in the config file for https://github.com/oras-project/distribution.

[shizhMSFT]

I've tested 0.9.0-alpha.1 against the basic auth enabled registry, and it works fine.

[jimmyraywv]

Ok, well, I am not sure where to proceed at this point. I documented my findings. oras and docker clients authenticate just fine, notation will not. I am running this oras registry in an EKS cluster, behind an external-facing ALB, with TLS enabled. I am not sure if that should matter; it didn't seem to negatively impact the oras or docker clis.

I tried to use a later oras registry, ghcr.io/oras-project/registry:v1.0.0-rc, but it looks like it uses the latest artifact spec, that is not yet supported in the oras cli. I guess I am blocked for now.

[shizhMSFT]

After a live debug session with @jimmyraywv, it turns out we have a bug in the oras-go.

See the bug details and the workaround below

• Fail to push if port 443 is explicitly specified in the hostname oras-project/oras-go#177

[jimmyraywv]

@shizhMSFT Just in case you wonder why I explicitly used the port, to begin with, I was following this: https://medium.com/@LachlanEvenson/container-signing-with-notary-v2-13dd425e95d6

[dtzar]

@shizhMSFT solved this problem: #170 (reply in thread)