Checked Docker Hub's Download rate limit on Circle CI

Conclusion

As of 11/03/2020 17:00, the 100 times / 6 hours limit had not yet been applied. According to the Official Announcement 11/05 02:00 JST It seems that the application will be completed at -05: 00.

(As of 2020/11/03 17:00) rate_limit2.png

The top is no credentials (anonymous user), and the bottom is the result of checking with my personal Docker Hub account. Both were limited, but up to 5000 times in 6 hours. By the way, CircleCI [with some exceptions, users have nothing to do right now](https://support.circleci.com/hc/en-us/articles/360050623311-Docker-Hub-rate-limiting- FAQ) has been announced. After the download rate limit is fully applied, I'm wondering if this job will show up to 100 times (for anonymous users) but it will feel like I can run the job without any problems.

Download rate limit and CircleCI

Docker Hub pull limit seems to be applied sequentially. Although there was an official announcement that it will be introduced on 2020/11/01, it seems that it will be gradually introduced from 11/02 as stated in the document. I haven't been able to confirm that the limit of 100 times in 6 hours has been activated as of 11/03 17:00 on my hand or on CircleCI.

** CircleCI users often don't have to deal with it right away. ** ** ** However, if the following cases apply, please read the Official FAQ. Better. ** ** (For some reason, as of 11/03/2020 [Japanese page](https://support.circleci.com/hc/ja/articles/360050623311-Docker-Hub%E3%81%AE%E3%83%AC% E3% 83% BC% E3% 83% 88% E5% 88% B6% E9% 99% 90% E3% 81% AB% E9% 96% A2% E3% 81% 99% E3% 82% 8B FAQ) Is old, so you should refer to the English page)

However, CircleCI does not deny that this limitation may apply to users in the future and recommends pulling the image with the appropriate credentials. For information on how to set credentials, see Official Guide and Detailed Articles Please see / 30a9728973b3dbca3f76).

How to find Download rate limit

Detailed explanation is left to Official explanation. Roughly speaking, if you get a token and make an HTTP request to a specific URL, it will tell you the current download restriction information.

$ TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token
$ curl -v -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest 2>&1 | grep RateLimit

** * Note: The curl request on the second line is considered as one image download. Don't do it blindly. ** ** DockerHub seems to consider the GET method request to the path / v2 / * / manifests / * as a single download.

When executed, the following result will be returned.

< RateLimit-Limit: 5000;w=21600
< RateLimit-Remaining: 4999;w=21600

The limit is 5000 times, the remaining number of times is 4999 times, and w is 21600s (6 hours) in the time window. The one consumed is due to the request itself asking for the remaining number.

CircleCI configuration file

.circleci/config.yml


version: 2.1

workflows:
  workflow:
    jobs:
      - build:
          context:
            - dockerhub-context
jobs:
  build:
    docker:
      - image: circleci/node:latest
        auth:
          username: $DOCKERHUB_USERNAME
          password: $DOCKERHUB_PASSWORD
    resource_class: small
    steps:
      - checkout
      - run: 
          name: Check rate limits with Circle CI credentials
          command: "curl -v -H \"Authorization: Bearer `curl 'https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull' | jq -r .token`\" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest 2>&1 | grep RateLimit"
      - run:
          name: Check rate limits with my own credentials
          command: "curl -v -H \"Authorization: Bearer `curl --user \"$DOCKERHUB_USERNAME:$DOCKERHUB_PASSWORD\" 'https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull' | jq -r .token`\" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest 2>&1 | grep RateLimit"

I wrote it like this. Since it was troublesome to set variables in the command, token acquisition and remaining count query are performed on the same line. We also created a context called dockerhub-context and set the environment variables DOCKERHUB_USERNAME and DOCKERHUB_PASSWORD. It is better to use the API key that can be created from the account settings of Docker Hub instead of the password used for logging in to the web page. The result is the image at the beginning. I would like to try it again when the restrictions are fully enabled.

** * Of course, the curl request in this script is also considered as one download (twice in total). It's probably okay within CircleCI, but don't mess around too much. ** **

Digression

Even if you make an inquiry to https://registry-1.docker.io/v2/*/manifests/*, the HEAD method does not seem to consume the number of downloads. That means that RateLimit-Limit is header information, so you can inquire with curl --head! ?? I thought, but it didn't work.

$ curl --head -H "Authorization: Bearer `curl 'https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull' | jq -r .token`" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest 2>&1
HTTP/1.1 200 OK
Content-Length: 2782
Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
Docker-Content-Digest: sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:767a3815c34823b355bed31760d5fa3daca0aec2ce15b217c9cd83229e0e2020"
Date: Tue, 03 Nov 2020 08:55:59 GMT

reference

Recommended Posts

Checked Docker Hub's Download rate limit on Circle CI
I tried using ECR Public instead of Docker Hub (Docker Hub Download Rate Limit measures)
Docker Container pull has a rate limit enforced