[PYTHON] Create a private repository with AWS CodeArtifact

June 2020 AWS Code Brothers suddenly reborn. Its name is AWS CodeArtifact. A private repository service that allows you to securely store, publish, and share software packages. It is already generally available in the Tokyo region and is also compatible with VPC Endpoint.

:rocket: Introducing AWS CodeArtifact: A fully managed software artifact repository service https://aws.amazon.com/jp/about-aws/whats-new/2020/06/introducing-aws-codeartifact-a-fully-managed-software-artifact-repository-service/

Similar tools and services include JFrog Artifactory and GitHub Package Registry, I think Azure Artifacts is famous.

What are you happy about

There are two main benefits to repository managers like AWS CodeArtifact.

1. Official repository cache

When developing software, create an ecosystem such as package managers for various languages It has become commonplace to use it.

CodeArtifact can be used as a proxy for corresponding package managers such as maven and npm. Administrators can control access to packages and are approved for use within the organization Only the latest packages will be available to developers.

Also, depending on your organization's policies, you may not be able to freely access public repositories. By using VPC Endpoint, it was cached in CodeArtifact without going through the Internet. You can access the package. You can also expect a reduction in download time.

As of June 2020, the following external connections are supported. It seems to be the minimum compared to other companies' services, but I look forward to future updates.

2. Manage your own package

CodeArtifact provides a central place to store packages built within your organization. Because the stored packages can be safely shared and made available to existing package managers This will reduce development time.

Important concepts to remember in CodeArtifact

Repository

You can create multiple repositories within your account. (Maximum 100) A repository can contain packages in multiple supported languages. The repository is the unit of endpoint when accessed from each package manager.

domain

CodeArtifact is a unique concept. Manage each repository in a single domain.

Each package is used via the repository, but the actual data is not the repository It is saved in the domain. Specific packages in multiple repositories within the same domain Even if you use it, it will be saved only once, regardless of the number of repositories. In other words, storage charges will not be duplicated.

Domains are also available for cross-accounts. Create a common domain for your organization Each account can use its own repository.

Upstream repository

You can set other repositories as upstream repositories for each repository. This makes it a common repository within the organization, a repository for development teams, etc., depending on the application. Clients can still access from a single endpoint, even if they are separated.

Domain policy and repository policy

It can be set in the familiar json format.

Domain policy settings for accounts within your organization or specific IAM principals You can allow access to the domain.

Similarly, if you define a resource policy, for users who can access the repository and for the repository You can specify the actions that can be performed. As a result, in the case of the above configuration example, only read permission from within the organization is set in the organization shared repository. The development team repository can be configured to allow only development members to read and write.

Try

Creating domains and repositories

Select Create Repository at the top of the CodeArtifact console. Enter the repository name and in the public upstream repository settings Select the official repository to cache. Then select the domain where you want to register the repository. This account has not created a domain yet, so specify a domain name and create a new one. Additional settings allow you to choose the KMS key to use for encryption, either managed or specific CMK. Finally, check the settings and create the repository and domain. As you can see from the package flow, in the repository creation wizard Only public upstream repositories are available for selection. To set another repository as an upstream repository, edit after creating the repository Must be set.

How to connect to the repository

AWS CLI v2.0.21 or higher is required to get the connection information of the created repository. You can check the specific connection command for each tool from the repository screen of the console. For example, in the case of pip: The --domain-owner option is optional for the account that created the domain.

$ aws codeartifact login --tool pip --repository my-repo --domain my-domain --domain-owner 0123456789012
Successfully logged in to codeartifact for pip.

Internally, it seems that pip config set automatically sets global.index-url. You can see the repository endpoints with credentials in the pip config list. This credential expires after 12 hours and must be done each time.

Alternatively, use the aws codeartifact get-authorization-token command to get only the authentication token. You can also do a manual pip config set.

It seems that the logout command is not prepared, so to cancel it There seems to be no choice but to run pip config unset global.index-url.

Public repository cache

If you try installing the AWS SAM CLI, you will see that it is downloaded from CodeArtifact.

$ pip install -U aws-sam-cli
Collecting aws-sam-cli
  Downloading https://my-domain-123456789012.d.codeartifact.ap-northeast-1.amazonaws.com/pypi/my-repo/simple/aws-sam-cli/0.52.0/aws_sam_cli-0.52.0-py3-none-any.whl (443kB)

You can also see that it is registered in the created repository on the console. In fact, the public upstream repository, pypi-store, connects to and caches PyPI.

Sharing your own package

Here is provided for the Python Packaging User Guide tutorial Try uploading the sample project to CodeArtifact.

Clone https://github.com/pypa/sampleproject.

$ git clone https://github.com/pypa/sampleproject.git
Cloning into 'sampleproject'...
remote: Enumerating objects: 21, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (19/19), done.
remote: Total 441 (delta 7), reused 6 (delta 1), pack-reused 420
Receiving objects: 100% (441/441), 116.64 KiB | 365.00 KiB/s, done.
Resolving deltas: 100% (216/216), done.

Package with the following command.

$ cd sampleproject
$ python setup.py bdist_wheel

For Python, twine is required to upload the package, so install it.

$ pip install -U twine

Set the twine credentials with the login command. Internally, the settings for CodeArtifact connection are added to ~ / .pypirc.

$ aws codeartifact login --tool twine --repository my-repo --domain my-domain
Successfully logged in to codeartifact for twine.

Upload to CodeArtifact with the following command.

$ twine upload -r codeartifact dist/*
Uploading distributions to https://my-domain-123456789012.d.codeartifact.ap-northeast-1.amazonaws.com/pypi/my-repo/
Uploading sampleproject-1.3.1-py3-none-any.whl

I was able to confirm that it was uploaded on the console as well. You can easily share your own package.

reference

** AWS CodeArtifact product page ** https://aws.amazon.com/jp/codeartifact/ CodeArtifact User Guide https://docs.aws.amazon.com/codeartifact/latest/ug/welcome.html ** Publish / upload a library created in Python to PyPI ** https://qiita.com/icoxfog417/items/edba14600323df6bf5e0

that's all. I'm glad if you can use it as a reference.