Build Azure Pipelies with Azure DevOps in a Linux self-hosted environment

This article describes how to build a self-hosted agent environment that builds with a virtual machine prepared by Azure DevOps.

What are Azure DevOps? Or what can you do? Since other people have written in detail about, I will omit it here and touch only the main subject.

Why self-host?

In Azure DevOps, the service that builds is Azure Pipelines. By default, the build is done in a Microsoft-hosted environment. What's more, they are provided free of charge.

So why bother paying for the virtual machine and preparing your own self-hosted environment? This is because Microsoft-hosted environments have the following restrictions: (As of 02/20/2020)

The Microsoft-hosted agent limits are as follows:

--"At least" 10 GB of storage for source files and build output

• time limit --Open source project --10 parallel free jobs that can run up to 6 hours each time --There is no particular upper limit per month --Contact Microsoft to raise the limit --Closed source project --Parallel free jobs that can run up to 1 hour each time --Up to 30 hours per month --There will be a charge for raising the upper limit --The execution machine is a `DS2_v2'virtual machine
  • 2 vCPU
  • 7 GiB Memory
  • 14 GiB SSD
• Such
• Such

However, if you're using it in an open source project, you should be able to build with a Microsoft-hosted agent unless it's a very large project.

However, in enterprise systems, system development is often done from closed sources, and the one-hour constraint each time often does not have enough time to build and test. In such cases, self-hosting is a viable option.

In addition, ** self-hosting allows you to freely set the size of the virtual machine **, so you can use a large virtual machine as the build machine and reduce the overall build and test time.

What do you want to achieve in a self-hosted environment?

This time, what we want to achieve is simple. Allows Azure Pipelines to call agents to run in virtual machines that you set up yourself. The big picture looks like the following figure.

The sequence of flows is executed in numerical order in the figure.

1. The developer pushes the source code to the master branch
2. Azure Repos launches Azure Pipelines build sequence triggered by being pushed
3. Azure Pipelines asks a self-hosted agent on the virtual machine to build
4. The agent on the virtual machine builds the source code
5. The agent on the virtual machine notifies Azure Pipelines of the result regardless of the success or failure of the build.
6. Azure Pipelines emails developers the result of the build

Prerequisites

Now, let's set up the self-hosting environment from here, but here are some prerequisites to keep things simple.

First, the authority of the user who works must meet the following conditions.

1. Have permission to create Azure AD users and groups
2. Have the role of project administrator for Azure DevOps
3. You have permission to create Ubuntu 18.04 virtual machines

Next, we assume that you have some preparation for Azure resources as well.

1. You have already created an organization on Azure DevOps
2. You have already created an Azure DevOps project
3. You already have a repository for builds in Azure Repos
4. You have already set up resource groups and virtual machines for self-hosting. --You must be able to log in to Ubuntu Server 18.04 --You must have installed Git 2.9.0 or later --The user has sudo privileges --Not required for agent installation

The following is not required, but you may find it easier for later.

--Create a Key Vault in a resource group and the working user has permission to create a secret

Overview of work

It is important for future automation to understand the flow of work before going into individual work. Also, if you understand what to set or not to set when working, you will be able to understand the meaning of the work better, and you will be able to select what should be automated and divide the work.

The work to be done here can be roughly divided into the following three types.

1. Setting up and associating each Azure resource
2. Setting up an agent for a virtual machine
3. Create a new pipeline

As for ʻazure-pipelines.yml`, which is required for the last new pipeline, Ops may not be able to know the details depending on the build target, so you may ask Dev to describe it.

Setting up and associating each Azure resource

Here, we will mainly create what we need in Azure DevOps. Here's what to create:

--Azure Active Directory user --Personal access token

• Agent Pool

Then, I will explain step by step.

Creating a user to run the agent

The most important task this time is to associate Azure Pipelines with agents in a self-hosted environment. Otherwise, Azure Pipelines will not be able to determine which agent to call, and the agent will not be able to determine what task to perform.

This time, create a new user and register as an agent administrator for Azure Pipelines. Note that you can skip this step if you want to associate an agent administrator with an existing user.

However, due to the convenience of using the agent administrator's personal access token, it may be easier (maybe) to prepare a dedicated user to manage the token.

It's a shame, but don't worry about the different themes in the screenshots ...

Create Azure Active Directory user

First, we need a user to manage the Agent Queue, so create an Azure AD user with the following information:

In addition, edit the authentication contact information and enter the email address of your Azure AD administrator or administrator mailing list.

Add users to Azure DevOps

Add the user created in Azure Active Directory to Azure DevOps. A user with Azure DevOps admin rights signs in to Azure DevOps and clicks ʻOrganization settings` in the following figure to open the organization settings screen.

Then select ʻUsers` in the left pane.

Finally, enter your information and click ʻAdd`.

Creating a personal access token for an agent admin user

Sign in to Azure DevOps with the user you just created. After signing in, edit your profile to keep your email address valid.

After signing in, select your project and click Personal access tokens from the profile at the top right of the screen.

Next, since no access token should have been issued at this stage, click New Token in the right pane.

An input dialog opens on the right side of the screen, so enter the required information.

Set permissions for the following scopes.

When you click Create, the personal access token will be displayed as shown below. Click the copy icon and save it somewhere.

Please note that ** Azure DevOps does not store this personal access token **. Be sure to save it somewhere so you can forget it.

Here's a tip, but it's a good idea to manage both your password and your personal access token as a Key Vault secret. If you associate this with a custom policy that checks the expiration date of Key Vault and audit it, you can issue a new personal access token and always use a valid token before the personal access token expires. I will.

Actually, you only need a personal access token when you create it ...?

Creating an Agent Pool

Now, here is the creation of the Agent Pool, which is the point of this time.

Select a project and click Project settings.

Click ʻAgent pools` in the left pane.

Click ʻAdd pool` in the upper right corner of the right pane.

Enter the required information and click Create. This time we'll name it Hosted Agent Pool.

The Agent Pool has been created, so click it.

Click New Agent in the right pane.

A dialog for the Agent will appear. Click the Linux tab, make sure the platform is x64, click the copy icon and save the download link.

Setting up an agent for a virtual machine

To download the agent earlier, follow these steps:

1. Log in to the virtual machine you have already prepared.
2. Create a directory called myagent.
3. Use the download link to download the agent and uncompress the compressed file.

The command is:

mkdir myagent && cd myagent
wget https://vstsagentpackage.azureedge.net/agent/2.164.8/vsts-agent-linux-x64-2.164.8.tar.gz
tar zxvf vsts-agent-linux-x64-2.164.8.tar.gz

Next, configure the agent.

./config.sh

You will see a prompt similar to the following:

  ___                      ______ _            _ _
 / _ \                     | ___ (_)          | (_)
/ /_\ \_____   _ _ __ ___  | |_/ /_ _ __   ___| |_ _ __   ___  ___
|  _  |_  / | | | '__/ _ \ |  __/| | '_ \ / _ \ | | '_ \ / _ \/ __|
| | | |/ /| |_| | | |  __/ | |   | | |_) |  __/ | | | | |  __/\__ \
\_| |_/___|\__,_|_|  \___| \_|   |_| .__/ \___|_|_|_| |_|\___||___/
                                   | |
        agent v2.164.8             |_|          (commit 99c93e1)


>>License agreement:

To build the source from the TFVC repository, you must accept the Team Explorer Everywhere license agreement. This step is not necessary if you are building the source from a Git repository.

You can find a copy of the Team Explorer Everywhere license agreement here:
  /home/hayashi_toshiki/myagent/externals/tee/license.html

(Y/N)Do you accept the Team Explorer Everywhere license agreement now??Enter(If N, press Enter) >

The input procedure is as follows.

1. Type Y and press ʻEnter`.
2. Enter the URL of the server. The value is https://dev.azure.com/ {organization name}. After typing, press ʻEnter`.
3. Enter the authentication type. Since we are using a personal access token, just press ʻEnter`.
4. Enter the agent pool. Type Hosted Agent Pool and press ʻEnter`.
5. Enter the agent name. We will use the default, so just press ʻEnter`.
6. Enter the working folder. We will use the default, so just press ʻEnter`.

Service registration

Register as a service (Daemon) so that the virtual machine agent can run automatically.

The command to register the service is:

cd ~/myagent
sudo ./svc.sh install

Then enter the command to start the service.

sudo ./svc.sh start

This completes the agent setup on the virtual machine.

The command to get the service status is as follows.

sudo ./svc.sh status

Creating a new pipeline

Create a new pipeline to verify that Azure Pipelines and the self-hosted agent are properly associated.

In Azure DevOps, click Builds from Pipelines in the left pane.

Click New pipeline in the right pane.

This time we're going to browse the Azure Repos repository, so click ʻAzure Repos Git`.

Then you will be prompted to select a repository. Please select any repository.

Next, create a new ʻazure-pipelines.yml? Or do you use an existing one? Select about. This time we'll create a simple YAML file, so select Starter pipeline`.

Finally, the YAML file edit page will appear. Edit it as follows.

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- master

# ----------------------------------------------------
#It is important to specify the name of the Agent Pool here.
# ----------------------------------------------------
pool: Hosted Agent Pool

steps:
- script: echo Hello, world!
  displayName: 'Run a one-line script'

- script: |
    echo Add other tasks to build, test, and deploy your project.
    echo See https://aka.ms/yaml
  displayName: 'Run a multi-line script'

Click Save and run to save the YAML and run the pipeline.

A dialog appears in which you can enter the commit message for the edited YAML file. Just click Save and run.

After a while, the Job will run and you will see a page similar to the following, indicating that the Hosted Agent Pool has successfully built.

You've now verified that the agent in your self-hosted environment and Azure Pipelines are working together.

Finally

This time, I explained how to set up an agent in a self-hosted environment and use your own build environment.

The following topics are not mentioned for the sake of brevity in this article.

--Windows self-hosted environment --How to write ʻazure-pipelines.yml` --How to deploy --Test method --Setup automation

I would like to write about them if I have another chance.

If you're thinking of trying out Azure DevOps, we hope you find it helpful.