Build Azure Pipelies with Azure DevOps in a Linux self-hosted environment

This article describes how to build a self-hosted agent environment that builds with a virtual machine prepared by Azure DevOps.

What are Azure DevOps? Or what can you do? Since other people have written in detail about, I will omit it here and touch only the main subject.

Why self-host?

In Azure DevOps, the service that builds is Azure Pipelines. By default, the build is done in a Microsoft-hosted environment. What's more, they are provided free of charge.

So why bother paying for the virtual machine and preparing your own self-hosted environment? This is because Microsoft-hosted environments have the following restrictions: (As of 02/20/2020)

The Microsoft-hosted agent limits are as follows:

--"At least" 10 GB of storage for source files and build output

However, if you're using it in an open source project, you should be able to build with a Microsoft-hosted agent unless it's a very large project.

However, in enterprise systems, system development is often done from closed sources, and the one-hour constraint each time often does not have enough time to build and test. In such cases, self-hosting is a viable option.

In addition, ** self-hosting allows you to freely set the size of the virtual machine **, so you can use a large virtual machine as the build machine and reduce the overall build and test time.

What do you want to achieve in a self-hosted environment?

This time, what we want to achieve is simple. Allows Azure Pipelines to call agents to run in virtual machines that you set up yourself. The big picture looks like the following figure.

セルフホスト環境

The sequence of flows is executed in numerical order in the figure.

  1. The developer pushes the source code to the master branch
  2. Azure Repos launches Azure Pipelines build sequence triggered by being pushed
  3. Azure Pipelines asks a self-hosted agent on the virtual machine to build
  4. The agent on the virtual machine builds the source code
  5. The agent on the virtual machine notifies Azure Pipelines of the result regardless of the success or failure of the build.
  6. Azure Pipelines emails developers the result of the build

Prerequisites

Now, let's set up the self-hosting environment from here, but here are some prerequisites to keep things simple.

First, the authority of the user who works must meet the following conditions.

  1. Have permission to create Azure AD users and groups
  2. Have the role of project administrator for Azure DevOps
  3. You have permission to create Ubuntu 18.04 virtual machines

Next, we assume that you have some preparation for Azure resources as well.

  1. You have already created an organization on Azure DevOps
  2. You have already created an Azure DevOps project
  3. You already have a repository for builds in Azure Repos
  4. You have already set up resource groups and virtual machines for self-hosting. --You must be able to log in to Ubuntu Server 18.04 --You must have installed Git 2.9.0 or later --The user has sudo privileges --Not required for agent installation

The following is not required, but you may find it easier for later.

--Create a Key Vault in a resource group and the working user has permission to create a secret

Overview of work

It is important for future automation to understand the flow of work before going into individual work. Also, if you understand what to set or not to set when working, you will be able to understand the meaning of the work better, and you will be able to select what should be automated and divide the work.

The work to be done here can be roughly divided into the following three types.

  1. Setting up and associating each Azure resource
  2. Setting up an agent for a virtual machine
  3. Create a new pipeline

As for ʻazure-pipelines.yml`, which is required for the last new pipeline, Ops may not be able to know the details depending on the build target, so you may ask Dev to describe it.

Setting up and associating each Azure resource

Here, we will mainly create what we need in Azure DevOps. Here's what to create:

--Azure Active Directory user --Personal access token

Then, I will explain step by step.

Creating a user to run the agent

The most important task this time is to associate Azure Pipelines with agents in a self-hosted environment. Otherwise, Azure Pipelines will not be able to determine which agent to call, and the agent will not be able to determine what task to perform.

This time, create a new user and register as an agent administrator for Azure Pipelines. Note that you can skip this step if you want to associate an agent administrator with an existing user.

However, due to the convenience of using the agent administrator's personal access token, it may be easier (maybe) to prepare a dedicated user to manage the token.

It's a shame, but don't worry about the different themes in the screenshots ...

Create Azure Active Directory user

First, we need a user to manage the Agent Queue, so create an Azure AD user with the following information:

Setting items Set value
username Azure-Pipelines-Agent-Queue-Admin-001
name Azure Pipelines Queue Admin 001
Name Admin - 001
sex Azure Pipelines Agent Queue
password Any password
Place of use Japan

In addition, edit the authentication contact information and enter the email address of your Azure AD administrator or administrator mailing list.

Add users to Azure DevOps

Add the user created in Azure Active Directory to Azure DevOps. A user with Azure DevOps admin rights signs in to Azure DevOps and clicks ʻOrganization settings` in the following figure to open the organization settings screen.

Organization settings

Then select ʻUsers` in the left pane.

Users

Finally, enter your information and click ʻAdd`.

Setting items Set value
Users The Azure AD user you just created
Access level Basic
Add to projects Project name to add
Azure DevOps Groups Project Contributors

User information

Creating a personal access token for an agent admin user

Sign in to Azure DevOps with the user you just created. After signing in, edit your profile to keep your email address valid.

After signing in, select your project and click Personal access tokens from the profile at the top right of the screen.

User Profile

Next, since no access token should have been issued at this stage, click New Token in the right pane.

New Token

An input dialog opens on the right side of the screen, so enter the required information.

Config Token

Set permissions for the following scopes.

scope Set value
Agent Pools Read & manage
Build Read & execute
Packaging Read, write, & manage
Release Read, write, execute, & manage
Test Management Read & write

When you click Create, the personal access token will be displayed as shown below. Click the copy icon and save it somewhere.

Success Creation PAT

Please note that ** Azure DevOps does not store this personal access token **. Be sure to save it somewhere so you can forget it.

Here's a tip, but it's a good idea to manage both your password and your personal access token as a Key Vault secret. If you associate this with a custom policy that checks the expiration date of Key Vault and audit it, you can issue a new personal access token and always use a valid token before the personal access token expires. I will.

Actually, you only need a personal access token when you create it ...?

Creating an Agent Pool

Now, here is the creation of the Agent Pool, which is the point of this time.

Select a project and click Project settings.

Project settings

Click ʻAgent pools` in the left pane.

Agent pools

Click ʻAdd pool` in the upper right corner of the right pane.

Add pool

Enter the required information and click Create. This time we'll name it Hosted Agent Pool.

Config pool

The Agent Pool has been created, so click it.

Hosted Agent Pool

Click New Agent in the right pane.

New Agent

A dialog for the Agent will appear. Click the Linux tab, make sure the platform is x64, click the copy icon and save the download link.

Copy Download Link

Setting up an agent for a virtual machine

To download the agent earlier, follow these steps:

  1. Log in to the virtual machine you have already prepared.
  2. Create a directory called myagent.
  3. Use the download link to download the agent and uncompress the compressed file.

The command is:

mkdir myagent && cd myagent
wget https://vstsagentpackage.azureedge.net/agent/2.164.8/vsts-agent-linux-x64-2.164.8.tar.gz
tar zxvf vsts-agent-linux-x64-2.164.8.tar.gz

Next, configure the agent.

./config.sh

You will see a prompt similar to the following:


  ___                      ______ _            _ _
 / _ \                     | ___ (_)          | (_)
/ /_\ \_____   _ _ __ ___  | |_/ /_ _ __   ___| |_ _ __   ___  ___
|  _  |_  / | | | '__/ _ \ |  __/| | '_ \ / _ \ | | '_ \ / _ \/ __|
| | | |/ /| |_| | | |  __/ | |   | | |_) |  __/ | | | | |  __/\__ \
\_| |_/___|\__,_|_|  \___| \_|   |_| .__/ \___|_|_|_| |_|\___||___/
                                   | |
        agent v2.164.8             |_|          (commit 99c93e1)


>>License agreement:

To build the source from the TFVC repository, you must accept the Team Explorer Everywhere license agreement. This step is not necessary if you are building the source from a Git repository.

You can find a copy of the Team Explorer Everywhere license agreement here:
  /home/hayashi_toshiki/myagent/externals/tee/license.html

(Y/N)Do you accept the Team Explorer Everywhere license agreement now??Enter(If N, press Enter) >

The input procedure is as follows.

  1. Type Y and press ʻEnter`.
  2. Enter the URL of the server. The value is https://dev.azure.com/ {organization name}. After typing, press ʻEnter`.
  3. Enter the authentication type. Since we are using a personal access token, just press ʻEnter`.
  4. Enter the agent pool. Type Hosted Agent Pool and press ʻEnter`.
  5. Enter the agent name. We will use the default, so just press ʻEnter`.
  6. Enter the working folder. We will use the default, so just press ʻEnter`.

Service registration

Register as a service (Daemon) so that the virtual machine agent can run automatically.

The command to register the service is:

cd ~/myagent
sudo ./svc.sh install

Then enter the command to start the service.

sudo ./svc.sh start

This completes the agent setup on the virtual machine.

The command to get the service status is as follows.

sudo ./svc.sh status

Creating a new pipeline

Create a new pipeline to verify that Azure Pipelines and the self-hosted agent are properly associated.

In Azure DevOps, click Builds from Pipelines in the left pane.

Builds

Click New pipeline in the right pane.

New Pipeline

This time we're going to browse the Azure Repos repository, so click ʻAzure Repos Git`.

Azure Repos Git

Then you will be prompted to select a repository. Please select any repository.

Next, create a new ʻazure-pipelines.yml? Or do you use an existing one? Select about. This time we'll create a simple YAML file, so select Starter pipeline`.

DevOps-Pipelines-Configure-Edited.png

Finally, the YAML file edit page will appear. Edit it as follows.

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- master

# ----------------------------------------------------
#It is important to specify the name of the Agent Pool here.
# ----------------------------------------------------
pool: Hosted Agent Pool

steps:
- script: echo Hello, world!
  displayName: 'Run a one-line script'

- script: |
    echo Add other tasks to build, test, and deploy your project.
    echo See https://aka.ms/yaml
  displayName: 'Run a multi-line script'

Click Save and run to save the YAML and run the pipeline.

Save and Run

A dialog appears in which you can enter the commit message for the edited YAML file. Just click Save and run.

Run

After a while, the Job will run and you will see a page similar to the following, indicating that the Hosted Agent Pool has successfully built.

Build Succeeded

You've now verified that the agent in your self-hosted environment and Azure Pipelines are working together.

Finally

This time, I explained how to set up an agent in a self-hosted environment and use your own build environment.

The following topics are not mentioned for the sake of brevity in this article.

--Windows self-hosted environment --How to write ʻazure-pipelines.yml` --How to deploy --Test method --Setup automation

I would like to write about them if I have another chance.

If you're thinking of trying out Azure DevOps, we hope you find it helpful.

Recommended Posts

Build Azure Pipelies with Azure DevOps in a Linux self-hosted environment
[Linux] Build a jenkins environment with Docker
[Linux] Build a Docker environment with Amazon Linux 2
Build a Django environment with Vagrant in 5 minutes
Build a LAMP environment with Vagrant (Linux + Apache + MySQL + PHP)
Build a Selenium environment on Amazon Linux 2 in the shortest time
Building a Python3 environment with Amazon Linux2
Easily build a development environment with Laragon
Build a Tensorflow environment with Raspberry Pi [2020]
Build a Fast API environment with docker-compose
Build a python virtual environment with pyenv
Build a modern Python environment with Neovim
Build a CentOS Linux 8 environment with Docker and start Apache HTTP Server
Build a LAMP environment in a very short time
Build a C language development environment with a container
Build a python environment with ansible on centos6
Start Django in a virtual environment with Pipenv
Create a virtual environment with conda in Python
[Python] Build a Django development environment with Docker
Create a python3 build environment with Sublime Text3
[Memo] Build a virtual environment with Pyenv + anaconda
Set a fixed IP in the Linux environment
Build a virtual environment with pyenv and venv
Build a Django development environment with Doker Toolbox
Work in a virtual environment with Python virtualenv.
Build a Python environment with OSX El capitan
Build a Minecraft plugin development environment in Eclipse
Quickly build a Python Django environment with IntelliJ
Build a mruby development environment for ESP32 (Linux)
Build a Python machine learning environment with a container
Build a python execution environment with VS Code
Build a python virtual environment with virtualenv and virtualenvwrapper
Build a python environment for each directory with pyenv-virtualenv
[Linux] WSL2 Build an environment for laravel7 with Ubuntu 20.04
Build a machine learning application development environment with Python
Build a python virtual environment with virtualenv and virtualenvwrapper
Build a development environment with Poetry Django Docker Pycharm
How to build a Python environment on amazon linux 2
Build a Django environment for Win10 (with virtual space)
Build a numerical calculation environment with pyenv and miniconda3
I tried to build an environment with WSL + Ubuntu + VS Code in a Windows environment
Try running python in a Django environment created with pipenv
Install LAMP on Amazon Linux 2 and build a WordPress environment.
Build a Django development environment with Docker! (Docker-compose / Django / postgreSQL / nginx)
Build a machine learning scikit-learn environment with VirtualBox and Ubuntu
[Memo] Build a development environment for Django + Nuxt.js with Docker
SSH restrictions in Linux environment
Collaborate in a remote environment
Build a Go development environment with VS Code's Remote Containers
Build a Python development environment in Eclipse (add HTML editor)
Build a LAMP environment [CentOS 7]
(Now) Build a GPU Deep Learning environment with GeForce GTX 960
Build python3 environment with ubuntu 16.04
No more dual boot or VM! Let's build a Linux environment with WSL2 & Windows Terminal [WSL2]
What I was addicted to when dealing with huge files in a Linux 32bit environment
Build python environment with direnv
[Django] Build a Django container (Docker) development environment quickly with PyCharm
Build a comfortable development environment with VSCode x Remote Development x Pipenv
Build a machine learning environment
How to build a python2.7 series development environment with Vagrant
Build a python environment with pyenv (OS X El Capitan 10.11.3)