[PYTHON] Until the infrastructure engineer builds GitLab to run CI / CD
Background
Since joining the company as an infrastructure engineer and working on AWS projects, I've been writing more Python for CloudFormation and Lambda. The name of Git / GitHub / GitLab was at a level I've heard, but the version control mentioned above is mandatory, and I will utilize my own GitLab. Although I was an amateur, I immediately felt the convenience and thought that I should set up GitLab after studying Docker on ESXi at home. It is useful for handling files on your desktop PC at home and your laptop on the go.
Since it is built on ESXi at home, if it breaks, I want to avoid having to investigate the construction method from 0, so I will write it here as a memorandum.
Docker I want to build GitLab with Docker-compose, so install Docker.
- As an aside, I had a hard time with snap Docker when installing k8s, so I installed Docker by the following method. If you don't touch k8s, you can use snap.
environment
| item | value |
|---|---|
| OS | Ubuntu 19.04 |
- As of December 2019, k8s does not support docker ver.19, so it is installed by specifying the version.
sudo addgroup --system docker
sudo usermod -aG docker user01
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
sudo apt-get update
apt-cache madison docker-ce
sudo apt-get install docker-ce=5:18.09.9~3-0~ubuntu-bionic docker-ce-cli=5:18.09.9~3-0~ubuntu-bionic containerd.io
sudo systemctl start docker
sudo systemctl enable docker
GitLab Git Lab is a version control tool, but it also has other useful features. Please see the official website for details.
--Functions other than Git
- CI/CD
- Issues
- Wiki
- etc
GItLab DockerCompose If you download the following file and want to access it from another PC Change GITLAB_HOST = to your local IP address.
https://github.com/sameersbn/docker-gitlab/blob/master/docker-compose.yml
version: '2'
services:
redis:
restart: always
image: sameersbn/redis:4.0.9-2
command:
- --loglevel warning
volumes:
- redis-data:/var/lib/redis:Z
postgresql:
restart: always
image: sameersbn/postgresql:10-2
volumes:
- postgresql-data:/var/lib/postgresql:Z
environment:
- DB_USER=gitlab
- DB_PASS=password
- DB_NAME=gitlabhq_production
- DB_EXTENSION=pg_trgm
gitlab:
restart: always
image: sameersbn/gitlab:12.5.2
depends_on:
- redis
- postgresql
ports:
- "10080:80"
- "10022:22"
volumes:
- gitlab-data:/home/git/data:Z
environment:
- DEBUG=false
- DB_ADAPTER=postgresql
- DB_HOST=postgresql
- DB_PORT=5432
- DB_USER=gitlab
- DB_PASS=password
- DB_NAME=gitlabhq_production
- REDIS_HOST=redis
- REDIS_PORT=6379
- TZ=Asia/Kolkata
- GITLAB_TIMEZONE=Kolkata
- GITLAB_HTTPS=false
- SSL_SELF_SIGNED=false
- GITLAB_HOST=localhost
- GITLAB_PORT=10080
- GITLAB_SSH_PORT=10022
- GITLAB_RELATIVE_URL_ROOT=
- GITLAB_SECRETS_DB_KEY_BASE=long-and-random-alphanumeric-string
- GITLAB_SECRETS_SECRET_KEY_BASE=long-and-random-alphanumeric-string
- GITLAB_SECRETS_OTP_KEY_BASE=long-and-random-alphanumeric-string
- GITLAB_ROOT_PASSWORD=
- GITLAB_ROOT_EMAIL=
- GITLAB_NOTIFY_ON_BROKEN_BUILDS=true
- GITLAB_NOTIFY_PUSHER=false
- [email protected]
- [email protected]
- [email protected]
- GITLAB_BACKUP_SCHEDULE=daily
- GITLAB_BACKUP_TIME=01:00
- SMTP_ENABLED=false
- SMTP_DOMAIN=www.example.com
- SMTP_HOST=smtp.gmail.com
- SMTP_PORT=587
- [email protected]
- SMTP_PASS=password
- SMTP_STARTTLS=true
- SMTP_AUTHENTICATION=login
- IMAP_ENABLED=false
- IMAP_HOST=imap.gmail.com
- IMAP_PORT=993
- [email protected]
- IMAP_PASS=password
- IMAP_SSL=true
- IMAP_STARTTLS=false
- OAUTH_ENABLED=false
- OAUTH_AUTO_SIGN_IN_WITH_PROVIDER=
- OAUTH_ALLOW_SSO=
- OAUTH_BLOCK_AUTO_CREATED_USERS=true
- OAUTH_AUTO_LINK_LDAP_USER=false
- OAUTH_AUTO_LINK_SAML_USER=false
- OAUTH_EXTERNAL_PROVIDERS=
- OAUTH_CAS3_LABEL=cas3
- OAUTH_CAS3_SERVER=
- OAUTH_CAS3_DISABLE_SSL_VERIFICATION=false
- OAUTH_CAS3_LOGIN_URL=/cas/login
- OAUTH_CAS3_VALIDATE_URL=/cas/p3/serviceValidate
- OAUTH_CAS3_LOGOUT_URL=/cas/logout
- OAUTH_GOOGLE_API_KEY=
- OAUTH_GOOGLE_APP_SECRET=
- OAUTH_GOOGLE_RESTRICT_DOMAIN=
- OAUTH_FACEBOOK_API_KEY=
- OAUTH_FACEBOOK_APP_SECRET=
- OAUTH_TWITTER_API_KEY=
- OAUTH_TWITTER_APP_SECRET=
- OAUTH_GITHUB_API_KEY=
- OAUTH_GITHUB_APP_SECRET=
- OAUTH_GITHUB_URL=
- OAUTH_GITHUB_VERIFY_SSL=
- OAUTH_GITLAB_API_KEY=
- OAUTH_GITLAB_APP_SECRET=
- OAUTH_BITBUCKET_API_KEY=
- OAUTH_BITBUCKET_APP_SECRET=
- OAUTH_SAML_ASSERTION_CONSUMER_SERVICE_URL=
- OAUTH_SAML_IDP_CERT_FINGERPRINT=
- OAUTH_SAML_IDP_SSO_TARGET_URL=
- OAUTH_SAML_ISSUER=
- OAUTH_SAML_LABEL="Our SAML Provider"
- OAUTH_SAML_NAME_IDENTIFIER_FORMAT=urn:oasis:names:tc:SAML:2.0:nameid-format:transient
- OAUTH_SAML_GROUPS_ATTRIBUTE=
- OAUTH_SAML_EXTERNAL_GROUPS=
- OAUTH_SAML_ATTRIBUTE_STATEMENTS_EMAIL=
- OAUTH_SAML_ATTRIBUTE_STATEMENTS_NAME=
- OAUTH_SAML_ATTRIBUTE_STATEMENTS_USERNAME=
- OAUTH_SAML_ATTRIBUTE_STATEMENTS_FIRST_NAME=
- OAUTH_SAML_ATTRIBUTE_STATEMENTS_LAST_NAME=
- OAUTH_CROWD_SERVER_URL=
- OAUTH_CROWD_APP_NAME=
- OAUTH_CROWD_APP_PASSWORD=
- OAUTH_AUTH0_CLIENT_ID=
- OAUTH_AUTH0_CLIENT_SECRET=
- OAUTH_AUTH0_DOMAIN=
- OAUTH_AUTH0_SCOPE=
- OAUTH_AZURE_API_KEY=
- OAUTH_AZURE_API_SECRET=
- OAUTH_AZURE_TENANT_ID=
volumes:
redis-data:
postgresql-data:
Start docker
docker-compose up -d
Just do it. You can access it if port10080 is free.
CI/CD Runnsers settings are required to run CI / CD. Runnsers requires Runner or k8s. This time I built it with Runnser.
-
Create a Project with GitLab
-
Go to the Project page and go to Settings on the left
-
CI/CD -> Runners -> Expand
-
Follow "Set up a specific Runner manually" to set up.
-
Installation Official page
```bash sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64 sudo chmod +x /usr/local/bin/gitlab-runner sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash sudo /usr/local/bin/gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner sudo systemctl enable gitlab-runner sudo systemctl start gitlab-runner ```
-
Linking with GitLab Official page
sudo /usr/local/bin/gitlab-runner register Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com ) #CI in GitLab/Paste the URL on the CD Settings page Please enter the gitlab-ci token for this runner #CI in GitLab/Paste the token on the CD Settings page Please enter the gitlab-ci description for this runner #Display name in GitLab Please enter the gitlab-ci tags for this runner (comma separated): #Tag name in GitLab Please enter the executor: ssh, docker+machine, docker-ssh+machine, kubernetes, docker, parallels, virtualbox, docker-ssh, shell: docker #This time we will use docker, so enter docker. Please enter the Docker image (eg. ruby:2.1): Python:3.7 #This time we are dealing with python, so python3.Enter the docker image of 7.
If Runnser is added to Runners activated for this project in the CI / CD Settings page, it is successful. If it does not appear, please try linking with GitLab above again.
- Set CI / CD in the created Project
- Create gitlab-ci.yml directly under Project
image: python:3-alpine # docker image
before_script: #Command to be executed first
- pip install pytest pytest-cov autopep8 radon
stages: #You can specify the order of some jobs you want to execute
- build
- test
job1: #job name(Any)
stage: build #Set as build on stage
script: #Command description in list
- autopep8 -i testCode.py
- radon mi -s testCode.py
- radon cc -s testCode.py
job2:
stage: test
script:
- pytest -v --cov=.
The above describes in stages to execute job2 if job1 succeeds. Please refer to the following article for gitlab-ci.yml.
-GitLab CI / CD Pipeline Settings Reference
I don't have another important CD, but I hope you can refer to the AWS Lambda article.
-I tried using lambroll & GitLab CI / CD Pipeline
Summary
As a result of writing it as a memorandum, it has become an official copy. I was able to set up GitLab's CI / CD without much addiction. From now on, I would like to speed up development in a positive sense. (What is an infrastructure engineer ...) Since it is operated by imitating the official and pioneer's appearance, I would appreciate it if you could tell me if there is something wrong.