[Java] From new project creation to automatic test / build / deployment realization

TL;DR Automatically when changes are made to the git branch

Build a Java project where is done from scratch.

What appears

OSS

service

Implementation of sample project

Finally, the directory structure looks like this. I will make it step by step. You can also clone it from GitHub.

testproject/
    ├ src/
    │   ├ main/
    │   │   └ java/
    │   │        └testpackage/
    │   │            └Main.java
    │   └ test/
    │       └ java/
    │           └testpackage/  
    │              └MainTest.java   
    ├ .cifcleci/
    │   └ config.yml
    ├ schema.sql
    ├ pom.xml
    ├ deploy.yaml
    └ Dockerfile

1. Schema definition

Describe the table schema. You can put it in the resource folder, but this time I will put it in the top directory. No database name is specified.

schema.sql


CREATE TABLE user (id int, name varchar(10));

2. Create maven project

Implement a minimal Java project that inserts the appropriate records into the table above.

Use JDBC, JUnit, maven-assembly-plugin.

pom.xml


<project xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>aaaanwz</groupId>
	<artifactId>test</artifactId>
	<version>0.0.1</version>
	<name>testproject</name>
	<properties>
		<maven.compiler.source>11</maven.compiler.source>
		<maven.compiler.target>11</maven.compiler.target>
	</properties>
	<dependencies>
		<!-- JDBC driver -->
		<dependency>
			<groupId>mysql</groupId>
			<artifactId>mysql-connector-java</artifactId>
			<version>8.0.17</version>
		</dependency>
		<!-- JUnit -->
		<dependency>
			<groupId>org.junit.jupiter</groupId>
			<artifactId>junit-jupiter-engine</artifactId>
			<version>5.3.1</version>
			<scope>test</scope>
		</dependency>
	</dependencies>
	<build>
		<plugins>
			<!-- JUnit test -->
			<plugin>
				<artifactId>maven-surefire-plugin</artifactId>
				<version>3.0.0-M3</version>
			</plugin>
			<!-- Build runnable jar -->
			<plugin>
				<artifactId>maven-assembly-plugin</artifactId>
				<executions>
					<execution>
						<phase>package</phase>
						<goals>
							<goal>single</goal>
						</goals>
					</execution>
				</executions>
				<configuration>
					<archive>
						<manifest>
							<mainClass>testpackage.Main</mainClass>
						</manifest>
					</archive>
					<descriptorRefs>
						<descriptorRef>jar-with-dependencies</descriptorRef>
					</descriptorRefs>
				</configuration>
			</plugin>
		</plugins>
	</build>
</project>

Implement the Main class that only inserts one record. The connection information to the database is obtained from environment variables.

src/main/java/testpackage/Main.java


package testpackage;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;

public class Main {

  public static void main(String[] args) throws ClassNotFoundException, SQLException {
    final String host = System.getenv("DB_HOST");
    final String dbname = System.getenv("DB_NAME");
    execute(host, dbname);
  }

  static void execute(String host, String dbname) throws ClassNotFoundException, SQLException {
    Class.forName("com.mysql.cj.jdbc.Driver");
    Connection conn = DriverManager.getConnection(
        "jdbc:mySql://" + host + "/" + dbname + "?useSSL=false", "root", "");
    PreparedStatement stmt = conn.prepareStatement("INSERT INTO user(id,name) VALUES(?, ?)");
    stmt.setInt(1, 1);
    stmt.setString(2, "Yamada");
    stmt.executeUpdate();
  }

}

Write a test that just confirms that the record has been inserted.

src/test/java/testpackage/MainTest.java


package testpackage;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.fail;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;

import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;

import testpackage.Main;

class MainTest {
  Statement stmt;

  @BeforeEach
  void before() throws ClassNotFoundException, SQLException {
    Class.forName("com.mysql.cj.jdbc.Driver");
    Connection conn =
        DriverManager.getConnection("jdbc:mySql://localhost/test?useSSL=false", "root", "");
    stmt = conn.createStatement();
  }

  @AfterEach
  void after() throws SQLException {
    stmt.executeUpdate("TRUNCATE TABLE user");
  }

  @Test
  void test() throws Exception {
    Main.execute("localhost", "test");
    try (ResultSet rs = stmt.executeQuery("SELECT * FROM user WHERE id = 1;")) {
      if (rs.next()) {
        assertEquals("Yamada", rs.getString("name"));
      } else {
        fail();
      }
    }
  }
}

3. Create Dockerfile

Write a Dockerfile. Since the tests will be done in a separate step from the docker build, add -DskipTests to skip the build-time tests.

Build with maven: 3.6 and multistage build with openjdk11: apline-slim to reduce image size. It is about 800MB for maven: 3.6 and about 300MB for openjdk11: alpine-slim. At the moment, the free tier of ECR is 500MB, which is a big difference for personal development.

Dockerfile


FROM maven:3.6 AS build

ADD . /var/tmp/testproject/
WORKDIR /var/tmp/testproject/
RUN mvn -DskipTests package

FROM adoptopenjdk/openjdk11:alpine-slim

COPY --from=build /var/tmp/testproject/target/test-0.0.1-jar-with-dependencies.jar /usr/local/
CMD java -jar /usr/local/test-0.0.1-jar-with-dependencies.jar.jar

4. Create k8s yaml file

Write yaml to deploy the container built in 3. to k8s. Since this program executes SQL in a single shot and ends, let's set it to kind: Job. Replace the Docker registry url as appropriate. Define the connection information to the DB with ʻenv. The value of DB_HOST is mysql` because you are connecting via Kubernetes Service.

k8s-job.yaml


apiVersion: batch/v1
kind: Job
metadata:
  name: test
spec:
  template:
    spec:
      containers:
      - name: test
        image: your-docker-registry-url/testimage:latest
        imagePullPolicy: Always
        env:
        - name: DB_HOST
          value: "mysql"
        - name: DB_NAME
          value: "ekstest"
      restartPolicy: Never
  backoffLimit: 0

5. Create circleci config file

This is the key to this article. Write configuration settings for automated testing / building on CircleCI.

yml:.circleci/config.yml


version: 2.1
orbs:
  aws-ecr: circleci/[email protected]
  aws-eks: circleci/[email protected]
  kubernetes: circleci/[email protected]
jobs:
  test: #① Run JUnit test
    docker:
      - image: circleci/openjdk:11
      - image: circleci/mysql:5.7
        environment:
          MYSQL_ALLOW_EMPTY_PASSWORD: yes
          MYSQL_DATABASE: test
        command: [--character-set-server=utf8, --collation-server=utf8_general_ci, --default-storage-engine=innodb]
    steps:
      - checkout
      - run:
          name: Waiting for MySQL to be ready
          command: dockerize -wait tcp://localhost:3306 -timeout 1m
      - run:
          name: Install MySQL CLI; Create table;
          command: sudo apt-get install default-mysql-client && mysql -h 127.0.0.1 -uroot test < schema.sql
      - restore_cache:
          key: circleci-test-{{ checksum "pom.xml" }}
      - run: mvn dependency:go-offline
      - save_cache:
          paths:
            - ~/.m2
          key: circleci-test-{{ checksum "pom.xml" }}
      - run: mvn test
      - store_test_results:
          path: target/surefire-reports
  deploy: #③ kubectl apply to EKS
    executor: aws-eks/python3
    steps:
      - checkout
      - kubernetes/install
      - aws-eks/update-kubeconfig-with-authenticator:
          cluster-name: test-cluster
          aws-region: "${AWS_REGION}"
      - run:
          command: |
            kubectl apply -f k8s-job.yaml
          name: apply
workflows:
  test-and-deploy:
    jobs: 
      - test: #① Run JUnit test
          filters:
            branches:
              only: develop
      - aws-ecr/build-and-push-image: #② Build the container and push to ECR
          filters:
            branches:
              only: master
          create-repo: true
          repo: "testimage"
          tag: "latest"
      - deploy: #③ kubectl apply to EKS
          requires:
            - aws-ecr/build-and-push-image

① Execution of JUnit test

I've tried using filter to run mvn test when changes are made to the develop branch. At this time, a test MySQL instance is launched and the test database is prepared.

It is explained in detail in the official documentation.

② Build the container and push to ECR

When changes are made to the master branch, build according to the Dockerfile and push to ECR. You can read more about it in the Orb Quick Start Guide (https://circleci.com/orbs/registry/orb/circleci/aws-ecr).

③ kubectl apply to EKS

After ② is executed, the Job defined in 4. is executed. There is an explanation in circleci / aws-eks, but I changed the sample code to be simpler. According to the @ 0.2.1 documentation, the parameter ʻaws-region is not Required, but it seems to be required in practice. The environment variable ʻAWS_REGION is requested by push of ECR, so I used it as it is.

Infrastructure settings

Mostly a collection of links to official documents.

1. ECR, EKS environmental preparation

1-1. Basic settings

Launch test-cluster.

1-2. Deploy MySQL on EKS

Kubernetes officially has a document called Deploy MySQL. Prepare the ʻekstest database and the ʻuser table from the kubectl exec -it mysql-0 mysql command.

2. CircleCI project settings

2-1. Project setup

2-2. Setting environment variables

--Set the environment variables Required in circleci / aws-ecr to CircleCI Project [Set](https: // circleci .com /docs/2.0/env-vars/#setting-an-environment-variable-in-a-project). I will omit the details of the privileges required for Role.

Summary

--push to develop branch The mvn test will run and the test report will appear in CircleCI's Test summary. --push to master branch The docker image is pushed to ECR and Job starts at EKS. A ʻid: 1 name: Yamada` record is inserted into MySQL on EKS.

What did you think. I hope you will try various things such as dropping the test on purpose.

Recommended Posts

[Java] From new project creation to automatic test / build / deployment realization
New features from Java7 to Java8
Automatic creation of Java unit test result report
Changes from Java 8 to Java 11
Sum from Java_1 to 100
Eclipse ~ Java project creation ~
From Java to Ruby !!
Catch up on new features from Java 7 to Java 9 at once
Migration from Cobol to JAVA
Connect from Java to PostgreSQL
From Ineffective Java to Effective Java
How to create a new Gradle + Java + Jar project in Intellij 2016.03
[Gradle] Build a Java project with a configuration different from the convention
Build a Java project with Gradle
Java to be involved from today
From Java to VB.NET-Writing Contrast Memo-
Java, interface to start from beginner
Addicted to project imports from GitHub
The road from JavaScript to Java
[Java] Conversion from array to List
Change from SQLite3 to PostgreSQL in a new Ruby on Rails project