The story of introducing Gradle as a retrofit to an existing system that did not manage packages

About this material

<!-TODO: Be careful if the various links and notations are for 2018. Pay particular attention to the twitter link! !! -> The presentation material of the title session of JJUG CCC 2019 Spring has been released on Qiita (twitter: # ccc_a2 //twitter.com/search? q =% 23ccc_a2)).

<!-TODO: Replace with screenshot image of Speakers deck-> <!-TODO: Link destination required correction! !! -> image.png

The content is almost the same as the above slide, so I hope you can see it in an easy-to-read manner.

Introduction

Maven, Gradle, etc., which are now commonplace ...

<!-TODO: Keep the back of the time series->

Our company M3 Co., Ltd. was founded in 2000 (J2SE, the era around Java 2). I've been using Java since the days when Maven, Gradle, and Ivy weren't there yet.

The result of making full use of Java in the absence of Maven and Gradle ...

When you do git clone, the following file will be ...

lib/app/hibernate-validator-4.3.0.Final.jar
lib/app/aws-java-sdk-core-1.10.27.jar
lib/app/aws-java-sdk-s3-1.10.27.jar
lib/app/generator-runtime-0.3.0-20050901.1909.jar
lib/app/xalan.jar
lib/app/crimson.jar
                 ...Continues endlessly...

As a result of the increasing number of libraries used over history, 182 .jars were rolling.

Operations that directly manage .jar

I was in a state of maintenance with such a process:

適当なところから手作業で jar をダウンロードして Git に入れるという闇のオペレーションの図

The pain of upgrading the library

Example: I want to upgrade the version of ʻaws-java-sdk` to use new features

aws-java-sdk system indirectly depends on multiple libraries

I don't know how to upgrade 182 .jars

Give up on using new AWS features: cry:

Indirect dependence

I'm not using a jar, so I want to delete it

Maybe one of the jars depends on it ...

Unnecessary jar cannot be cleaned

To hinder JVM version upgrades and vulnerability countermeasures: cry:

I can't use various tools

I want to check vulnerability information

Check 182 .jars with vulnerability information on a regular basis

Tough & limited

I tend to leave it alone

Gradle, maven, etc. can be automated with OWASP dependency-check plugin ...

The background that had become like this

Other than this system, maven and Gradle have already been installed, but ...

--The system has been around since its inception and has a particularly long history. --Ant's build script is tightly assembled, and batch replacement is difficult ――Since the usual development work goes around without problems ... (* As long as the darkness around the library is ignored) </ small>

Under these circumstances, there may be cases in the world where package management tools are not included.

Put in your package management tools (Gradle, maven, etc.)!

But how do you implement it in an existing project ...?

――What do you do about the points that tend to increase the work load? ――What are the side effects and effects of adding it?

→ Go to the next chapter

<!-==================================== # Section: Strategy Part (Gradle) == ==================================->

Package management tool implementation strategy

~ Historic system improvements __Possible __scope definition ~

Challenges in implementing package management tools

Retrofitting into existing systems presents a number of challenges:

――Is it good that the jar changes before and after the introduction of the tool? --What to do with jars that are difficult to re-obtain or that do not match central? --Would you like to modify the build process?

It's better to have a policy so you don't get swayed by challenges.

Policy set in our case

--Do not make a difference in the .jar file before and after the introduction of the package management tool. --To reduce concerns about the impact on the app and the burden of testing --Make sure that the only unavoidable difference is a harmless difference (method will be described later) </ small> --Inventory and ver up are the scope of the next improvement --Do not touch the build process --First, improve package management!

Scoped to benefit from package management tools, Created a situation where you can take on small and meaningful challenges.

State before introducing the package management tool

適当なところから手作業で jar をダウンロードして Git に入れるという闇のオペレーションの図

This happened with the introduction of the package management tool

jar のダウンロードだけを Gradle 化した図

The pros and cons of raising .jar to Git

: + 1: Benefits:

--Easy to migrate from Ant etc. --Always use the same jar file

: -1: Disadvantages:

--Ant remains --If you really have a problem with Ant, replace it at that time --Repositories will grow with repeated updates of .jar --Sometimes it will be separated as Git submodule

Legacy improvement strategy

Identify the true trouble and merit to change

Consider the necessary and sufficient minimum scope for that.

Achieve the goal with small challenge (high cost performance, small damage even if you fail) </ small>

It ’s not “Legacy, so you have to kill everything !!!” Improvements while following the existing good points and points that do not need to be changed.

Technique (tactic) part

Package management tool deployment technique: How to retrofit Gradle

~ Specific measures to deal with various issues that tend to be retrofitted ~

Gradle or Maven

Gradle is recommended for applications like this one:

--Flexibility: Flexible processing can be described in Groovy and Kotlin DSL --Maven may not be able to handle the composition of historic projects --Investigative: Groovy Javadoc is easier to read (subjective) </ small> --Productivity: Take full advantage of the IDE's debugger --You don't have to fight like a black box --Reproducibility: Gradle wrapper makes it (relatively) easy to ensure behavioral reproducibility

* In general, Maven is better in some cases </ small>

File copy with Gradle

In order to introduce Gradle while utilizing the existing build system I want to copy the following files to the specified directory with Gradle:

--Jar that depends on compile time and execution time --Jar that depends on unit test

  • Source jar

I also want to automatically maintain the .classpath file (described later) </ small>.

Download the jar file with Gradle

Surprisingly easy:

def compileJarCopyTo = 'development/lib/app'

task downloadJars() {
  doLast {
    //At the copy destination*.Erase the jar
    delete fileTree(dir: compileJarCopyTo, include: '*.jar')

    //make a copy
    copy {
      from configurations.compile
      into compileJarCopyTo
    }
  }
}

copy task

Convenient to specify the jar to be copied and include, exclude, etc .:

copy {
  // testCompile =Copy the dependent jars in test
  from configurations.testCompile

  into testCompileJarCopyTo

  //Ignore jars that match the specified pattern
  exclude 'jmockit-*.jar'
}

You can also copy the Source jar

To copy the Source jar, write code to manipulate these goggles:

import org.gradle.jvm.JvmLibrary
import org.gradle.language.base.artifact.SourcesArtifact
import org.gradle.api.artifacts.query.ArtifactResolutionQuery
import org.gradle.internal.component.external.model.DefaultModuleComponentIdentifier

(ʻorg.gradle.internal` has also appeared ...) </ small>

→ Refer to the code for Gradle 2.x of the predecessor, and refer to the Gradle 5.x compatible version
Copy source jar files into directory (Qiita ) Posted in

Generate a .classpath file

A configuration file that allows the IDE to load jars and source code. Originally for Eclipse, it also supports InteliJ IDEA.

I want to maintain this file with Gradle because it's a big deal.

It can be generated just by applying Gradle's ʻeclipse` plugin ...

.classpath Absolute path problem

The path in .classpath generated by Plugin is an absolute path. It is inconvenient because it becomes an environment-dependent path and cannot be shared with Git.

solution: --Create .classpath each time by gradle eclipseClasspath at your own hands -: -1: It takes time for everyone to do it every time -: -1: The reproducibility of the build decreases (even though it worked at hand! Phenomenon) --Generate .classpath in relative path notation with Gradle and commit with jar file -: + 1: Just execute it together with a copy of the Jar file etc. OK

Generate .classpath in relative path notation

ʻEclipseplugin'spathVariables'.': Instead of projectDir`, if you extend it using the plugin hook as shown in Reference site good:

eclipse.classpath.file {
    whenMerged { classpath ->
        classpath.entries.findAll { entry -> entry.kind == 'lib' }.each {
            it.path = it.path.replaceFirst("${projectDir.absolutePath}/", '')
            it.exported = false
        }
    }
}

ʻEclipse.classpath.file`` whenMerged`

You can freely control the contents of the .classpath file with whenMerged

For example:

--Rewrite the reference path of the jar file with arbitrary logic --In our case, only some files are placed in different directories. -- classpath.entries.sort controls the sort order of .classpath --Useful for load-order-dependent library measures such as JMockit --classpath.getEntries (). removeAll {return (conditional expression)} --You can exclude jars that you don't want the IDE to load

Change of jar file name

Manual management jar filenames and Gradle-derived jars often do not match. For example:

--Manually managed file name: crimson.jar --Gradle download result: crimson-1.1.3.jar

There is a problem when referencing with a fixed file name. In particular, there are the following:

--A fixed file name in the classpath of build tools such as Ant --File name fixed in the classpath option of the JVM when the program starts

Workaround for jar file name change

  1. (Not recommended) Rename to the old file name in post-processing -: -1: Accident when the name of the rename source group or artifact changes -: -1: I'm having trouble dealing with new jars
  2. Be unaware of individual libraries -: + 1: Design OK just by reading all the jars under a specific directory --Leverage * wildcards in Ant, JVM classpath (> = 6), shell, etc. --Note that the interpretation of * is slightly different.

The latter is a hassle, but it is a temporary effort, so it is better to deal with it in the long run.

<!-==================================== # Section: Strategy Part (Jar Management) = ===================================->

Package management tool implementation technique: Managing jars for various reasons

~ Techniques for handling jar files that came from beyond history ~

Common things in old jar files

<! ---- The jar contained in the all-in-one jar is squishy->

--I was able to get the jar again, but the binaries do not match --SNAPSHOT version --Although it is non-SNAPSHOT, it has been replaced and released. --Different binaries are placed between Maven repositories --Original patch of unknown background: imp: --The official page has disappeared and you cannot re-obtain the jar or source

Find out what caused the jar binary mismatch

Don't leave the discrepancy, just check the specific differences:

--You can limit the scope and risk of introducing a package management tool. --The range to be tested and the test method will be clarified.

I issued a SHA-1 hash of the jar file before and after Gradle conversion, We examined the specific differences for jars whose hashes do not match.

How to check the difference of jar file

The jar file is a zip and when unzipped you will see the following files:

  1. .class file (Java bytecode)
  2. Metadata files such as META-INF
  3. Other resource files such as property files

Except for .class, most text comparison tools can be used for comparison.

Example when there is a difference in META-INF / MANIFEST.MF of lucene-analyzers-2.4.0: </ small>

8c8
< Implementation-Version: 2.4.0 701827 - 2008-10-05 16:46:27
> Implementation-Version: 2.4.0 701827 - 2008-10-05 16:44:47

How to check jar diffs: class file

The .class file is a Java bytecode binary.

By converting to text representation with javap -c included in the JDK It becomes possible to compare with the text comparison tool.

Example when there is a difference in guice-servlet-1.0.jar:

2c2
< final class com.google.inject.servlet.ServletScopes$1 implements com.google.inject.Scope {
> class com.google.inject.servlet.ServletScopes$1 implements com.google.inject.Scope {

(replaced by jar with final added) </ small>

Non-retrievable library

Measures other than suspension of use of the library:

-(: imp: Deprecated) Upload to public repositories such as Central -: -1: License or morally unfavorable -: -1: To publish an unconfirmed jar that matches a known jar -(: thinking: subtle) Use the .java file of the target library directly -: -1: Since the source is used directly, original modifications and tight coupling tend to occur. -: -1: There is a problem depending on the license -(Recommended) Managed by private repository for internal use -: + 1: Rights problems are less likely to occur -: + 1: Easy to grasp and delete the usage status of the library

Private repository

Not limited to non-retrievable jars, but also for managing in-house libraries, etc. It is convenient to have a private repository.

Common construction method:

  1. Browse the local maven repository with file: // --Needs synchronization between machines --Tends to compromise build reproducibility --There is also a risk of losing the jar and the trouble of backing up.
  2. Run repository servers such as Nexus and Artifactory (convenient) --I won't go into the details of how to set it up, but if you're setting up a new one, the Nexus is easier.

Use of private repository

  1. The libraries that exist in the world are still obtained from maven central etc.
  2. Get private things from your private repository

With Gradle, you can do this simply by:

repositories {
  maven {
    url 'http://central.maven.org/maven2/'    
  } // mavenCentral()I do not dare to use the abbreviated notation(See below)

  maven {
    url "http://URL of a private repository"
  }
}

Upload the jar to the repository

<!-TODO: This description may not be necessary depending on the reader-level assumptions ... But should I write something like Gradle for people who don't know maven? ->

Write the metadata in Maven XML (pom.xml) format and upload it with the jar.

It may seem awkward at first glance, but it's easy if you focus on this purpose:

<project>
  <modelVersion>4.0.0</modelVersion>

  <!--Identification information for this jar itself-->
  <groupId>com.m3</groupId> <artifactId>m3generator</artifactId>
  <version>1.0.0</version>

  <!--Information on the jars this jar depends on-->
  <dependencies>
    <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>1.2.17</version>
    </dependency>
    ...

(Artifactory) </ small> Beware of Auto generated POM

It looks convenient because Artifactory automatically generates a POM ...

<dependencies> is often empty or incorrect

I can't get information about the dependent libraries of the uploaded jar

The benefits of dependency management with package management tools are lost

Let's write <dependencies> properly without throwing it to automatic generation.

Overlapping libraries between repositories

If there are libraries with the same name but do not match (with differences) You should explicitly control which one is loaded:

--Upload to private repository as another version --Or, control on Gradle side (below)

repositories {
  maven { // Maven central
    url 'http://central.maven.org/maven2/'    

    content {
      excludeGroup 'com.sun.jmx' //Don't get this group from here
      excludeModule 'jdom', 'jdom' //Don't get this library from here
    }
    //MavenCentral to make the above description()I do not use the abbreviated notation such as
  }

Finally

Next Step

What you especially want to do when you include a package management tool:

--Version Collision detection --Duplicate detection on classpath --Vulnerability information check by OWASP dependency check

A deeper explanation on the JJUG 2018 Spring slides:

--Title: How to make a compromise between Spring Boot and general libraries -Speaker Deck and Qiita The same content is posted on

  • The latter half of the material is information that can be used regardless of Spring Framework.

Summary

This time, the process when the package management tool was installed:

--Define goals ――Draw a vision of how package management will improve --Build a strategy ――What to do for realization ・ What do you dare not to do? --Solving problems with techniques: Various ideas mentioned this time --Utilization of Gradle --Jar diff check --Utilization of private repository

Even in situations where you may think that the darkness is so deep that you can't touch it __ Set the appropriate scope and realize with technology __ to move forward.

Recommended Posts