A story that made it as easy as possible to check the operation when automatically creating a library update PR with Github Dependabot

What is Dependabot in the first place?

A feature called Dependabot on Github Did you know that there is? If there is an old dependency (library) used in the repository, it is a function that automatically issues an updated Pull Request. Many of you may have seen the following PR in public repositories.

Originally an independent service, it was acquired by Github and incorporated as a native feature, making it easier to deploy. You can enable library update PR automatic creation just by placing the configuration file under .github/dependabot.yml.

I will omit the explanation of Dependabot itself this time, but in the project I am involved in, I have started operation so that the following configuration file will be checked at 9 o'clock on Monday.

version: 2
updates:
  - package-ecosystem: "gradle"
    directory: "/"
    target-branch: "develop"
    schedule:
      interval: "weekly"
      day: "monday"
      time: "09:00"
      timezone: "Asia/Tokyo"
    reviewers:
      - "ignis-ltd/with-android"

First of all, if you want to know more about Github Dependabot, the following official documents and articles will be helpful.

--Automatically update dependencies --GitHub Docs -Customize the behavior of dependabot on GitHub

• Currently, if you google with "Dependabot", there will be many articles on the marketplace version of the era when it was an independent service before it was acquired, but there is a high possibility that it will be abolished and integrated into native functions. Please be careful

Dependabot operational issues

Dependabot will be issued one PR for each library update. Since some library updates may cause problems, I think that it is the correct form to be individualized in order to isolate the problem, but if there are multiple updates, some problems will remain. ..

I think that similar issues may occur on the server side, but since it is an Android application in my project, if 5 PRs are issued at once as described above, check out the branch 5 times, build and install it. I had to check the operation.

Of course, you can reduce the risk by building a test on CI and automating it, but in the case of an application, there is still concern about UI collapse etc., so I would like to visually check the operation once before merging. .. Or if you deploy each branch with CI, you don't have to check out, but it's still a little difficult just to download, install and start 5 times.

solution

As an approach I thought about, I considered a method of ** merging each PR branch issued by dependabot into one branch, building from CI, and leaving a link to the binary as a comment for each PR **. .. I will introduce the actual method below.

Consolidate branches published by Dependabot

First, let's create a Ruby script that extracts and integrates all updated Git branches.

1. ** Get a list of branches starting with the string dependabot / **
2. ** Create a patch file for the update commit from the current branch (develop) **
3. ** Apply all patch files obtained in 2. **

(Ruby power is at the bottom, so don't miss it: pray :)

Gemfile

source 'https://rubygems.org'

gem 'git'

merge_dependabot_branchs.rb

require "git"

dependabotBranchs = []
git_client = Git.open(Dir.pwd)
git_client.fetch
git_client.branches.each do |branch|
  if branch.name.start_with?('dependabot/')
    dependabotBranchs.append(branch)
  end
end

return if dependabotBranchs.size <= 0

dependabotBranchs.each do |dependabotBranch|
  system("git format-patch -1 #{dependabotBranch.gcommit} --unified=0")
end
system("git apply 0001-*.patch --unidiff-zero")

A little commentary

The key here is how to create and apply a patch file. I'm generating a patch file with git format-patch, but ** usually the changes in the 3 lines before and after are also included in Diff **. In other words, if there are already changes within 3 lines before and after, such as when the updated library lines are next to each other, it will be judged as a conflict and will not be automatically integrated. Conflicts should not occur if only the changes in the version upgrade by Dependabot are extracted, so even if there are changes in the near line, we would like you to integrate them without considering them as conflicts. Therefore, by adding the --unified = 0 parameter, the previous and next lines are adjusted not to be included in the Diff. ([Reference](https://git-scm.com/docs/git-format-patch#Documentation/git-format-patch.txt --- unifiedltngt))

I also use a trick when applying a patch file with git apply, ** normally apply with a patch file that does not contain the surrounding lines (--unified = 0 is specified). Will fail **, so I added the --unidiff-zero parameter so that it is applied ignoring the previous and next lines. ([Reference](https://git-scm.com/docs/git-apply#Documentation/git-apply.txt --- unidiff-zero))

The processing in this area was insufficient with ruby-git that I was using at the beginning, so I rely on the system call.

At this point, you should have a branch with integrated library updates, so if you build in this state, you can generate integrated binaries.

Put a link to the binary integrated into the PR issued by Dependabot

Assuming that a binary is generated on CI and a link URL is also issued, let's build a Ruby script that leaves a comment on the PR generated by the original Dependabot. This time, use Github API instead of Git to extract.

1. ** Extract branches starting with the string dependabot / in the currently open Pull Request **
2. ** Paste the pull request list integrated with the binary link for the extracted pull request as a comment **

(Ruby power bottom code)

comment_dependabot_prs.ruby

require 'net/http'
require 'uri'
require 'json'

uri = URI.parse("https://api.github.com/repos/ignis-ltd/with_android/pulls")
request = Net::HTTP::Get.new(uri)
request["Accept"] = "application/vnd.github.v3+json"
request["Authorization"] = "token #{ENV['GITHUB_API_TOKEN']}"

req_options = {
  use_ssl: uri.scheme == "https",
}

response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
  http.request(request)
end

dependabotPullRequests = []
responseBodyJson = JSON.parse(response.body, symbolize_names: true)
responseBodyJson.each do |pull|
  if pull[:head][:ref].start_with?('dependabot/')
    dependabotPullRequests.append(pull)
  end
end

dependabotPullRequests.each do |pull|
  uri = URI.parse("https://api.github.com/repos/ignis-ltd/with_android/issues/#{pull[:number]}/comments")
  request = Net::HTTP::Post.new(uri)
  request.body = JSON.dump({
    "body" => "Deployed a binary that merged the following branches\n" + dependabotPullRequests.map { |pull| pull[:html_url] }.join(" ") + "\n\n#{ENV['INSTALL_PAGE_URL']}"
  })
  request["Accept"] = "application/vnd.github.v3+json"
  request["Authorization"] = "token #{ENV['GITHUB_API_TOKEN']}"

  req_options = {
    use_ssl: uri.scheme == "https",
  }

  response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
    http.request(request)
  end
end

A little commentary

Basically, it's just implemented straightforwardly using the Github API.

The following environment variables are required to execute the above Ruby script. Please change the URL for Github API as appropriate Please do not include the token information of Github directly, but use the secret environment variable of CI.

-** GITHUB_API_TOKEN ** ... Please specify a Token to comment on Github -Get it by referring to Use Personal Access Token --GitHub Docs Please give me -** INSTALL_PAGE_URL ** ... Please specify the link URL to the generated binary ――This time, I will not mention building and deploying with CI, so I will omit it, but I personally recommend Bitrise for CI of the application.

If you can build up to this point, you can leave the following comments in the Pull Request issued by Dependabot. You will be able to refer to which branch you have integrated and the integrated binary from any Pull Request.

Run scripts and builds on time in CI after running Dependabot

In my project, Dependabot runs at 9:00 on Monday, so I've adjusted the CI containing this script to run at 10:00 on Monday. The following is a setting example in Bitrise. Please adjust this area to your liking.

Supplement or caution

The script that integrates the first Dependabot branch is based on Git branch information, but the script for later comments searches based on PullRequest, so (I don't think it's basically) PullRequest If there is an unpublished Dependabot branch, or if you make changes to the branch or PR during the build, the content of the comment and the actual state of the binary will be different.

Also, since the branch name is searched for prefix with the character string dependabot /, it will malfunction when the corresponding branch is created manually, so it may be necessary to provide a slightly stricter judgment logic. (A feeling of horizontal wear)

end

It was quite a skill, but I think that doing so far will lead to cost reduction because you can check the operation once every week. I don't think there are so many libraries in a small program, so you may not feel the need to do so, but if the scale exceeds 100,000 lines, the number of installed libraries will be enormous and the operation check cost will be high. I can't make a fool of myself, so by taking such measures, I can make the best use of Dependabot.

Also, this time we are talking about integrating Dependabot on CI, but since we can hardly talk about the CI side only about the part of writing a script and integrating it with spirit, I would like to talk somewhere. ..