The story of migrating from Paperclip to Active Storage

Hello. @mshibuya. Currently, I am helping ZENKIGEN Co., Ltd. as a side business, and I am in charge of improving Rails of Web interview service harutaka. This time, I would like to talk about the transition from Paperclip to Active Storage that was done there.

This article is the entry for the 18th day of Rails Advent Calendar 2020.

Motivation

Active Storage appeared in Rails 5.2 released in April 2018, and soon after the deprecation of Paperclip was announced, more than two years have passed. Paperclip will not be maintained anymore, so we need to consider other means.

Paperclip has been used for a long time in harutaka as well, and it has already been configured to partially use Active Storage in search of a transition to a new method. However, the part that uses Paperclip remains as it is, and the state where it coexists with ActiveStorage is not preferable for maintenance, so we decided to renew the whole system to a new method.

Considered migration destination

There are several options for a library that provides a file upload function, so we organized the characteristics of each and selected migration destination candidates.

ActiveStorage As mentioned above, it is a file upload function implemented as part of the standard Rails function.

Pros ――It is a part of Rails and is expected to become the de facto standard in the Rails area in the future. --Similarly, active maintenance is expected to continue --Has already been partially used in harutaka --Paperclip has officially specified the migration destination, and there is also a Migration procedure.

Cons --Inferior in functionality compared to Paperclip and CarrierWave ――It's made opinionated, and it seems to be difficult if you use it that does not match the idea

kt-paperclip Paperclip fork maintained by Kreeti.

Pros --Following Paperclip, a proven and dead library ――Since it is used in the main part even in harutaka, there is little trouble of migration ――Multifunctional as it is

Cons ――It is a fork version, not maintenance by the head family, and the future future is uncertain.

CarrierWave It's the second major file upload library after Paperclip.

Pros

Cons --Slightly complicated to use --There is no record of use in harutaka, so it will be a completely new introduction

Based on the above comprehensively, we decided to unify the existing Paperclip implementation into ActiveStorage by integrating it into ActiveStorage.

Transition policy

I want to make the usability as close as possible to the existing one

I wasn't involved in full-time development, so I didn't want to overburden other developers.

I want to be able to switch back when something goes wrong

The function to save and browse images and videos is a highly important part of harutaka, so if there are any problems with this migration after the production release, immediately switch back to the Paperclip implementation and use it normally. I aimed to be able to continue.

I want to avoid migrating the data already stored in S3

It takes time to migrate the saved data, and it is necessary to stop using the system during that time or prepare a mechanism that can reflect the data update during migration, so it will be more thoughtful, so at least at the initial timing of migration I didn't want to.

Therefore, we aim to manage the missing functions by patching Active Storage ...

Features missing in Active Storage

Well, here, various missing functions come out due to the simple and optimized construction of Active Storage. I will introduce what kind of function was missing and what was done with it.

Note that the code illustrated here assumes Active Storage 5.2. It may not work as it is in other versions, so please read it as you like.

File delivery using CloudFront signed URL

First of all, Active Storage of course supports data storage and distribution with S3 as the back end, but surprisingly it does not support distribution using CloudFront as standard.

However, the solution to this is relatively simple. Since ActiveStorage is designed so that various storage backends such as local disk and S3 can be replaced as a service.

require 'active_storage/service/s3_service'

module ActiveStorage
  class Service::CloudFrontService < ActiveStorage::Service::S3Service
    def url(key, expires_in:, filename:, disposition:, content_type:)
      instrument :url, key: key do |payload|
        generated_url = Aws::CF::Signer.sign_url "https://#{CLOUD_FRONT_HOST}/#{key}"
        payload[:url] = generated_url
        generated_url
      end
    end
  end
end

Create CloudFrontService in the form of inheriting S3Service like, and in storage.yml

production:
  service: CloudFront
  access_key_id: xxx
  secret_access_key: xxx
  ...

Then, you can create a state of "save the file in S3 and deliver it with the URL signed by CloudFront". (* Cloudfront-signer gem is used here, and its setting is required separately)

Ability to receive URLs and save data

Paperclip has a function to download and save data from a URL when it receives a URL instead of the file itself. This is implemented as one of Paperclip's IOAdapters, UriAdapter, but since there is no similar mechanism in ActiveStorage, it needs to be implemented as a patch.

The image looks like this. Monkey patch ActiveStorage :: Attached.

ActiveStorage::Attached.prepend Module.new {
  def create_blob_from(attachable)
    case attachable
    when String
      uri = URI.parse(attachable) rescue nil
      if uri.is_a?(URI::HTTP)
        file = DownloadedFile.new uri
        ActiveStorage::Blob.create_after_upload! \
          io: file.io,
          filename: file.filename,
          content_type: file.content_type
      elsif attachable.present?
        super
      end
    else
      super
    end
  end
}

class DownloadedFile
  attr_reader :io

  def initialize(uri)
    @uri = uri
    @io = uri.open
  end

  def content_type
    @io.meta["content-type"].presence
  end

  def filename
    CGI.unescape(@uri.path.split("/").last || '')
  end
end

Customize the save destination path to S3

Paperclip makes it possible to specify the path of the file save destination with great flexibility by URL Interpolation. On the other hand, ActiveStorage has no room for such customization, and the save destination path of the file is always a randomly generated character string generated by generate_unique_secure_token. ActiveStorage seems to have chosen not to include this response with a fairly strong will, and has rejected the PR received in the past, so it seems unlikely that it will be included in the future ...

So I'll patch it and do something about it. After making it possible to pass the proc that generates the key on the model side like this,

  has_one_attached :image, key: -> (filename) { "files/image/#{record.class.generate_unique_secure_token}/#{filename}" }

Route this proc to ActiveStorage :: Blob

ActiveSupport.on_load(:active_storage_blob) do
  prepend Module.new {
    def key
      self[:key] ||= if attachment
        key_proc = options[:key] #The one who has been around
        (key_proc && attachment.instance_exec(filename, &key_proc)) || super
      else
        super
      end
    end
  }
end

If there is no value, proc will be instance_exec to generate the desired key.

Named style

Paperclip has a concept of style for Generate Thumbnail Image, and you can name the image size to generate.

has_attached_file :photo, styles: {thumb: "100x100#"}

Of course, ActiveStorage also supports thumbnail generation, but this is a method of dynamically passing the size and generating it when using it, not when saving the image.

<%= image_tag user.avatar.variant(resize: "100x100").service_url %>

However, it is easier to understand the purpose if it has a name, and it can prevent images of strangely similar sizes from being scattered, so you want to be able to do this, right?

<%= image_tag user.avatar.variant(:thumb).service_url %>

I will patch it there. From the model side

  has_one_attached :image, variants: {thumb: "100x100#"}

After making it possible to specify like this, route this option to ActiveStorage :: Blob again

ActiveSupport.on_load(:active_storage_blob) do
  prepend Module.new {
    def variants
      options[:variants] || {} #The one who ran around
    end

    def variant(style_or_transformations)
      if style_or_transformations.try(:to_sym) == :original
        self
      elsif variable? && variants[style_or_transformations]
        Variant.new(self, variants[style_or_transformations])
      else
        super
      end
    end
  }
end

It can be realized by.

Save the value in a Paperclip column

Imagine a situation where you discover a problem and switch back after releasing the ActiveStorage implementation and using it for a while. Since S3, which is a storage backend, is used in common with Paperclip/ActiveStorage, there is no problem, but since it is after ActiveStorage migration, newly uploaded files are the tables on the ActiveStorage side (ActiveStorage :: Attachment and ActiveStorage :: Blob in the model). ) Contains a value, but the column (* _file_name, * _ file_size ..., etc.) of each model used on the Paperclip side does not contain a value.

In this case, when switching back, it is necessary to move the reverse data from the Active Storage side to the Paper clip side. To prevent that, after uploading the file to the Active Storage side, try putting a process to write the value to the column used on the Paperclip side as well.

In the model

has_one_attached :image
after_save { image.replicate_for_paperclip! }

And patch ActiveStorage :: Attached :: One

ActiveStorage::Attached::One.prepend Module.new {
  def replicate_for_paperclip!
    return unless attached?
    attributes = {}
    attributes["#{name}_file_name"] = filename.to_s if record.attributes.has_key?("#{name}_file_name")
    attributes["#{name}_content_type"] = content_type if record.attributes.has_key?("#{name}_content_type")
    attributes["#{name}_file_size"] = byte_size if record.attributes.has_key?("#{name}_file_size")
    attributes["#{name}_updated_at"] = blob.created_at if record.attributes.has_key?("#{name}_updated_at")
    record.assign_attributes(attributes)
    record.save! if record.changed?
  end
}

Now you can fill the value in the Paperclip side column when uploading Active Storage.

Summary

We introduced the migration from Paperclip to ActiveStorage and how to make up for the missing features in ActiveStorage. We believe that the above policy has made it possible to implement the application with as low a risk as possible, even though it has undergone major changes related to the foundation of the application. (But the production release is yet to come. I hope nothing goes wrong ...)

I'm sure there are quite a few Rails applications that have been using Paperclip and haven't decided what to do in the future, so I hope this article is helpful.

I wish you a good file upload life!

Recommended Posts

The story of migrating from Paperclip to Active Storage
The story of raising Spring Boot from 1.5 series to 2.1 series part2
The story of migrating a stray batch without an owner from EC2 to a Docker environment
The story of RxJava suffering from NoSuchElementException
The story of switching from Amazon RDS for MySQL to Amazon Aurora Serverless
The story of introducing Ajax communication to ruby
The story of raising Spring Boot 1.5 series to 2.1 series
The story of adding the latest Node.js to DockerFile
From the introduction of devise to the creation of the users table
How to write Scala from the perspective of Java
Procedure to use S3 of LocalStack for Active Storage
Migrating from vargrant to docker
Get to the abbreviations from 5 examples of iterating Java lists
20190803_Java & k8s on Azure The story of going to the festival
The story of throwing BLOB data from EXCEL in DBUnit
How to get the longest information from Twitter as of 12/12/2016
The story of pushing Java to Heroku using the BitBucket pipeline
[Apache Tomcat] The story of using Apache OpenWebBeans to enable CDI
[Ruby on Rails] I want to get the URL of the image saved in Active Storage
[Java version] The story of serialization
The story of Collectors.groupingBy that I want to keep for posterity
The story of @ViewScoped consuming memory
The story of toString () starting with passing an array to System.out.println
From introduction to use of ActiveHash
The story I wanted to unzip
[Rails] How to use Active Storage
Notes on migrating from CircleCI 1.0 to 2.0
From fledgling Java (3 years) to Node.js (4 years). And the impression of returning to Java
From introduction to usage of byebug
The road from JavaScript to Java
[Gradle] [checkstyle] What to do if the active setting of Checkstyle is removed by "Refresh Gradle project" from Eclipse
The story of forgetting to close a file in Java and failing
The story of acquiring Java Silver in two months from completely inexperienced.
Confirmation and refactoring of the flow from request to controller in [httpclient]
The story of releasing the Android app to the Play Store for the first time.
Strict_loading function to suppress the occurrence of N + 1 problem added from rails 6.1
Iterative processing of Ruby using each method (find the sum from 1 to 10)
From Java9, the constructor of the class corresponding to primitive types is deprecated.
I translated the grammar of R and Java [Updated from time to time]
Change the half-width space of STS (Spring Tool Suite) from "u" to "・"
From pulling docker-image of rails to launching
The story of encountering Spring custom annotation
[Challenge CircleCI from 0] Learn the basics of CircleCI
The secret to the success of IntelliJ IDEA
Investigate the replacement from Docker to Podman.
[Ruby] From the basics to the inject method
The story of updating SonarQube's Docker Container
Story when moving from Spring Boot 1.5 to 2.1
How to determine the number of parallels
The story of AppClip support in Anyca
Changes when migrating from Spring Boot 1.5 to Spring Boot 2.0
Ruby from the perspective of other languages
Changes when migrating from Spring Boot 2.0 to Spring Boot 2.2
How to sort the List of SelectItem
Output of the book "Introduction to Java"
Precautions when migrating from VB6.0 to JAVA
The story of writing Java in Emacs
The process of introducing Vuetify to Rails
Find the difference from a multiple of 10
[Promotion of Ruby comprehension (1)] When switching from Java to Ruby, first understand the difference.
Summary of points I was worried about when migrating from java to kotlin