About the problem of deadlock in parallel processing in gem'sprockets' 4.0

About gem'sprockets'

A library that compiles sweets files. However, starting with rails 5.1, webpacker is recommended for the javascript compiler, and from rails 6.0 the default for the javascript compiler has been changed to webpacker. Currently used as the default for CSS and image compilation.

CSS, image -> sprockets javascript -> webpacker

phenomenon

Execution result is not returned by asset precompilation.

Solutions

1. Set to disable concurrency.

By setting either of the following in the application settings, you can disable parallel processing and avoid problems.

Rails.config.assets.export_concurrent = false
Sprockets.export_concurrent = false

2. Use version 3 of sprockets

Maintenance of gem'sprockets' is slow, and it is expected that it will take time to fix, so it is one to use 3 series.

Details

Sprockets 4.0 is now multithreaded and parallel running to improve compilation performance. See: https://github.com/rails/sprockets/pull/469

Concurrency uses Concurrent :: Promise and thenwait!Inpromises.each (&: wait!).

promises = args.flatten.map do |path|
  Concurrent::Promise.execute(executor: executor) do
    environment.find_all_linked_assets(path) do |asset|
      yield asset
    end
  end
end
promises.each(&:wait!)

See: https://github.com/rails/sprockets/pull/469/files#diff-0d972076f2fbe7a73281fcc0282e57a1618bba231bc8023a4d5bc5705fb06da7R123-R130

At this time, the wait! method waits for the timeout time until the processing is completed, but since no argument is specified, nil is entered and it keeps waiting forever.

def wait!(timeout = nil)
  wait(timeout).tap { raise self if rejected? }
end

See: https://github.com/ruby-concurrency/concurrent-ruby/blob/ffed3c3c0518030b0ed245637703089fa1f0eeee/lib/concurrent/concern/obligation.rb#L79-L89

If a deadlock occurs during concurrency, the asset precompilation process will never end. It seems that multiple threads are deadlocking trying to compile the same file during concurrency. A bug in ruby-concurrency? There is a possibility of. https://github.com/ruby-concurrency/concurrent-ruby/issues/870

There are also many similar issues. See: https://github.com/rails/sprockets/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+thread+OR+concurrent

reference

https://github.com/rails/sprockets/issues/640 https://github.com/rails/sprockets/issues/581 https://github.com/ruby-concurrency/concurrent-ruby/issues/870 https://github.com/ruby-concurrency/concurrent-ruby/blob/ffed3c3c0518030b0ed245637703089fa1f0eeee/lib/concurrent/synchronization/lockable_object.rb#L6-L20

Other

For discussions on sprockets, you can also study the WTF page below. https://discuss.rubyonrails.org/t/sprockets-abandonment/74371/24

Recommended Posts

About the problem of deadlock in parallel processing in gem'sprockets' 4.0
Order of processing in the program
About the idea of anonymous classes in Java
Think about the JAVA = JAVAscript problem (needed in the future)
About the handling of Null
About the description of Docker-compose.yml
Explanation of the FizzBuzz problem
Measured parallel processing in Java
About the behavior of ruby Hash # ==
About the symbol <%%> in Rails erb
Implementation of asynchronous processing in Tomcat
A memorandum of the FizzBuzz problem
About the role of the initialize method
I thought about the strategy of introducing Combine in iOS development
Expression used in the fizz_buzz problem
Think about the 7 rules of Optional
Problem that the attribute of User model becomes nil in ActionMailer
About file copy processing in Java
Summary about the introduction of Device
About the log level of java.util.logging.Logger
The problem that the contents of params are completely displayed in the [Rails] view
[Ruby basics] About the role of true and break in the while statement
(Determine in 1 minute) About the proper use of empty ?, blank? And present?
About the continuous division method learned in the 4th grade of elementary school
[Java] java.lang.NoClassDefFoundError: Addressing the problem of falling in org / jsoup / safety / Whitelist
About the phenomenon that StackOverflowError occurs in processing using Java regular expressions
Get the result of POST in Java
About the version of Docker's Node.js image
[GCD] Basics of parallel programming in Swift
Implementation of multi-tenant asynchronous processing in Tomcat
About the operation of next () and nextLine ()
I investigated the internal processing of Retrofit
The story of AppClip support in Anyca
About the initial display of Spring Framework
About UI thread processing in Android asynchronous
About the error message Invalid redeclaration of'***'
About the treatment of BigDecimal (with reflection)
About the number of threads of Completable Future
The story of writing Java in Emacs
About the mechanism of the Web and HTTP
Write the movement of Rakefile in the runbook
Creating a sample program using the problem of a database specialist in DDD Improvement 2
Character string comparison: I was caught in the skill check problem of Paiza
Be careful about upgrade if you use | etc. in the URL of Tomcat
Creating a sample program using the problem of a database specialist in DDD Improvement 1
About the meaning of type variables, E, T, etc. used in generics used in Java
[Order method] Set the order of data in Rails
The story of low-level string comparison in Java
[Java] Handling of JavaBeans in the method chain
About the confusion seen in startup Java servers
Think about the combination of Servlet and Ajax
The story of making ordinary Othello in Java
About the official start guide of Spring Framework
About the description order of Java system properties
A story about the JDK in the Java 11 era
Parallel and parallel processing in various languages (Java edition)
The story of learning Java in the first programming
Feel the passage of time even in Java
About next () and nextLine () of the Scanner class
Import files of the same hierarchy in Java
Offline real-time how to write Implementation example of the problem in E05 (ruby, C11)