[Ruby] Invalid multibyte char (UTF-8) Error resolution method [CSV]

Currently, I am helping to collect data for restaurant consulting by outsourcing (scraping tabelog with Rails application and ruby script), but I had a hard time with an error in character encoding, so here is the solution I will share it.

environment

Error code


SyntaxError ((irb):1: invalid multibyte char (UTF-8))

When I ran the ruby scraping file from the terminal, I got this error.

Apparently, the character code is bad and an error is occurring. Windows has a default character code of ANSI when you save the file, and you need to change this character code to UTF-8 and save it!

** Just start Notepad and change the character code from ANSI to UTF-8 (no BOM recommended) when you drag and drop the file you want to change and save it as a name! ** ** Reference

It has nothing to do with programming. .. ww

I changed the code of the script from UTF-8 to cp932 and did various things, but it didn't make much sense. .. Lol

By the way, this alone did not solve the error ...

In the end, the code looks like this ↓

scrp_tabelog.rb


CSV.open('new_log_file.csv', "w", liberal_parsing: true) do |csv|
  csv << ['Contract sequence number', 'Store name', 'date', 'Evaluation score', '', '', '', '', '', '', '', 'Number of reviews(Case)', 'Who went(Man)', '行きたいMan(Man)']
  csv << ['', '', '', 'Comprehensive', 'Night', 'Noon', 'Cooking / taste', 'service', 'atmosphere', 'CP', 'Liquor / drink']

  CSV.foreach('shop_point_list.csv', headers: false, liberal_parsing: true) do |recent_row|
    #Write the process
  end
end

The point here is ** liberal_parsing: true **. From ruby 2.4.0, it is an option added to CSV, and if this is set to true, it seems that double quotes appearing in the data can be parsed if the data is not enclosed in double quotes.

Now I managed to pull in the data without causing any errors!

Thank you for reading to the end!

We output what we have learned every day! We would appreciate it if you could comment on your impressions and suggestions!

Recommended Posts

[Ruby] Invalid multibyte char (UTF-8) Error resolution method [CSV]
[Ruby on Rails] undefined method ʻid'for nil: NilClass error resolution method
Spring Boot + PostgreSQL error resolution method
[Ruby on Rails] Follow function undefined method ʻid'for nil: NilClass error resolution
RSpec error resolution undefined method `feature'for RSpec: Module
[Ruby on Rails] NoMethodError undefined method `devise_for'error resolution