Currently, I am helping to collect data for restaurant consulting by outsourcing (scraping tabelog with Rails application and ruby script), but I had a hard time with an error in character encoding, so here is the solution I will share it.
environment
SyntaxError ((irb):1: invalid multibyte char (UTF-8))
When I ran the ruby scraping file from the terminal, I got this error.
Apparently, the character code is bad and an error is occurring. Windows has a default character code of ANSI when you save the file, and you need to change this character code to UTF-8 and save it!
** Just start Notepad and change the character code from ANSI to UTF-8 (no BOM recommended) when you drag and drop the file you want to change and save it as a name! ** ** Reference
It has nothing to do with programming. .. ww
I changed the code of the script from UTF-8 to cp932 and did various things, but it didn't make much sense. .. Lol
In the end, the code looks like this ↓
scrp_tabelog.rb
CSV.open('new_log_file.csv', "w", liberal_parsing: true) do |csv|
csv << ['Contract sequence number', 'Store name', 'date', 'Evaluation score', '', '', '', '', '', '', '', 'Number of reviews(Case)', 'Who went(Man)', '行きたいMan(Man)']
csv << ['', '', '', 'Comprehensive', 'Night', 'Noon', 'Cooking / taste', 'service', 'atmosphere', 'CP', 'Liquor / drink']
CSV.foreach('shop_point_list.csv', headers: false, liberal_parsing: true) do |recent_row|
#Write the process
end
end
The point here is ** liberal_parsing: true **. From ruby 2.4.0, it is an option added to CSV, and if this is set to true, it seems that double quotes appearing in the data can be parsed if the data is not enclosed in double quotes.
Now I managed to pull in the data without causing any errors!
We output what we have learned every day! We would appreciate it if you could comment on your impressions and suggestions!
Recommended Posts