[PYTHON] Scraping google search (image)

I referred to the following site. https://qiita.com/taedookim/items/63759e79426514c8a729

Clone from github

Use google-images-download. The head family seems to have stopped updating, so from the fork destination.

git clone https://github.com/Joeclinton1/google-images-download.git

Copy the following file to the location you want to use, edit a little

Download folder \ google-images-download \ google_images_download \ google_images_download.py

google_images_download.py line 935

NG: json_file = json.load(open(arguments['config_file']))
OK: json_file = json.load(open(arguments['config_file'], encoding='utf-8'))

Now you can search in Japanese.

Create the following python file in the copy destination folder

image_scraper.py

import os

from google_images_download import googleimagesdownload   #importing the library

#Change the current directory to the folder containing the executable file
os.chdir(os.path.dirname(os.path.abspath(__file__)))
print('Changed current working directory')

response = googleimagesdownload()   #class instantiation
paths = response.download({"config_file": "config.json"})   #passing the arguments to the function
print(paths)   #printing absolute paths of the downloaded images

Copy the version of Chromedriver you are using to the same folder

Download from below https://chromedriver.chromium.org/downloads

Rename and edit sample_config.json to config.json

Once this is done, it will be available. It seems that you can set multiple search queries.

Renaming is unnecessary, but sample will be inappropriate.

config.json

{
    "Records": [
        {
            "keywords": "apple",
            "limit": 5,
            "color": "green",
            "print_urls": true
        },
        {
            "keywords": "universe",
            "limit": 15,
            "size": "large",
            "print_urls": true
        }
    ]
}

Run image_scraper.py

From VS Code or from the console.

How to use config.json

Details (English) https://google-images-download.readthedocs.io/en/latest/arguments.html

Basic structure

Observe the following structure. If "limit" is 100 or more, "chrome driver" is required.

{
    "Records": [
        {
            "keywords": "hoge",
            "limit": 777,
            "format": "png",
            "print_urls": true,
            "chromedriver": "chromedriver.exe"
        }
    ]
}

Valid options, choices, etc.