I was able to convert it to an exe with Pyinstaller, but it crashed when I started the exe file. Now that I know how to make it work perfectly, make a note of it.
OS:Windows 10 Language: Python 3.7 scrapy version: 1.7.3.
You have created a scrapy spider, and you have a script file that runs the scrapy file.
The following configuration demo_scrapy/ ├── scrapy.cfg ├── lauch_demo.py ← This is the script file that runs the scrapy file └── demo_scrapy ├── init.py ├── pycache ├── items.py ├── middlewares.py ├── pipelines.py ├── settings.py └── spiders ├── init.py ├── demo_spider.py └── pycache
Contents of lauch_demo.py import datetime import time from scrapy.spiderloader import SpiderLoader from scrapy.crawler import CrawlerProcess from scrapy.utils.project import get_project_settings
now = datetime.datetime.now() dt_now_date = now.strftime("%Y-%m-%d-%H%M%S") settings = get_project_settings()
settings.set('FEED_URI', 'demo' + '_' + dt_now_date + '.csv') process = CrawlerProcess(settings) process.crawl('demo_spider', domain='demo.com') process.start() # the script will block here until the crawling is finished
print ("completed")
Here, when launch_demo.py is started as usual, it works normally, However, the exe file does not work even if it is converted to exe with Pyinstaller.
Perform the following method.
Create a scrapy folder in any location for the VERSION and mime.types files and save it there. VERSION and mime.types C: \ Users \ username \ AppData \ Local \ Programs \ Python \ Python37-32 \ Lib \ site-packages \ scrapy It is stored in.
Add hidden-import. At the command prompt pyinstaller --hidden-import scrapy.spiderloader --hidden-import scrapy.statscollectors --hidden-import scrapy.logformatter --hidden-import scrapy.extensions --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.corestats --hidden-import scrapy.extensions.telnet --hidden-import scrapy.extensions.memusage --hidden-import scrapy.extensions.memdebug --hidden-import scrapy.extensions.closespider --hidden-import scrapy.extensions.feedexport --hidden-import scrapy.extensions.logstats --hidden-import scrapy.extensions.spiderstate --hidden-import scrapy.extensions.throttle --hidden-import scrapy.core.scheduler --hidden-import scrapy.squeues --hidden-import queuelib --hidden-import scrapy.core.downloader --hidden-import scrapy.downloadermiddlewares --hidden-import scrapy.downloadermiddlewares.robotstxt --hidden-import scrapy.downloadermiddlewares.httpauth --hidden-import scrapy.downloadermiddlewares.downloadtimeout --hidden-import scrapy.downloadermiddlewares.defaultheaders --hidden-import scrapy.downloadermiddlewares.useragent --hidden-import scrapy.downloadermiddlewares.retry --hidden-import scrapy.downloadermiddlewares.ajaxcrawl --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.httpcompression --hidden-import scrapy.downloadermiddlewares.redirect --hidden-import scrapy.downloadermiddlewares.cookies --hidden-import scrapy.downloadermiddlewares.httpproxy --hidden-import scrapy.downloadermiddlewares.stats --hidden-import scrapy.downloadermiddlewares.httpcache --hidden-import scrapy.spidermiddlewares --hidden-import scrapy.spidermiddlewares.httperror --hidden-import scrapy.spidermiddlewares.offsite --hidden-import scrapy.spidermiddlewares.referer --hidden-import scrapy.spidermiddlewares.urllength --hidden-import scrapy.spidermiddlewares.depth --hidden-import scrapy.pipelines --hidden-import scrapy.dupefilters --hidden-import scrapy.core.downloader.handlers.datauri --hidden-import scrapy.core.downloader.handlers.file --hidden-import scrapy.core.downloader.handlers.http --hidden-import scrapy.core.downloader.handlers.s3 --hidden-import scrapy.core.downloader.handlers.ftp --hidden-import scrapy.core.downloader.webclient --hidden-import scrapy.core.downloader.contextfactory "C:/path/to/launch_demo.py"
To execute. If you get "module not found XXX", add --hidden-import XXX.
Store the scrapy folder created in 1 in the created. \ Dist \ lauch_demo \ folder.
Running the exe file works fine.
How to create a single executable file in windows 10 with scrapy and pyinstaller? https://stackoverflow.com/questions/55331478/how-to-create-a-single-executable-file-in-windows-10-with-scrapy-and-pyinstaller
Pyinstaller scrapy error: https://stackoverflow.com/questions/25557693/pyinstaller-scrapy-error
If you have any questions, please let us know. Thank you very much.
Recommended Posts