//*[@id="latest-items"]/div/ul/li/div[2]/a/span[2]

Just do. I will actually try it.

In [5]: response.xpath('//*[@id="latest-items"]/div/ul/li/div[2]/a/span[2]/text()').extract()
Out[5]:
['Engineers can't go home on premium Friday',
 'If you can't beat the machine, become a machine!',
 'Data in the cloud may disappear!',
 'What is the unexpectedly large number of paperless offices?',
 'Unfortunately common points of unpopular male engineers',
 'What you need to do when you challenge advanced programming work',
 'New Year's Day 2017 was a second longer than usual',
 'The latest situation of the engineer's advent calendar',
 'There are "unexpected enemies" in the Amazon cloud',
 'When will Mizuho Bank's system be completed?',
 'Do you remember the nostalgic "Konami Command"?',
 '"DV" has a different meaning in the engineer community',
 'Amazing evolution of the game over the last 40 years',
 '"Pit" hidden in long autumn night programming',
 'Former Sony engineers are popular at work']

I got all the titles. If you compare the above HTML and XPath, the title tag```…It seems that you can just specify directly, but this page is

 <div id = "ranking-items" style = "display: none;"> <!-In order of popularity->
  <div class="article-list ranking category">
    <ul class="ranked business">
      <li class="clearfix">
       ...
 <div id = "latest-items"> <!-Latest order->
  <div class="article-list">
    <ul class="business">
      <li class="clearfix">

The structure is almost the same as the latest order under the popularity order, so if you are not careful, extra data will be mixed. When I actually try it,

In [6]: response.xpath('//span[@class="column-main-ttl"]/text()').extract()
Out[6]:
 ['Engineers aren't playing with Pokemon GO! ',
 "When will Mizuho Bank's system be completed?",
 'Unfortunate commonality of unpopular male engineers',
 'Engineers can't go home on premium Friday',
 'Students who no longer know desktop PCs! ',
 'Why former Sony engineers are popular in the workplace',
 'Cloud data may disappear! ',
 "Why I don't envy Yahoo 3 days a week",
 "The memory of the first computer I bought is vivid",
 'What is the most profitable programming language',
 "Who is attracted to" Famicom Mini "?",
 'Programming has become a very popular lesson! ',
 "" Self-driving cars "do not run automatically",
 "The truth about engineer girls'same clothes and staying suspicions'",
 "New employees will learn the basics by" creating minutes "",
 'Engineers can't go home on premium Friday',
 'If you can't beat the machine, become a machine! ',
 'Cloud data may disappear! ',
 'What is the unexpectedly large number of paperless offices? ',
 'Unfortunate commonality of unpopular male engineers',
 'What you need to do when you challenge advanced programming work',
 'New Year's Day 2017 was a second longer than usual',
 'The latest situation of the engineer's advent calendar',
 "There are" unexpected enemies "in the Amazon cloud",
 "When will Mizuho Bank's system be completed?",
 "Do you remember the nostalgic" Konami Command "?",
 "In the engineer area," DV "has a different meaning",
 'Amazing evolution of the game in the last 40 years',
 "The" pitfalls "hidden in long autumn night programming",
 'Why former Sony engineers are popular at work']

The same data as is obtained twice. That is, the XPath must uniquely point to the required data.

##Extract the link The link URL to the manga posting page is the title when you look at the HTML<span>Parent tag<a>It is written in the href of. The XPath pointing to this looks like this:

//*[@id="latest-items"]/div/ul/li/div[2]/a/@href

Last@hrefRefers to the href attribute of the a tag. This time, I want to extract the attribute value of the a tag instead of the text node of the child of the a tag, so I am doing as above. When you actually move this

In [7]: response.xpath('//*[@id="latest-items"]/div/ul/li/div[2]/a/@href').extract()
Out[7]:
['/articles/-/161892',
 '/articles/-/159846',
 '/articles/-/157777',
 '/articles/-/153378',
 '/articles/-/153367',
 '/articles/-/152301',
 '/articles/-/152167',
 '/articles/-/149922',
 '/articles/-/149911',
 '/articles/-/146637',
 '/articles/-/146559',
 '/articles/-/144778',
 '/articles/-/144756',
 '/articles/-/142415',
 '/articles/-/142342']

It will be. Now that you have an XPath to get the title and link of each manga, you can retrieve the necessary information by creating a scraping program based on this.

##Export the acquired data If you only scrape once, you can output the required data as it is. First, save the scraped data in a variable and then output it to a file.

In [8]: titles = response.xpath('//*[@id="latest-items"]/div/ul/li/div[2]/a/span[2]/text()').extract()

In [9]: links = response.xpath('//*[@id="latest-items"]/div/ul/li/div[2]/a/@href').extract()

In [10]: f = open('bohebohe.txt', 'w')

In [11]: for title, link in zip(titles, links):
    ...:     f.write(title + ', ' + link + '\n')

In [12]: f.close()

with thisbohebohe.txtThe result of scraping was written to the file.

$ cat bohebohe.txt
 Engineers can't go home on Premium Friday, / articles /-/161892
 If you can't beat the machine, become a machine! , / articles /-/ 159846
 Data in the cloud may disappear! , / articles /-/ 157777
 What is the unexpectedly large number of paperless offices? , / articles /-/ 153378
 Unfortunately commonality of unpopular male engineers, / articles /-/ 1533767
 What you need to do to tackle advanced programming work, / articles /-/ 152301
 New Year's Day 2017 was a second longer than usual, / articles /-/ 152167
 Engineer's Advent Calendar Latest Circumstances, / articles /-/ 149922
 There are "unexpected enemies" in the Amazon cloud, / articles /-/ 149911
 When will Mizuho Bank's system be completed, / articles /-/ 146637
 Do you remember the nostalgic "Konami Command", / articles /-/ 146559
 "DV" has a different meaning in the engineer community, / articles /-/ 144778
 Amazing evolution of the game over the last 40 years, / articles /-/ 144756
 "Pitfalls" in long autumn night programming, / articles /-/ 142415
 Former Sony engineers are popular at work, / articles /-/ 142342

#in conclusion

Debugging XPath that specifies data while creating a program is a bit of a hassle, and sometimes it's a waste to write a program for something that is used only once. In such a case, the scrapy shell, which allows you to try various things interactively and run the python script as it is, is quite convenient, and it is useful for various experiments such as just wanting to extract a little data from the page created in the past.

#Bonus XPath A brief description of the XPath used in this article. As an example of HTML

 1: <div id="latest-items">
 2:  <div class="article-list">
 3:    <ul class="business">
 4:      <li class="clearfix">
 5:        <div class="ttl">
 6:          <a href="/articles/-/161892" class="link-box">
 7: <span class = "column-ttl"> Incorporated as work time </ span> <br>
 8: <span class = "column-main-ttl"> Engineers can't go home on Premium Friday </ span>
 9: <span class = "date"> March 12, 2017 </ span>
11:          </a>
12:        </div>
13:      </li>
14:    </ul>
15:  </div>
16:</div>

Is used.