LinkExtractor

wljdeMacBook-Pro:~ wlj$ scrapy shell "http://www.bjrbj.gov.cn/mzhd/detail_29974.htm"

wljdeMacBook-Pro:~ wlj$ scrapy shell "http://www.bjrbj.gov.cn/mzhd/detail_29974.htm"

>>> response.url
‘https://item.btime.com/m_9b62d3a9239a9473c‘\

>>> from scrapy.linkextractors import LinkExtractor

>>> response.xpath(‘//div[@class="xx_neirong"]/h1/text()‘).extract()[0] 

‘北京社保开户流程是怎么个流程‘

 1 wljdeMacBook-Pro:Desktop wlj$ scrapy shell "http://hr.tencent.com/position.php?"
 2 2018-06-21 21:12:40 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: scrapybot)
 3 2018-06-21 21:12:40 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 18.4.0, Python 3.6.5 (default, Apr 25 2018, 14:23:58) - [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)], pyOpenSSL 18.0.0 (OpenSSL 1.1.0h  27 Mar 2018), cryptography 2.2.2, Platform Darwin-17.5.0-x86_64-i386-64bit
 4 2018-06-21 21:12:40 [scrapy.crawler] INFO: Overridden settings: {‘DUPEFILTER_CLASS‘: ‘scrapy.dupefilters.BaseDupeFilter‘, ‘LOGSTATS_INTERVAL‘: 0}
 5 2018-06-21 21:12:40 [scrapy.middleware] INFO: Enabled extensions:
 6 [‘scrapy.extensions.corestats.CoreStats‘,
 7  ‘scrapy.extensions.telnet.TelnetConsole‘,
 8  ‘scrapy.extensions.memusage.MemoryUsage‘]
 9 2018-06-21 21:12:40 [scrapy.middleware] INFO: Enabled downloader middlewares:
10 [‘scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware‘,
11  ‘scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware‘,
12  ‘scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware‘,
13  ‘scrapy.downloadermiddlewares.useragent.UserAgentMiddleware‘,
14  ‘scrapy.downloadermiddlewares.retry.RetryMiddleware‘,
15  ‘scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware‘,
16  ‘scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware‘,
17  ‘scrapy.downloadermiddlewares.redirect.RedirectMiddleware‘,
18  ‘scrapy.downloadermiddlewares.cookies.CookiesMiddleware‘,
19  ‘scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware‘,
20  ‘scrapy.downloadermiddlewares.stats.DownloaderStats‘]
21 2018-06-21 21:12:40 [scrapy.middleware] INFO: Enabled spider middlewares:
22 [‘scrapy.spidermiddlewares.httperror.HttpErrorMiddleware‘,
23  ‘scrapy.spidermiddlewares.offsite.OffsiteMiddleware‘,
24  ‘scrapy.spidermiddlewares.referer.RefererMiddleware‘,
25  ‘scrapy.spidermiddlewares.urllength.UrlLengthMiddleware‘,
26  ‘scrapy.spidermiddlewares.depth.DepthMiddleware‘]
27 2018-06-21 21:12:40 [scrapy.middleware] INFO: Enabled item pipelines:
28 []
29 2018-06-21 21:12:40 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
30 2018-06-21 21:12:40 [scrapy.core.engine] INFO: Spider opened
31 2018-06-21 21:12:41 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://hr.tencent.com/position.php> from <GET http://hr.tencent.com/position.php>
32 2018-06-21 21:12:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://hr.tencent.com/position.php> (referer: None)
33 [s] Available Scrapy objects:
34 [s]   scrapy     scrapy module (contains scrapy.Request, scrapy.Selector, etc)
35 [s]   crawler    <scrapy.crawler.Crawler object at 0x107617c18>
36 [s]   item       {}
37 [s]   request    <GET http://hr.tencent.com/position.php>
38 [s]   response   <200 https://hr.tencent.com/position.php>
39 [s]   settings   <scrapy.settings.Settings object at 0x10840e748>
40 [s]   spider     <DefaultSpider ‘default‘ at 0x1086c6ba8>
41 [s] Useful shortcuts:
42 [s]   fetch(url[, redirect=True]) Fetch URL and update local objects (by default, redirects are followed)
43 [s]   fetch(req)                  Fetch a scrapy.Request and update local objects 
44 [s]   shelp()           Shell help (print this help)
45 [s]   view(response)    View response in a browser
46 >>> response.url
47 ‘https://hr.tencent.com/position.php‘
48 >>> from scrapy.linkextractors import LinkExtractor
49 >>> link_list=LinkExtractor(allow=("start=\d+"))
50 >>> link_list.extract_links(response)
51 [Link(url=‘https://hr.tencent.com/position.php?&start=10#a‘, text=‘2‘, fragment=‘‘, nofollow=False), Link(url=‘https://hr.tencent.com/position.php?&start=20#a‘, text=‘3‘, fragment=‘‘, nofollow=False), Link(url=‘https://hr.tencent.com/position.php?&start=30#a‘, text=‘4‘, fragment=‘‘, nofollow=False), Link(url=‘https://hr.tencent.com/position.php?&start=40#a‘, text=‘5‘, fragment=‘‘, nofollow=False), Link(url=‘https://hr.tencent.com/position.php?&start=50#a‘, text=‘6‘, fragment=‘‘, nofollow=False), Link(url=‘https://hr.tencent.com/position.php?&start=60#a‘, text=‘7‘, fragment=‘‘, nofollow=False), Link(url=‘https://hr.tencent.com/position.php?&start=70#a‘, text=‘...‘, fragment=‘‘, nofollow=False), Link(url=‘https://hr.tencent.com/position.php?&start=3800#a‘, text=‘381‘, fragment=‘‘, nofollow=False)]
52 >>>