目標(biāo)：爬取某一學(xué)習(xí)網(wǎng)站上課程信息，前期調(diào)試僅獲取課程名稱
爬蟲(chóng)文件：

import scrapy
from xtzx.items import XtzxItem

from scrapy.http import Request

class LessonSpider(scrapy.Spider):

name = 'lesson'
allowed_domains = ['xuetangx.com']
start_urls = ['http://www.xuetangx.com/courses/course-v1:TsinghuaX+80512073X+2018_T1/about']
'''
def start_requests(self):
    ua={"User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"}
    yield Request("www.xuetangx.com/courses/course-v1:TsinghuaX+80512073X+2018_T1/about",headers=ua)
'''
def parse(self, response):
    item=XtzxItem()
    item["title"]=response.xpath("http://div[@class='title_detail'/h3[@class='courseabout_title']/text()").extract()
    print(item["title"])

執(zhí)行日志：

2018-04-28 11:08:33 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: xtzx)
2018-04-28 11:08:33 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.7, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 17.9.0, Python 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)], pyOpenSSL 17.5.0 (OpenSSL 1.1.0h 27 Mar 2018), cryptography 2.2.2, Platform Windows-10-10.0.16299-SP0
2018-04-28 11:08:33 [scrapy.crawler] INFO: Overridden settings: {'SPIDER_MODULES': ['xtzx.spiders'], 'BOT_NAME': 'xtzx', 'NEWSPIDER_MODULE': 'xtzx.spiders', 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko'}
2018-04-28 11:08:33 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2018-04-28 11:08:34 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-04-28 11:08:34 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-04-28 11:08:34 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-04-28 11:08:34 [scrapy.core.engine] INFO: Spider opened

----------好像從這開(kāi)始出問(wèn)題

2018-04-28 11:08:34 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-04-28 11:08:34 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-04-28 11:08:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.xuetangx.com/cours...:TsinghuaX+80512073X+2018_T1/about> (referer: None)
2018-04-28 11:08:34 [scrapy.core.scraper] ERROR: Spider error processing <GET http://www.xuetangx.com/cours...:TsinghuaX+80512073X+2018_T1/about> (referer: None)
Traceback (most recent call last):
File "d:python3.5libsite-packagesparselselector.py", line 228, in xpath

**kwargs)

File "srclxmletree.pyx", line 1577, in lxml.etree._Element.xpath
File "srclxmlxpath.pxi", line 307, in lxml.etree.XPathElementEvaluator.__call__
File "srclxmlxpath.pxi", line 227, in lxml.etree._XPathEvaluatorBase._handle_result
lxml.etree.XPathEvalError: Invalid predicate

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "d:python3.5libsite-packagestwistedinternetdefer.py", line 653, in _runCallbacks

current.result = callback(current.result, *args, **kw)

File "E:pythonxtzxxtzxspiderslesson.py", line 16, in parse

item["title"]=response.xpath("http://div[@class='title_detail'/h3[@class='courseabout_title']/text()").extract()

File "d:python3.5libsite-packagesscrapyhttpresponsetext.py", line 119, in xpath

return self.selector.xpath(query, **kwargs)

File "d:python3.5libsite-packagesparselselector.py", line 232, in xpath

six.reraise(ValueError, ValueError(msg), sys.exc_info()[2])

File "d:python3.5libsite-packagessix.py", line 692, in reraise

raise value.with_traceback(tb)

File "d:python3.5libsite-packagesparselselector.py", line 228, in xpath

**kwargs)

File "srclxmletree.pyx", line 1577, in lxml.etree._Element.xpath
File "srclxmlxpath.pxi", line 307, in lxml.etree.XPathElementEvaluator.__call__
File "srclxmlxpath.pxi", line 227, in lxml.etree._XPathEvaluatorBase._handle_result
ValueError: XPath error: Invalid predicate in //div[@class='title_detail'/h3[@class='courseabout_title']/text()
2018-04-28 11:08:35 [scrapy.core.engine] INFO: Closing spider (finished)
2018-04-28 11:08:35 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 301,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 24409,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2018, 4, 28, 3, 8, 35, 118088),
'log_count/DEBUG': 2,
'log_count/ERROR': 1,
'log_count/INFO': 7,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'spider_exceptions/ValueError': 1,
'start_time': datetime.datetime(2018, 4, 28, 3, 8, 34, 418003)}
2018-04-28 11:08:35 [scrapy.core.engine] INFO: Spider closed (finished)

感覺(jué)程序很簡(jiǎn)單，但是就是不行，其他items都是常規(guī)的設(shè)置，pipelines里面沒(méi)有添加新的內(nèi)容，然后settings里面就修改了一下ROBOTSTXT_OBEY的值
網(wǎng)上查了很久這樣的錯(cuò)誤，都沒(méi)找到相應(yīng)的方法，也試過(guò)偽裝瀏覽器爬取也沒(méi)用，自學(xué)，沒(méi)有老師，完全沒(méi)轍了，求助各位.