從管道中總是拿不到數(shù)據(jù),但是如果直接在執(zhí)行時(shí)使用:
$ scrapy crawl pythonname -o -t
是可以拿到數(shù)據(jù)的,不知道問題出在哪百度半天無果,請(qǐng)各位指教,代碼如下:
douban.py
# -*- coding: utf-8 -*-
import scrapy
from text3_scrapydouban.items import Text3Item
class DoubanSpider(scrapy.Spider):
name = 'douban'
# allowed_domains = ['https://book.tuicool.com']
count=0
urls= 'https://book.douban.com/tag/小說?start='
start_urls = [ urls + str(count)+ '&type=T']
def parse(self, response):
items = []
node_list = response.xpath("http://div[@class='info']")#拿到最外層div
for node in node_list:
item = Text3Item() # itemd 的類用來存儲(chǔ)信息
# extract 將 Xpath 對(duì)象轉(zhuǎn)換為 Unicode方便解讀
bookname = node.xpath("./h2/a/text()").extract()[0].strip() # rstrip()將右邊空格去掉
title = node.xpath("./div[1]/text()").extract()[0].split("/")
author = title[0].strip()
publishTime = title[-2]
bookprice = title[-1].strip()
grade = node.xpath("./div[2]/span[2]/text()").extract()[0] # 等級(jí)
evaluate = node.xpath("./p/text()").extract()[0]
item['bookname'] = bookname
item["author"] = author
item["time"] = publishTime
item["price"] = bookprice
a = "分"
item["grade"] = grade
item["evaluate"] = evaluate
items.append(item)
return items # 引擎自動(dòng)判斷此為item數(shù)據(jù),使用會(huì)自動(dòng)交給管道
piplines.py
class Text3ScrapydoubanPipeline(object):
def __init__(self):
self.file = open('dou.txt', 'wb')
def process_item(self, item, spider):
self.file.write(item)
return item
def spider_closed(self, spider):
self.file.close()
items.py
import scrapy
class Text3Item(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
bookname = scrapy.Field()
author = scrapy.Field()
time = scrapy.Field()
price = scrapy.Field()
grade = scrapy.Field() # 豆瓣口碑
evaluate = scrapy.Field() # 簡述
settings 設(shè)置是正確的
運(yùn)行結(jié)果:
F:\GitHubCode\Code\Python\text3_scrapydouban>scrapy crawl douban
2018-02-03 14:39:49 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: text3_scrapydouban)
2018-02-03 14:39:49 [scrapy.utils.log] INFO: Versions: lxml 4.1.1.0, libxml2 2.9.5, cssselect 1.0.3, parsel 1.3.1, w3lib 1.19.0, Twisted 17.9.0, Python 3.6.4 (v3.6.4:d48eceb, Dec 19 2017, 06:54:40) [MSC v.1900 64 bit (AMD64)], pyOpenSSL 17.5.0 (OpenSSL 1.1.0g 2 Nov 2017), cryptography 2.1.4, Platform Windows-10-10.0.14393-SP0
2018-02-03 14:39:49 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'text3_scrapydouban', 'NEWSPIDER_MODULE': 'text3_scrapydouban.spiders', 'SPIDER_MODULES': ['text3_scrapydouban.spiders'], 'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)'}
2018-02-03 14:39:49 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2018-02-03 14:39:49 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-02-03 14:39:49 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-02-03 14:39:49 [scrapy.middleware] INFO: Enabled item pipelines:
['text3_scrapydouban.pipelines.Text3ScrapydoubanPipeline']
2018-02-03 14:39:49 [scrapy.core.engine] INFO: Spider opened
2018-02-03 14:39:49 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-02-03 14:39:49 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-02-03 14:39:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://book.douban.com/tag/%E5%B0%8F%E8%AF%B4?start=0&type=T> (referer: None)
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[日] 東野圭吾',
'bookname': '解憂雜貨店',
'evaluate': '現(xiàn)代人內(nèi)心流失的東西,這家雜貨店能幫你找回——\n'
'僻靜的街道旁有一家雜貨店,只要寫下煩惱投進(jìn)卷簾門的投信口,第二天就會(huì)在店后的牛奶箱里得到回答。\n'
'因男友身患絕... ',
'grade': '8.6',
'price': '39.50元',
'time': ' 2014-5 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[英] 肯·福萊特',
'bookname': '巨人的隕落',
'evaluate': '在第一次世界大戰(zhàn)的硝煙中,每一個(gè)邁向死亡的生命都在熱烈地生長——威爾士的礦工少年、剛失戀的美國法律系大學(xué)生、窮困潦倒的俄國兄弟、富有英俊的英格蘭伯爵,以及癡... ',
'grade': '8.9',
'price': '129.80元',
'time': ' 2016-5-1 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '亦舒',
'bookname': '我的前半生',
'evaluate': '一個(gè)三十幾歲的美麗女人子君,在家做全職家庭主婦。卻被一個(gè)平凡女人奪走丈夫,一段婚姻的失敗,讓女主角不得不堅(jiān)強(qiáng),變得更美麗,有了事業(yè),最終遇見一個(gè)更值得愛的男... ',
'grade': '7.9',
'price': '22.00元',
'time': ' 2007-8 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[哥倫比亞] 加西亞·馬爾克斯',
'bookname': '百年孤獨(dú)',
'evaluate': '《百年孤獨(dú)》是魔幻現(xiàn)實(shí)主義文學(xué)的代表作,描寫了布恩迪亞家族七代人的傳奇故事,以及加勒比海沿岸小鎮(zhèn)馬孔多的百年興衰,反映了拉丁美洲一個(gè)世紀(jì)以來風(fēng)云變幻的歷史。... ',
'grade': '9.2',
'price': '39.50元',
'time': ' 2011-6 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[美] 卡勒德·胡賽尼',
'bookname': '追風(fēng)箏的人',
'evaluate': '12歲的阿富汗富家少爺阿米爾與仆人哈桑情同手足。然而,在一場風(fēng)箏比賽后,發(fā)生了一件悲慘不堪的事,阿米爾為自己的懦弱感到自責(zé)和痛苦,逼走了哈桑,不久,自己也跟... ',
'grade': '8.9',
'price': '29.00元',
'time': ' 2006-5 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '林奕含',
'bookname': '房思琪的初戀樂園',
'evaluate': '令人心碎卻無能為力的真實(shí)故事。\n'
'性、權(quán)力、升學(xué)主義──青澀的表皮、變態(tài)社會(huì)的日常\n'
'如果這件事情正在發(fā)生,我們要怎麼假裝世界上沒有人以強(qiáng)暴小女孩為樂?\n'
'「我下... ',
'grade': '8.8',
'price': 'NTD320',
'time': ' 2017-2-7 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[意] 埃萊娜·費(fèi)蘭特',
'bookname': '新名字的故事',
'evaluate': '只有你身為女人 才會(huì)知道這些丑陋的秘密\n'
'兩個(gè)女人 50年的友誼和戰(zhàn)爭\n'
'如何成為那個(gè)更強(qiáng)大的她,又不被她戰(zhàn)勝\n'
'那不勒斯四部曲NO.2\n'
'《我的天才女友》后續(xù), ... ',
'grade': '9.1',
'price': '59.00元',
'time': ' 2017-4 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[美] 戴維·伽特森',
'bookname': '雪落香杉樹',
'evaluate': '編輯推薦:\n'
'1995福克納獎(jiǎng)得主\n'
'占據(jù)《紐約時(shí)報(bào)》暢銷榜首數(shù)月\n'
'美國書商協(xié)會(huì)年度之選\n'
'被譯成30余種文字,全球暢銷500萬冊(cè)\n'
'當(dāng)代美國文學(xué)史上罕見的超級(jí)暢銷... ',
'grade': '8.5',
'price': '52.00元',
'time': ' 2017-6 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[哥倫比亞] 加西亞·馬爾克斯',
'bookname': '霍亂時(shí)期的愛情',
'evaluate': '★馬爾克斯唯一正式授權(quán),首次完整翻譯\n'
'★《霍亂時(shí)期的愛情》是我最好的作品,是我發(fā)自內(nèi)心的創(chuàng)作。——加西亞?馬爾克斯\n'
'★這部光芒閃耀、令人心碎的作品是人類有史... ',
'grade': '9.0',
'price': '39.50元',
'time': ' 2012-9-1 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[俄] 維克托·阿斯塔菲耶夫',
'bookname': '魚王',
'evaluate': '長篇小說《魚王》是阿斯塔菲耶夫最具個(gè)性的一部代表作,俄羅斯當(dāng)代文學(xué)的經(jīng)典。全書由十三個(gè)內(nèi)容相對(duì)獨(dú)立的“敘事短篇小說”組成,全部圍繞著人與自然的關(guān)系,深入細(xì)致... ',
'grade': '9.0',
'price': '78.00元',
'time': ' 2017-4 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[英] 喬治·奧威爾',
'bookname': '1984',
'evaluate': '★村上春樹以《1Q84》向本書致敬\n'
'★著名學(xué)者劉紹銘經(jīng)典譯本內(nèi)地首次出版\n'
'★62種文字風(fēng)靡110個(gè)國家,全球銷量超過5000萬冊(cè)\n'
'★《時(shí)代周刊》“最好的10... ',
'grade': '9.3',
'price': '28.00',
'time': ' 2010-4-1 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[英] 肯·福萊特',
'bookname': '世界的凜冬',
'evaluate': '我親眼目睹,每一個(gè)邁向死亡的生命都在熱烈地生長。\n'
'全球讀者平均3個(gè)通宵讀完的超級(jí)小說《巨人的隕落》的續(xù)篇!\n'
'火遍全球的20世紀(jì)人類史詩“世紀(jì)三部曲”第二部。... ',
'grade': '9.0',
'price': '132.00(全三冊(cè))',
'time': ' 2017-3-1 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[英] 毛姆',
'bookname': '月亮和六便士',
'evaluate': '一個(gè)英國證券交易所的經(jīng)紀(jì)人,本已有牢靠的職業(yè)和地位、美滿的家庭,但卻迷戀上繪畫,像“被魔鬼附了體”,突然棄家出走,到巴黎去追求繪畫的理想。他的行徑?jīng)]有人能夠... ',
'grade': '9.0',
'price': '15.00元',
'time': ' 2006-8 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '余華',
'bookname': '活著',
'evaluate': '地主少爺福貴嗜賭成性,終于賭光了家業(yè)一貧如洗,窮困之中的福貴因?yàn)槟赣H生病前去求醫(yī),沒想到半路上被國民黨部隊(duì)抓了壯丁,后被解放軍所俘虜,回到家鄉(xiāng)他才知道母親已... ',
'grade': '9.1',
'price': '12.00元',
'time': ' 1998-5 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[美] 哈珀·李',
'bookname': '殺死一只知更鳥',
'evaluate': '成長總是個(gè)讓人煩惱的命題。成長有時(shí)會(huì)很緩慢,如小溪般唱著叮咚的歌曲趟過,有時(shí)卻如此突如其來,如暴雨般劈頭蓋臉……三個(gè)孩子因?yàn)樾℃?zhèn)上的幾樁冤案經(jīng)歷了猝不及防的... ',
'grade': '9.2',
'price': '32.00元',
'time': ' 2012-9 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '錢鍾書',
'bookname': '圍城',
'evaluate': '《圍城》是錢鐘書所著的長篇小說。第一版于1947年由上海晨光出版公司出版。1949年之后,由于政治等方面的原因,本書長期無法在中國大陸和臺(tái)灣重印,僅在香港出... ',
'grade': '8.9',
'price': '19.00',
'time': ' 1991-2 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[美] 庫爾特·馮內(nèi)古特',
'bookname': '囚鳥',
'evaluate': '我們都是受困于時(shí)代的“囚鳥”——既渴望逃離,又踟躕不前。\n'
'聯(lián)邦最低限度安保措施成人改造所里,頹唐的小老頭兒瓦爾特?斯代布克正在等待領(lǐng)他出獄的獄卒。\n'
'在他過去... ',
'grade': '8.0',
'price': '38.00元',
'time': ' 2017-6 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[美] 約翰·威廉斯',
'bookname': '斯通納',
'evaluate': '《斯通納》講述了生命中最重要的部分:愛,認(rèn)同,憐憫,志業(yè),傲骨,信任與死亡。\n'
'一個(gè)勇者有過的失敗不失意的人生:即使不能擁有完美的生活,所幸追求過完整的自我。... ',
'grade': '8.8',
'price': '39.00元',
'time': ' 2016-1 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '[意] 埃萊娜·費(fèi)蘭特',
'bookname': '我的天才女友',
'evaluate': '只有你身為女人才會(huì)知道這些丑陋的秘密\n'
'兩個(gè)女人,50年的友誼和戰(zhàn)爭\n'
'過去五年,幾乎所有歐美讀者都在談?wù)撍?、\n'
'全球暢銷百萬冊(cè) 被翻譯成40種語言\n'
'《金融時(shí)報(bào)》... ',
'grade': '8.5',
'price': '42.00元',
'time': ' 2017-1 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.scraper] ERROR: Error processing {'author': '路遙',
'bookname': '平凡的世界(全三部)',
'evaluate': '《平凡的世界》是一部現(xiàn)實(shí)主義小說,也是一部小說形式的家族史。作者濃縮了中國西北農(nóng)村的歷史變遷過程,在小說中全景式地表現(xiàn)了中國當(dāng)代城鄉(xiāng)的社會(huì)生活。在近十年的廣... ',
'grade': '9.0',
'price': '64.00元',
'time': ' 2005-1 '}
Traceback (most recent call last):
File "e:\software\python36\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "F:\GitHubCode\Code\Python\text3_scrapydouban\text3_scrapydouban\pipelines.py", line 16, in process_item
self.file.write(item)
TypeError: a bytes-like object is required, not 'Text3Item'
2018-02-03 14:39:50 [scrapy.core.engine] INFO: Closing spider (finished)
2018-02-03 14:39:50 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 324,
'downloader/request_count': 1,
'downloader/request_method_count/GET': 1,
'downloader/response_bytes': 13916,
'downloader/response_count': 1,
'downloader/response_status_count/200': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2018, 2, 3, 6, 39, 50, 457633),
'log_count/DEBUG': 2,
'log_count/ERROR': 20,
'log_count/INFO': 7,
'response_received_count': 1,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2018, 2, 3, 6, 39, 49, 703712)}
2018-02-03 14:39:50 [scrapy.core.engine] INFO: Spider closed (finished)報(bào)了這個(gè)異常
TypeError: a bytes-like object is required, not 'Text3Item'
主要是這一行報(bào)錯(cuò):
self.file.write(item)
item 類型要求是 bytes 的
可以改成這樣:
self.file.write(json.dumps(item))
主要代碼:
import json
class Text3ScrapydoubanPipeline(object):
def __init__(self):
self.file = open('dou.txt', 'wb')
def process_item(self, item, spider):
self.file.write(json.dumps(item))
return item
def spider_closed(self, spider):
self.file.close()北大青鳥APTECH成立于1999年。依托北京大學(xué)優(yōu)質(zhì)雄厚的教育資源和背景,秉承“教育改變生活”的發(fā)展理念,致力于培養(yǎng)中國IT技能型緊缺人才,是大數(shù)據(jù)專業(yè)的國家
達(dá)內(nèi)教育集團(tuán)成立于2002年,是一家由留學(xué)海歸創(chuàng)辦的高端職業(yè)教育培訓(xùn)機(jī)構(gòu),是中國一站式人才培養(yǎng)平臺(tái)、一站式人才輸送平臺(tái)。2014年4月3日在美國成功上市,融資1
北大課工場是北京大學(xué)校辦產(chǎn)業(yè)為響應(yīng)國家深化產(chǎn)教融合/校企合作的政策,積極推進(jìn)“中國制造2025”,實(shí)現(xiàn)中華民族偉大復(fù)興的升級(jí)產(chǎn)業(yè)鏈。利用北京大學(xué)優(yōu)質(zhì)教育資源及背
博為峰,中國職業(yè)人才培訓(xùn)領(lǐng)域的先行者
曾工作于聯(lián)想擔(dān)任系統(tǒng)開發(fā)工程師,曾在博彥科技股份有限公司擔(dān)任項(xiàng)目經(jīng)理從事移動(dòng)互聯(lián)網(wǎng)管理及研發(fā)工作,曾創(chuàng)辦藍(lán)懿科技有限責(zé)任公司從事總經(jīng)理職務(wù)負(fù)責(zé)iOS教學(xué)及管理工作。
浪潮集團(tuán)項(xiàng)目經(jīng)理。精通Java與.NET 技術(shù), 熟練的跨平臺(tái)面向?qū)ο箝_發(fā)經(jīng)驗(yàn),技術(shù)功底深厚。 授課風(fēng)格 授課風(fēng)格清新自然、條理清晰、主次分明、重點(diǎn)難點(diǎn)突出、引人入勝。
精通HTML5和CSS3;Javascript及主流js庫,具有快速界面開發(fā)的能力,對(duì)瀏覽器兼容性、前端性能優(yōu)化等有深入理解。精通網(wǎng)頁制作和網(wǎng)頁游戲開發(fā)。
具有10 年的Java 企業(yè)應(yīng)用開發(fā)經(jīng)驗(yàn)。曾經(jīng)歷任德國Software AG 技術(shù)顧問,美國Dachieve 系統(tǒng)架構(gòu)師,美國AngelEngineers Inc. 系統(tǒng)架構(gòu)師。