您现在的位置是: 网站首页> 学习笔记> scrapy scrapy
'RFPDupeFilter' has no attribute 'from_spider'
2021-12-18 [scrapy] 3413人已围观
今天新装的环境,64位python3.6.8,直接pip安装的scrapy与scrapy-redis
pip install scrapy
pip install scrapy-redis
新建项目,并启动爬虫时报如下异常:
Unhandled error in Deferred:
Traceback (most recent call last):
File "D:\pyenv\littlebee\lib\site-packages\scrapy\crawler.py", line 192, in crawl
return self._crawl(crawler, *args, **kwargs)
File "D:\pyenv\littlebee\lib\site-packages\scrapy\crawler.py", line 196, in _crawl
d = crawler.crawl(*args, **kwargs)
File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 1909, in unwindGenerator
return _cancellableInlineCallbacks(gen) # type: ignore[unreachable]
File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 1816, in _cancellableInlineCallbacks
_inlineCallbacks(None, gen, status)
--- <exception caught here> ---
File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 1661, in _inlineCallbacks
result = current_context.run(gen.send, result)
File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 63, in run
return f(*args, **kwargs)
File "D:\pyenv\littlebee\lib\site-packages\scrapy\crawler.py", line 89, in crawl
yield self.engine.open_spider(self.spider, start_requests)
builtins.AttributeError: type object 'RFPDupeFilter' has no attribute 'from_spider'
解决方法: 直接安装scrapy-redis默认早0.7.1,将版本降低到0.6.8即可解决
pip uninstall scrapy-redis
pip install scrapy-redis==0.6.8
解决上面问题后,重新启动爬虫,又会报以下异常:
Unhandled error in Deferred:
Traceback (most recent call last):
File "D:\pyenv\littlebee\lib\site-packages\scrapy\crawler.py", line 192, in crawl
return self._crawl(crawler, *args, **kwargs)
File "D:\pyenv\littlebee\lib\site-packages\scrapy\crawler.py", line 196, in _crawl
d = crawler.crawl(*args, **kwargs)
File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 1909, in unwindGenerator
return _cancellableInlineCallbacks(gen) # type: ignore[unreachable]
File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 1816, in _cancellableInlineCallbacks
_inlineCallbacks(None, gen, status)
--- <exception caught here> ---
File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 1661, in _inlineCallbacks
result = current_context.run(gen.send, result)
File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 63, in run
return f(*args, **kwargs)
File "D:\pyenv\littlebee\lib\site-packages\scrapy\crawler.py", line 89, in crawl
yield self.engine.open_spider(self.spider, start_requests)
builtins.ValueError: ("Failed to instantiate dupefilter class '%s': %s", 'scrapy.dupefilters.RFPDupeFilter', TypeError("__init__() got an unexpected keyword argument 'server'",))
在网上查了下资料,主要是没有设置DUPEFILTER_CLASS或者设置DUPEFILTER_CLASS错误,我就是把名字弄错了 正确的应该是:
DUPEFILTER_CLASS = 'scrapy_redis.dupefilter.RFPDupeFilter'
相关文章
文章评论
暂无评论添加评论
点击排行
本栏推荐
标签云
热评文章
- django使用qq邮箱发送邮件
- mysql8设置数据库远程连接
- pip修改下载源为国内源
- win10看不到win7共享的文件夹的解决方法
- SQLyog连接 Mysql 8.0.11 报error no.1251- Client does not support authentic...
- 使用Oracel Net Nanager配置Oracle数据库远程访问
- 将anaconda的下载源切换为国内的源
- Python+selenium+firefox设置代理IP
- selenium+firefox+js实现动态设置firefox浏览器代理IP
- scrapy文件下载(高新技术企业认定网)
- Python调用JS代码
- Chrome浏览器的overrides的使用
站点信息
- 建站时间:2021-01-01
- 网站程序:Django 3.1.2
- 文章统计:53篇
- 文章评论:31条
- 统计数据: