您现在的位置是：网站首页> 学习笔记> scrapy scrapy

'RFPDupeFilter' has no attribute 'from_spider'

2021-12-18 [scrapy] 3682人已围观

今天新装的环境，64位python3.6.8,直接pip安装的scrapy与scrapy-redis

pip install scrapy
pip install scrapy-redis

新建项目，并启动爬虫时报如下异常：

Unhandled error in Deferred:

Traceback (most recent call last):
  File "D:\pyenv\littlebee\lib\site-packages\scrapy\crawler.py", line 192, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "D:\pyenv\littlebee\lib\site-packages\scrapy\crawler.py", line 196, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 1909, in unwindGenerator
    return _cancellableInlineCallbacks(gen)  # type: ignore[unreachable]
  File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 1816, in _cancellableInlineCallbacks
    _inlineCallbacks(None, gen, status)
--- <exception caught here> ---
  File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 1661, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 63, in run
    return f(*args, **kwargs)
  File "D:\pyenv\littlebee\lib\site-packages\scrapy\crawler.py", line 89, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
builtins.AttributeError: type object 'RFPDupeFilter' has no attribute 'from_spider'

解决方法：直接安装scrapy-redis默认早0.7.1,将版本降低到0.6.8即可解决

pip uninstall scrapy-redis
pip install scrapy-redis==0.6.8

解决上面问题后，重新启动爬虫，又会报以下异常：

Unhandled error in Deferred:

Traceback (most recent call last):
  File "D:\pyenv\littlebee\lib\site-packages\scrapy\crawler.py", line 192, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "D:\pyenv\littlebee\lib\site-packages\scrapy\crawler.py", line 196, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 1909, in unwindGenerator
    return _cancellableInlineCallbacks(gen)  # type: ignore[unreachable]
  File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 1816, in _cancellableInlineCallbacks
    _inlineCallbacks(None, gen, status)
--- <exception caught here> ---
  File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 1661, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "D:\pyenv\littlebee\lib\site-packages\twisted\internet\defer.py", line 63, in run
    return f(*args, **kwargs)
  File "D:\pyenv\littlebee\lib\site-packages\scrapy\crawler.py", line 89, in crawl
    yield self.engine.open_spider(self.spider, start_requests)
builtins.ValueError: ("Failed to instantiate dupefilter class '%s': %s", 'scrapy.dupefilters.RFPDupeFilter', TypeError("__init__() got an unexpected keyword argument 'server'",))