Python爬蟲錯誤twisted.web.error.SchemeNotSupported: Unsupported scheme: b''

在使用scrapy爬蟲時，出現(xiàn)了下面了的錯誤

Traceback (most recent call last):
  File "E:\project\venv\lib\site-packages\twisted\internet\defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "E:\project\venv\lib\site-packages\twisted\python\failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "E:\project\venv\lib\site-packages\scrapy\core\downloader\middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File "E:\project\venv\lib\site-packages\scrapy\utils\defer.py", line 45, in mustbe_deferred
    result = f(*args, **kw)
  File "E:\project\venv\lib\site-packages\scrapy\core\downloader\handlers\__init__.py", line 65, in download_request
    return handler.download_request(request, spider)
  File "E:\project\venv\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 67, in download_request
    return agent.download_request(request)
  File "E:\project\venv\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 331, in download_request
    method, to_bytes(url, encoding='ascii'), headers, bodyproducer)
  File "E:\project\venv\lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 252, in request
    proxyEndpoint = self._getEndpoint(self._proxyURI)
  File "E:\project\venv\lib\site-packages\twisted\web\client.py", line 1635, in _getEndpoint
    return self._endpointFactory.endpointForURI(uri)
  File "E:\project\venv\lib\site-packages\twisted\web\client.py", line 1513, in endpointForURI
    raise SchemeNotSupported("Unsupported scheme: %r" % (uri.scheme,))
twisted.web.error.SchemeNotSupported: Unsupported scheme: b''

原因

經(jīng)調(diào)查是在下載中間件中設(shè)置代理是出現(xiàn)了錯誤：

class IpProxyDownloadMiddleware(object):
    PROXIES = ['110.52.235.131:9999','110.52.235.249:9999','112.17.38.141:3128']
 
    def process_request(self,request,spider):
        proxy = random.choice(self.PROXIES)
        request.meta['proxy'] = proxy

解決

在上面的下載中間件中設(shè)置代理時，需要添加協(xié)議名稱，“http://”或者“https://”。

如下：

class IpProxyDownloadMiddleware(object):
    PROXIES = ['110.52.235.131:9999','110.52.235.249:9999','112.17.38.141:3128']
 
    def process_request(self,request,spider):
        proxy = random.choice(self.PROXIES)
        request.meta['proxy'] = 'http://'+proxy

問題成功解決。

總結(jié)

以上為個人經(jīng)驗，希望能給大家一個參考，也希望大家多多支持腳本之家。

最新評論