Apply the Scrapy framework and configure dynamic IP processing to handle anti-crawling.
# settings Configure middlewareDOWNLOADER_MIDDLEWARES = { '': 543, # '': 544, # '': 545, '': 546, '': 547 } # settings Configure available dynamic IPPROXIES = [ "http://101.231.104.82:80", "http://39.137.69.6:8080", "http://39.137.69.10:8080", "http://39.137.69.7:80", "http://39.137.77.66:8080", "http://117.191.11.102:80", "http://117.191.11.113:8080", "http://117.191.11.113:80", "http://120.210.219.103:8080", "http://120.210.219.104:80", "http://120.210.219.102:80", "http://119.41.236.180:8010", "http://117.191.11.80:8080" ]
# middlewares configuration middlewareimport random class ProxyMiddleware(object): def process_request(self, request, spider): ip = (('PROXIES')) print('Test IP:', ip) ['proxy'] = ip class CheckProxyMiddleware(object): def process_response(self, request, response, spider): print('Proxy IP:', ['proxy']) return response
This is the end of this article about the implementation of Scrapy configuration dynamic proxy IP. For more related Scrapy dynamic proxy IP content, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!