i have issues scraping websites, while others work. example, works:
page = requests.get('https://wsj.com/', proxies=proxydict) but doesn't:
page = requests.get('https://www.privateequityinternational.com/', proxies=proxydict) i "max retries" error, though scrape 1 page (and haven't scraped before).
i've tried using header websites won't scrape hasn't worked. there specific header should use? how scrape second website i've shown above (www.privateequityinternational.com)? thank you.
the issue page served on http in browser not https, warning google when try access page using https:
in [1]: import requests ...: page = requests.get('http://www.wsj.com') ...: in [2]: page out[2]: <response [200]> 
Comments
Post a Comment