How to deal with python crawler ip blocking

1, slow down the crawling speed and reduce the pressure on the target website. But this will reduce the amount of grabbing per unit time class.

The second method is to break through the anti-crawling mechanism and continue high-frequency crawling by setting IP and other means. The anti-crawling mechanism of the website will check the visited IP address. In order to prevent IP from being blocked, HTTP can be used to switch between different IP crawls. Simply put, using proxy IP is to let the proxy server get the web page content for us and forward it back to our computer. In the choice of high-secret ip, IPIDEA provides high-secret and stable IP, pays more attention to the protection of users' privacy and ensures users' information security.

2. In this way, the target website doesn't know that we use the proxy, nor does it know our real IP address.

3. Establish an IP pool, as large as possible, and rotate different IPS evenly.

If you need to capture a large amount of data, it is recommended that you use HTTP proxy IP to quickly replace IP before or after being blocked. Here is a trick to use: recycle, replace an IP before it is blocked, and change it back later. In this way, a large number of accesses can be made with relatively few IP. The above is an introduction to the limited IP address of the crawler.