Chat records between Lin Shengbin and Zhu Xiaozhen

Hey, Xiaozhen, what are you busy with recently?

Zhu Xiaozhen: I am learning Python recently and want to further improve my skills.

Lin Shengbin: Wow, Python is indeed a promising language, and I am also learning it. How did you learn?

Zhu Xiaozhen: I found some Python tutorials on the Internet, and then did some exercises myself.

Lin Shengbin: Have you encountered any problems? I've encountered some difficulties before.

Zhu Xiaozhen: Actually, I have been learning Python crawlers recently. Some websites have strict anti-crawler mechanisms, and I don’t know how to deal with them.

Lin Shengbin: I have also encountered this problem, but I have found some solutions. You can try the following steps:

Step 1: Use Headers to simulate a browser

Some websites determine whether the visitor is a crawler by detecting User-Agent. If so, The crawler program will deny access. We can use Headers to simulate a browser and make the visitor look like a normal user.

```

importrequests

url=''

headers={

'User-Agent ':'Mozilla/5.0(WindowsNT10.0;Win64;x64)AppleWebKit/537.36(KHTML, likeGecko)Chrome/58.0.3029.110Safari/537.3'}

response=requests.get(url, headers= headers)

```

Step 2: Use proxy IP

Some websites determine whether the visitor is a crawler by detecting the IP address. If so, The crawler program will deny access. We can use proxy IP to change the visitor's IP address so that the visitor appears to be a different user.

```

importrequests

url=''

proxies={

''

p>

cookies={

'BAIDUID':'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:FG=1'

}

response=requests.get(url, cookies =cookies)

```

Zhu Xiaozhen: Wow, these methods are really practical. I’ll try them to see if they can solve my problem.

Lin Shengbin: I hope I can help you. If you have any other questions, you can ask me at any time.

Zhu Xiaozhen: OK, thank you for your help!