How to Set Up Rotating Proxies for Python Web Scraping

Why Rotate Proxies in Python?

If you're using Python's requests library to scrape data, sending hundreds of requests from your local machine will get your IP banned within minutes. To build a resilient scraper, you need to rotate your proxies, ensuring each request (or session) appears to come from a different user.

Step 1: Get Your Proxy Credentials

First, you need a proxy network that supports automatic rotation. With Proxelytics, this is incredibly easy. Once you create an account, navigate to your dashboard and generate your proxy credentials (username, password, endpoint, and port).

Our gateway automatically rotates your IP on our backend, so you only need to connect to a single endpoint.

Step 2: Basic Implementation with Requests

Here is a simple example of how to route a request through the Proxelytics rotating gateway.

import requests

# Replace with your actual Proxelytics credentials
proxy_host = "gate.proxelytics.com"
proxy_port = "8000"
proxy_user = "your_username"
proxy_pass = "your_password"

proxy_url = f"http://{proxy_user}:{proxy_pass}@{proxy_host}:{proxy_port}"

proxies = {
    "http": proxy_url,
    "https": proxy_url
}

url = "https://api.ipify.org?format=json"

try:
    response = requests.get(url, proxies=proxies, timeout=10)
    print("Success! Your IP is:", response.json()['ip'])
except Exception as e:
    print("Error:", e)

Every time you run this script, Proxelytics will assign a new residential IP to your request, and the API will return a different IP address!

Step 3: Maintaining Sticky Sessions

Sometimes, you need to keep the same IP address for a series of requests (e.g., logging in, navigating to a profile, and downloading data). We support "sticky sessions" by allowing you to append a session ID to your username.

import requests
import random

session_id = random.randint(1, 100000)
# Append session-id to the username
proxy_user = f"your_username-session-{session_id}"
proxy_url = f"http://{proxy_user}:your_password@gate.proxelytics.com:8000"

proxies = {"http": proxy_url, "https": proxy_url}

# Multiple requests using the same session
s = requests.Session()
s.proxies.update(proxies)

res1 = s.get("https://api.ipify.org")
print("Request 1 IP:", res1.text)

res2 = s.get("https://api.ipify.org")
print("Request 2 IP:", res2.text) 
# Both requests will output the SAME IP address.

Best Practices

Always combine proxy rotation with proper User-Agent rotation and randomized delays. Check our full API Docs for advanced integration examples with Scrapy, Playwright, and Selenium.

Ready to write your scraper? Grab a residential proxy plan and start extracting data safely today.