A practical guide to web scraping proxies which proxy type fits which job, how rotating and sticky sessions differ, how to connect to a rotating residential gateway, and a retry-and-rotate pattern that keeps a scraper collecting reliably.
Proxies for web scraping route your requests through a pool of different IP addresses, so a large collection job spreads across many IPs instead of hammering a site from one. That keeps data collection on publicly available pages steady and raises your success rate. This guide uses standard proxy configuration that works with any provider, with Proxy-Cheap as the reference service since its rotating residential line is built for exactly this workload.
- What web scraping proxies do
- Which proxy type for which job
- Rotating vs sticky sessions
- Connecting to a rotating residential gateway
- Sticky sessions when you need them
- A retry-and-rotate pattern
- Authentication and protocols
- Testing your proxy
- Common use cases
- Buying a proxy for web scraping
- Common issues and fixes
- FAQ
A web scraping proxy sits between your scraper and the target site, so each request leaves from the proxy IP instead of your own. Send a thousand requests through a rotating pool and they arrive from many different IPs, which spreads the load and keeps a large job moving smoothly.
The kind of IP behind the proxy is what decides how well it works. Residential IPs carry the identity of a genuine consumer connection, so they read as ordinary visitors to the sites you collect from. Datacenter IPs are faster and cheaper but clearly server-hosted. Matching the IP type to the target is most of the battle.
There is no single best proxy for web scraping; the right pick depends on the target:
| Proxy type | Speed | Identity | Best for |
|---|---|---|---|
| Datacenter | Fastest | Server-hosted | High-volume reads of open, public pages and documentation |
| Rotating residential | Moderate | Genuine consumer IPs, large pool | Volume collection where each request can use a fresh IP |
| ISP / static residential | Fast | Residential identity, fixed IP | Multi-step, session-bound scraping that needs one steady IP |
| Mobile | Moderate | Mobile carrier identity, highest trust | The most demanding targets and mobile-first sites |
Start on datacenter for open public data where speed matters. Move to rotating residential proxies for volume against sites that expect consumer traffic. Use ISP or static residential when a scrape spans a login or a multi-page session that must hold one IP.
Two session models cover almost every scraping job:
- Rotating. A new exit IP per request, drawn from a large pool. Ideal for distributed crawling and high-volume collection, where spreading requests across many IPs is the point.
- Sticky. The same IP held for a short window (around 30 minutes on rotating residential). Use it when a task spans several requests that must come from one IP, such as paginating through a session.
Reach for rotating by default on large crawls, and switch to sticky only for the multi-step flows that need continuity.
Rotating residential proxies use a single gateway address with username and password. Each request through the gateway returns a fresh exit IP from the pool, so rotation is automatic and there is nothing to cycle yourself. Proxy-Cheap exposes regional gateways, for example proxy-us.proxy-cheap.com:5959 (US) and proxy-eu.proxy-cheap.com:5959 (EU).
cURL:
curl -x http://USERNAME:[email protected]:5959 https://api.ipify.orgPython (requests):
import requests
gateway = "http://USERNAME:[email protected]:5959"
resp = requests.get(
"https://api.ipify.org",
proxies={"http": gateway, "https": gateway},
timeout=30,
)
print("Exit IP:", resp.text)Run the Python snippet a few times and the exit IP changes on each call, which confirms rotation is working. Rotating residential uses HTTP and username/password authentication.
When a scrape needs the same IP across several requests, rotating residential supports a sticky session that holds one exit IP for roughly 30 minutes. You request it through a session identifier issued from your dashboard credentials rather than by changing the gateway. The exact session-ID format is documented in the support knowledge base; for a fixed IP that lasts longer than a session window, a static residential or ISP proxy is the better tool.
Real scrapes hit transient failures: a timeout, a slow exit node, an empty response. Because the rotating gateway hands you a new IP on the next request, the simplest resilient pattern is to retry, and each retry naturally routes through a fresh IP.
import time
import requests
GATEWAY = "http://USERNAME:[email protected]:5959"
PROXIES = {"http": GATEWAY, "https": GATEWAY}
def fetch(url, attempts=4):
for attempt in range(1, attempts + 1):
try:
resp = requests.get(url, proxies=PROXIES, timeout=30)
resp.raise_for_status()
return resp.text
except requests.RequestException as exc:
wait = 2 ** attempt # exponential backoff
print(f"attempt {attempt} failed ({exc}); retrying in {wait}s")
time.sleep(wait)
raise RuntimeError(f"giving up on {url} after {attempts} attempts")
if __name__ == "__main__":
html = fetch("https://example.com")
print(len(html), "bytes")The full version, plus a Node example, is in examples/. For large jobs, run several workers concurrently and let each one use the gateway independently.
- Rotating residential authenticates with username and password over HTTP, which is what the examples above use.
- Static products (static residential, ISP, datacenter, mobile) add IP whitelist authentication and SOCKS5 on top of HTTP, which suits server-side scrapers that would rather not store credentials in code.
Pick username/password for quick local runs, and IP whitelist for fixed-IP servers and CI pipelines.
Confirm the proxy carries traffic and returns a pool IP rather than your own:
# Your real IP, no proxy
curl https://api.ipify.org; echo
# Through the rotating residential gateway
curl -x http://USERNAME:[email protected]:5959 https://api.ipify.org; echoIf the second command prints a different IP each time you run it, rotation is live. https://httpbin.org/ip and https://ifconfig.me/ip work as alternate echo endpoints.
Web scraping proxies support data collection across many workloads:
- E-commerce price and stock monitoring across regional storefronts.
- SERP and SEO research from a consistent market.
- Market and competitor research on publicly available pages.
- Ad verification, checking how campaigns render in a given region.
- Travel fare aggregation and brand protection research.
For the workload landing pages, see the data scraping and price and sales research use-case pages.
When you compare where to buy web scraping proxies, weigh proxy type, rotation and sticky-session options, authentication, country coverage, and the billing model. Match the type to the target rather than defaulting to the most expensive tier.
Proxy-Cheap is a reasonable default for individual developers and small teams:
- Rotating residential proxies on a pay-as-you-go, per-GB model for volume collection
- Static residential, ISP, datacenter, and mobile lines for session-bound and high-throughput jobs
- HTTP and SOCKS5 support on the static lines, with username/password or IP whitelist authentication
- Country coverage across 180+ countries, with 24/7 support and 256-bit SSL
Start with the rotating residential proxies line for volume. For session-bound scrapes use the static residential proxies or ISP proxies pages, for raw throughput the datacenter proxies page, and for sustained volume the unlimited bandwidth proxies page. Provisioning is handled by the REST API.
The exit IP never changes. You may be reusing one keep-alive connection. Open a fresh request per fetch, or confirm you are on the rotating gateway rather than a static IP.
Domain requests fail while IP requests work. DNS is not resolving locally. On the static SOCKS5 lines, use socks5h:// so the proxy resolves hostnames.
Authentication fails. Re-check the username, password, gateway host, and port from the dashboard. Rotating residential uses username/password, not IP whitelist.
Requests time out under load. Lower concurrency, raise the per-request timeout, and add the retry-and-rotate loop above so a slow exit node does not stall the whole job.
Web scraping proxies are intermediary IPs that your scraper routes requests through, so the target site sees the proxy IP instead of yours. Rotating pools spread a large job across many IPs, which keeps collection of publicly available data steady.
It depends on the target. Datacenter proxies are best for fast reads of open public pages, rotating residential proxies for volume against consumer-facing sites, ISP or static residential for session-bound scrapes, and mobile for the most demanding targets.
They serve different jobs. Residential proxies carry a genuine consumer identity and suit sites that expect real-visitor traffic, while datacenter proxies are faster and cheaper for open public data. Many scrapers use datacenter first and move to residential where a job needs it.
On a rotating gateway, each request returns a new exit IP from the pool automatically, so you do not manage IPs yourself. For steps that must share one IP, a sticky session holds a single IP for a short window.
The static lines (static residential, ISP, datacenter, mobile) support HTTP and SOCKS5. Rotating residential uses HTTP. Use socks5h on the SOCKS5 lines when you want the proxy to resolve DNS.
With a rotating residential gateway you get access to a large pool through one endpoint, so a single plan covers most jobs. Scale by bandwidth and concurrency rather than by buying many individual IPs, unless you need fixed IPs for session-bound work.
- Proxy-Cheap rotating residential proxies
- Proxy-Cheap datacenter proxies
- Proxy-Cheap data scraping use cases
- Proxy-Cheap support knowledge base
- Proxy-Cheap REST API documentation
MIT
This is a personal documentation repo, not an official Proxy-Cheap project. Code and setup use standard proxy configuration and work with any provider. Verify current pricing, gateway addresses, and country coverage on the live product page before relying on them.