Web scraping VPS: architecture, etiquette and when to add proxies

A laptop is fine for a one-off scrape. Once a job has to run every night, retry failures and keep months of history, it belongs on a server. A web scraping VPS gives you an always-on machine with a fixed IP, predictable resources and room for a database next to your crawler. This article covers how to lay out that server, how much memory headless browsers really need, how to stay a polite visitor, and when it makes sense to add dedicated proxies instead of simply crawling from the server's own IP.

Start with the rules, not the code

Before choosing hardware, check what you are allowed to collect. Three questions settle most cases:

• Does the site's robots.txt disallow the paths you want? It is a published request from the site owner. Respect it.
• What do the terms of use say? Many sites restrict automated access, and some offer an official API or data export that is both easier and permitted. Use it when it exists.
• Is there personal data involved? Collecting information about individuals brings privacy obligations that depend on where you and they are. This article is not legal advice; if personal data is in scope, get proper advice first.

Everything below assumes you are collecting public data you are allowed to collect, at a rate the site can comfortably handle. No server or proxy setup makes a prohibited scrape acceptable, and none can guarantee you will not be blocked.

A server layout that survives real workloads

Scrapers break in boring ways: a selector changes, a page times out, the process runs out of memory at 3 a.m. A simple, separated layout makes those failures cheap.

Keep fetchers and renderers as separate workers. Plain HTTP fetching is light and fast; a headless browser is heavy. Splitting them means you can run many cheap fetchers and only a few browsers, and route a URL to a browser only when the plain request does not return the data.

Store the raw response alongside parsed fields when you can. When a parser bug appears, you can re-parse yesterday's pages instead of crawling the site again.

Sizing a web scraping VPS: browsers decide the memory

Cloudj servers in the US start at 1 vCPU, 1 GB RAM and 20 GB disk and scale to custom sizes up to 104 vCPU and 512 GB RAM, so you can match the box to the crawler rather than the other way round.

• HTTP-only scrapers use little memory per request. A 1 to 2 GB server handles a scheduler, a queue and a pool of async fetchers for modest volumes.
• Headless browsers are the expensive part. Each Chromium instance with a few open tabs can take several hundred MB, and pages with heavy scripts take more. Plan RAM around the number of concurrent browser contexts, then add headroom for the database.
• CPU type. Shared ("Availability") CPU suits HTTP fetchers that mostly wait on the network. If you render pages continuously, dedicated CPU gives dedicated cores for that steady load.
• Disk. Raw HTML adds up quickly. Compress it, rotate it, or ship it off the server.

Start small, watch the console's CPU, RAM and network graphs during a real run, and resize CPU and RAM or grow the disk when the numbers say so.

Traffic matters too. Monthly US plans include 5 TB of traffic per month; on hourly billing, traffic is charged per GB used. Page-rendering crawls that download images and scripts use much more traffic than HTML-only fetches, so block images and fonts in the browser when you do not need them.

Why a US server

Pick the location by where the target sites are served from and what audience their content is meant for. For US retailers, listings and news, a server in the United States sees the same pages a US visitor sees and keeps each request round trip short. Cloudj has US cloud servers in New York, Los Angeles, Santa Clara, Seattle, Chicago, Dallas, Atlanta and Miami. The US VPS guide helps you choose between them, and the Americas location guide compares other options in the region.

Being a polite crawler

Good etiquette is also good engineering: polite crawlers get blocked less and break less.

1. Set an honest User-Agent that names your project and gives a contact address.
2. Rate-limit per domain, not globally. One request every few seconds per site is a reasonable starting point for small sites; follow any crawl delay the site publishes.
3. Back off on errors. Treat HTTP 429 and 503 as a signal to slow down, and honour Retry-After headers.
4. Cache and use conditional requests. Send If-Modified-Since or ETag headers so unchanged pages cost the site almost nothing.
5. Crawl off-peak for the site's audience when freshness allows.

A minimal polite fetch in Python looks like this:

import time, requests

session = requests.Session()
session.headers["User-Agent"] = "acme-price-study/1.0 (contact: [email protected])"

def fetch(url, delay=3):
    r = session.get(url, timeout=30)
    if r.status_code in (429, 503):
        wait = int(r.headers.get("Retry-After", 60))
        time.sleep(wait)
        return fetch(url, delay)
    time.sleep(delay)
    return r

When to add dedicated proxies

A VPS has one public IP. That is enough for many legitimate jobs, and you should not add proxies by default. Proxies make sense in a few specific situations:

• You need to see a location other than the server's. For example, checking how US and Vietnamese storefronts display prices, or verifying regional ads, requires IPs in those countries.
• Your volume is spread across many sites and you want to spread load across several clean IPs rather than concentrate it on one.
• You need the IP to look like a home connection for regional content checks, where some sites show datacenter IP ranges different content.

Cloudj sells dedicated IPv4 proxies in Vietnam, the United States, Japan, South Korea, Singapore, Hong Kong and the United Kingdom. Each IP is a static ISP IP assigned to one customer only. In the United States and Vietnam there are also static residential ISP proxies: a residential IP that stays the same. All support HTTP, and SOCKS5 where the location supports it. See the US proxy page for options there, and the dedicated proxy guide for how the types differ.

For scrapers, IP whitelisting is often the cleanest setup: add your VPS's IP to the proxy order's whitelist (up to 20 addresses) and your code connects without storing a password. The Python scraping with proxies and Playwright and Puppeteer proxy guides show the code.

Proxies do not change the rules. Follow each site's terms, keep your rate limits, and do not use proxies to get around a block a site has deliberately put in place.

FAQ

Can I scrape from a VPS without proxies?

Yes. For many jobs, a single server IP with polite rate limits is enough. Add proxies only when you need other locations or need to spread load.

How much RAM do I need for Playwright?

Plan for several hundred MB per concurrent browser context, plus memory for your database. A 4 GB server is a comfortable starting point for a handful of parallel browsers.

Is hourly billing a good fit for scraping?

For short, occasional crawls, yes, but delete the server afterwards because stopped hourly servers are still charged. For nightly jobs, a monthly plan with included traffic is easier to budget.

Is web scraping legal?

It depends on the data, the site's terms and your jurisdiction. Respect robots.txt and terms of use, avoid personal data unless you have a lawful basis, and get legal advice for anything unclear.

Set up your crawler

Deploy a small US server, run one real crawl while watching the metrics, and size up from there. Exact prices for servers and proxies are shown in the Cloudj console before you order, and you can compare them on the pricing page.