Proxy Architecture for Automation at Scale

One browser and one proxy is a configuration. Fifty browsers across a dozen metros with rotation, budgets, and failure handling is an architecture, and teams that never draw that architecture end up rediscovering it through outages. The difference between the two is not cleverness, it is a handful of decisions made explicitly instead of by accident: how identities map to workers, who owns rotation, where proxy configuration lives, and what happens when a link goes bad. This article lays out a reference shape for automation fleets on mobile proxies that has the virtue of being boring, in the way production systems should be.

Start from the identity model, not the worker count

The foundational question is not how many workers you need, it is how many concurrent identities the workload requires. An identity is one proxy, one browser profile, and the state that accumulates between them. Jobs that are stateless can share identities across time; jobs that carry sessions, logins to your own accounts, or multi-step flows need an identity held exclusively for their duration.

Write the mapping down as a rule the scheduler enforces: every job declares whether it needs an exclusive identity or can borrow from a shared pool, and no job ever touches a proxy outside its assignment. Most fleet incidents trace back to two jobs silently sharing an identity that one of them assumed was exclusive, and the fix is making the assumption impossible rather than documented.

Centralize proxy configuration behind one interface

Hard-coded proxy addresses scattered through job configs is the architecture equivalent of unlabeled wires. Put a single configuration service, even a well-managed table, between your fleet and your workers: workers ask it for an identity lease, receive proxy endpoint, credentials, metro, and rotation link, and return the lease when done.

This one indirection buys every operational capability you will later want: swapping a failed proxy without touching worker code, moving a workload between metros by changing a row, draining a proxy for maintenance by refusing new leases, and auditing exactly which job held which identity at any timestamp. The day a target site reports odd behavior, that audit trail is the difference between a five-minute answer and an afternoon of grepping.

Make rotation an orchestrated event, not a side effect

At scale, rotation needs an owner, and the owner should be the orchestration layer, not individual jobs improvising. The lease model gives you the natural hook: rotation happens when a lease is returned, performed by the lease manager, which triggers the rotation link, polls until the IP actually changes, and only then marks the identity available again.

Centralizing this yields properties scattered rotation cannot: no worker ever receives a proxy mid-switchover, rotation frequency per identity is visible in one place, and unlimited rotation gets used deliberately at session boundaries instead of reflexively on errors. Keep timer-based rotation as a background safety net on identities meant for stateless work, with intervals long enough that no leased job gets interrupted.

Budget data per identity, enforce it per job

Each proxy carries 15 GB of data per day, and a fleet without metering discovers its budget the unpleasant way. Meter bytes at the worker, attribute them to the job and identity, and roll them up hourly. Full browser loads are expensive; the metering is what tells you whether the expense is content or waste.

The architectural wins that follow from metering:

• Per-job byte quotas that stop a runaway crawler before it drains an identity's whole day
• Resource-blocking policies, images and fonts off by default, justified by measured savings
• Placement decisions informed by cost per observation, not intuition
• Early warning when a target site redesign suddenly doubles page weight
• Fair sharing when several teams draw from one fleet

Design for partial failure from day one

Mobile infrastructure is real radio hardware on real carrier networks, and individual links will have bad hours: a congested cell, a slow switchover, a modem that needs a reset. Architecture for scale means one identity's bad hour never becomes the fleet's bad hour.

Concretely: health-check identities with a lightweight probe through the proxy, quarantine ones that fail rather than retrying jobs into them, and size the pool with headroom so quarantine does not stall the queue. Route retries to a different identity than the one that failed, with backoff, so a struggling link is not hammered while it recovers. And keep failure metrics per metro as well as per identity, because a pattern across one city usually means local network conditions, which is information, not noise.

Spread across metros deliberately

Geography should be an explicit dimension of the architecture. With identities available across eight US metros, New York, Los Angeles, Chicago, Houston, Phoenix, Miami, North Carolina, and Boston, the scheduler can treat location as a job requirement the same way it treats exclusivity: an ad-verification job declares it needs Miami, a QA suite fans the same test across all eight, a monitoring workload spreads evenly for resilience.

Free location moves make the map reconfigurable, so capacity follows the work rather than the work bending to fixed capacity. One practical note for planning: Boston operates on 4G LTE, so if a particular job genuinely needs 5G-class bandwidth, schedule it to a 5G-capable metro and let Boston carry the ordinary page-load work, which LTE speeds of 20 to 45 Mbps handle comfortably.

Frequently asked