LLM Training Data Collection with Carrier IPs

Teams building or fine-tuning language models need text, and a lot of it comes from the web. That has made data collection one of the most scrutinized kinds of automation: publishers have added opt-out rules, sites have blocked AI crawlers outright, and courts are hearing copyright cases. A mobile proxy is a small part of this picture, and it is worth being clear about what that part is. This page explains where a dedicated NeonProxies carrier line genuinely helps a data pipeline, where it is the wrong tool, and the practices that keep collection defensible when someone asks how the data was gathered.

The honest scale question

Pretraining corpora are built from billions of pages, usually starting from open archives such as Common Crawl, licensed datasets and publisher agreements, then filtered hard. A single mobile line running at 20 to 45 Mbps on 4G, or 50 Mbps and up on 5G, is not the engine for that, and routing a mass crawl through carrier lines would be both slow and inappropriate. If you need web scale, start from the open archives and licensed sources.

Where lines do fit is targeted, smaller collection that needs a real US consumer vantage point: evaluation sets built from how pages actually render for mobile users in specific metros, regional variations of public content, localized product and service descriptions, and checks that a dataset's source pages still say what the dataset claims. Unlimited data on every line means a long-running collector of that kind never stops at a quota.

Consent and opt-out signals come first

A responsible collector checks permission before it checks the network. Read robots.txt and honor it, including the user-agent groups publishers have added for AI training crawlers. Respect page-level signals such as noai or noimageai meta tags where they are present, and site terms that prohibit use for machine learning. Identify your crawler with a descriptive user agent and a contact URL so publishers can reach you. Keep a record of what was collected, from where, when and under which rules, because you may need to show it later.

None of that changes when traffic runs through a proxy. A proxy that is used to hide a crawler from a publisher who has opted out is a proxy being used to ignore consent, and our terms do not allow it. The right use is the opposite: a transparent collector that happens to need a real mobile vantage point.

Pipeline design for the cases that fit

Give each collector worker its own line and keep it sticky for a batch, so each site sees one steady, identifiable client. Rate-limit per domain rather than per worker, so two workers never double the load on one publisher. Cache raw responses with their headers and timestamps, and deduplicate before you fetch again. When a site returns 429, 403 or a challenge page, record it and remove that domain from the queue for the run; do not rotate the line to try again.

For rendering-dependent sources, use a real browser with a mobile viewport and the metro's time zone, capture both the HTML and the visible text, and store the exit IP and metro in the record. That metadata is what makes a regional evaluation set trustworthy.

Quality and licensing checks

Filter personal data out before storage wherever you can, since public pages still contain names, emails and phone numbers that have no place in a training set. Track the license of every source in the manifest and exclude anything whose terms you cannot confirm. Review samples by hand; small targeted datasets are exactly where a few bad sources do disproportionate damage. Keep the collection code, the manifest and the filtering rules versioned together, so anyone auditing the dataset later can reproduce which pages were fetched, from which vantage point, and why each one was kept or removed.

Setting up a LLM training data collection proxy on NeonProxies

1. Decide which part of the dataset needs a real US mobile vantage point; source the rest from archives and licenses.
2. Build a robots.txt and opt-out checker that runs before every fetch.
3. Set a descriptive crawler user agent with a contact URL.
4. Assign one line per worker, sticky per batch, with per-domain rate limits.
5. Store raw responses with exit IP, metro, timestamp and license in a manifest.
6. Drop domains that answer with 403, 429 or challenges instead of retrying.

LLM training data collection proxy questions