A crawl is a run in which Upwork Insights reads Upwork's public agency directory, category by category, and then each agency's public page, recording one observation per page read.
The crawl starts from the agency directory, the same categories Upwork lists, and reads each category's listing pages in order until the listing ends. Every agency on a listing is then read from its own page, and the fields the page shows to a signed-out visitor are stored as an observation. Agencies no directory category lists, for example ones found through archived URLs or sent in by their owners, are read from a seed list the same way. A refresh pass re-reads every stored agency and adds a snapshot to each, which is how a second observation and a run-rate come about; at most one snapshot is kept per page per day.
The crawl is deliberate rather than fast. Each new category is read completely before the next is added, so a category page is never half a list, and the reading is paced so as not to load Upwork's servers. Upwork's directory stops serving listing pages after the first 10,000 rows of a category, which caps how much of the very largest categories any crawl can see.
A crawl reads only public pages. Nothing comes from a signed-in session, and if a figure is hidden on the page it is hidden in the record. The crawl is also not continuous: a page shows the figures as of its last read, which is why every agency page states that date.
How Upwork Insights uses it
The 28,921 agencies in the current index were read across 89 categories, and pages are re-read on a rolling schedule, claimed pages weekly, with every crawl followed by an index build dated Oct 31, 2026. Scope and cadence are in methodology, section 5.