Journalists who cannot code had an LLM write their scrapers, and published the prompts

What happened

Kuek Ser Kuang Keng and Federico Acosta Rainis, of the Pulitzer Center’s data team, published the method their team uses to build web scrapers with a large language model, with the prompts they actually type:

Building scrapers used to require solid coding skills. In our recent investigations, however, Large Language Models (LLMs) like ChatGPT, Google Gemini, or Claude helped us build scrapers for complex databases much faster and without advanced coding skills.

The Pulitzer Center published it on 13 October 2025; the Global Investigative Journalism Network republished it on 31 October. It is free to read.

Two targets are documented. For São Paulo’s public procurement database the team chained three scrapers, one for search results, one for each tender page, one for each evento page, and filtered on cação, pescado and peixe to find shark-meat tenders. They also scraped The Metals Company’s press-release archive. The pipeline reached roughly 75,000 tenders and more than 350,000 document pages; keyword filtering narrowed that to about a thousand candidates, of which people confirmed 701 as shark-meat tenders, reading them one at a time. The machine wrote the collection code. It did not do the verification.

The workflow, for an office of five with no engineers

You need someone patient who can copy, paste and read an error message.

0. Try the cheap route first. The guide starts with no-code browser extensions: Instant Data Scraper, Table Capture, Web Scraper, Data Miner. Before even that, ask the body holding the data for it, or file a request under the right-to-information rules that apply.
1. Diagnose the page. Right-click, “View Page Source”, search the code for a value you can see on the screen. Found: the page is static and easy. Not found: dynamic and harder. Or paste the address into the assistant and ask which it is.
2. For a static page, give the assistant six things in one message: the address; the pasted page source; the exact columns you want; the output format (a CSV file); how the site paginates; and where you will run the script, your own laptop or a free notebook in the browser. Then ask whether it needs anything else before it writes the script.
3. Set up once. Ask for setup instructions for your own operating system, and paste every error message back into the same conversation so it keeps the context.
4. For a dynamic page, which is most government portals, open the browser inspector and use “Copy element” on each dropdown, button and results table. Paste those raw snippets into the prompt beside plain-language steps: select this, click that, wait for the table, read the rows. Two instructions materially improve the result: tell it to print a progress message at each step, and tell it to click buttons using JavaScript.
5. Test small. “Scrape only the first two pages for testing.” Read the output by hand before the full run.
6. Chain, do not combine. Search results, then detail page, then sub-document: three small scrapers feeding each other lists of addresses. The guide says this beats one large script for beginners.
7. If you are blocked, read the error, “Forbidden”, “Too Many Requests”, then escalate slowly: random pauses between requests, a realistic browser identification header and window size, and developing with the browser visible before running it hidden.
8. Optional: have the assistant write the configuration file that runs your scraper in the cloud on a schedule, for free, with your own computer switched off.

What goes wrong

The guide’s own methodology page says LLMs remain unreliable at complex, multistep tasks, and the team caught an LLM inventing websites that do not exist. A scraper that silently drops rows produces a wrong story that looks exactly like a right one, and the guide offers no method for catching that. Count the rows the site says it has, count the rows you collected, and explain the difference before you write a word.

The guide puts legality out of scope and tells you to seek legal advice. Read that twice before you copy step 7. Faking a local address with a VPN, renting residential proxies, forging browser headers and imitating human behaviour, aimed at a government portal in a jurisdiction with a broad and vaguely enforced computer-access law, is where a terms-of-service question becomes a criminal one, at personal risk to whoever runs it. Ask a lawyer you trust first.

The tools named

ChatGPT, Google Gemini and Claude. The scripts are Python, using Requests and Beautifulsoup4 for static pages and Selenium or Playwright for dynamic ones. The guide also names Visual Studio Code, Google Colab, GitHub Actions and Chrome DevTools, the extensions above, and in the blocking section Proton VPN, NordVPN and Oxylabs. A public page’s source is public. Your research notes, target lists and sources are not.