Scrape the next batch

Takes the first 5 sources that are due, opens them one at a time, and pulls every article, listing, heading and link off each page. Nothing is filtered at this stage — everything found is stored, then handed to the extractor.

Driven from your browser so you can watch each source. One PHP request, no live log — exactly what cron does.

Batch queue

Loading…

Batch size

#	Source	Last scraped	Progress	Blocks	Kept
Loading…

Idle

Live log

Exactly what the scraper is doing, source by source.

Ready. Press "Scrape this batch" to begin.

This run

Nothing run yet.

All updates

Opportunities kept

the only number that ends up in your inbox

0 sources 0 blocks 0s

read 0 unchanged 0 failed 0

Opportunities foundnothing yet

Nothing yet.

Why content was droppednothing yet

After a run, this shows why most of the page was discarded — useful for tuning keywords on the Settings page.

Dropped, source by sourcenothing yet

After a run, every source is listed with what it found, what it kept, and why the rest went — so a source that is quietly discarding everything cannot hide behind the totals.

Scrape one source now

The queue only holds what is due. This runs any source immediately — useful straight after changing how it is read or filtered.

Find a source Searches the whole list, however long it is. Read it even if nothing has changed Off means an unchanged page is skipped, exactly as a normal run would skip it.

Crawl settings

How the scraper works through your source list. These apply to every run, including the automatic ones below.

Batch size Sources per batch. 5 is the default and is kind to shared hosting. Re-check after (hours) A source becomes due again once this much time has passed. Which source goes first Straight down the list walks by position — 1, 2, 3 … last, then back to 1 — remembering where it stopped, so every source gets its turn in a fixed rotation and you can say what the next run will take. By priority takes anything never scraped first, then the highest priority, then whatever has waited longest; fairer on a mixed list, but the order is not predictable. Either way a source is skipped while it is still inside the re-check window above. Disable after N failures 0 keeps trying forever. Follow item links Depth 1 opens up to 6 links per source for a fuller description.

User agent Identify yourself honestly. Some university sites block unknown agents. Reader proxy for blocked sites Tried only once a site has actually served a bot-protection challenge — an ordinary fetch never touches it. {url} is replaced with the page being opened. Leave it empty to have such pages simply report as blocked.

Reader proxy headers One per line, “Name: value”. Without a key r.jina.ai allows about 20 requests a minute for the whole server; a free jina.ai key added as Authorization: Bearer jina_… lifts that to 500. Reader proxy timeout (seconds) It opens the page in a browser and waits for the challenge to run, so it needs longer than an ordinary fetch. 60 is a sensible start; the 25 seconds used elsewhere is too short for this.

Respect robots.txt Strongly recommended. Skips any page the site asks crawlers not to fetch. Extract straight after scraping Off means the Extract page becomes a manual second step. Keep raw blocks for (days) Raw blocks let you re-extract without re-downloading. 0 keeps them forever.

Automatic runs

Cron does this same work on a schedule, with no browser open.

What this run did

…

What this says

Source	Found	Kept	Dropped	Why the rest went

Opportunities kept in this run

Reasons across every source

Every block, kept or dropped, stays on the Extract page for 14 days — open a source there to read the exact items behind these counts.