← Open the map · Neighborhood Intel · System notes · Oct 2026
Collectors fetch public data on a schedule and write it to a local file store. When someone clicks the map, the report is built only from that store, so a lookup never waits on a slow portal. This page walks through each step.
There are two halves, and the dashed line in the middle is the most important idea in the system. Everything to its left runs on a timer and talks to the outside world. Everything to its right runs when a user asks a question, and only reads files on disk.
Each data source is a small Python class under ni/sources/. It has two methods that never run at the same time. collect() runs on the schedule and writes files. metrics(loc) runs for each lookup and reads them back. The store between them is the only thing they share.
Cities switch adapters on in cities/<slug>.toml. National adapters (OSM, CPCB, UDISE+, CGWB) need only config to work in a new city. State adapters cover one state's portals.
@register. build_sources(city) turns each [[sources]] entry in the city's TOML into an instance.Every source declares how often its data actually changes, and cron runs one cadence at a time. Air quality is a live feed, so it runs hourly. School and groundwater data change a few times a year, so they run monthly. python -m ni cron prints these lines; they are not installed automatically.
# one cron line per cadence; each runs every city's sources on that cadence 5 * * * * cd neighborhood-intel && python -m ni collect --all --refresh hourly >> data/collect.log 2>&1 20 3 * * * cd neighborhood-intel && python -m ni collect --all --refresh daily >> data/collect.log 2>&1 # … weekly, monthly, yearly follow the same shape
The weekly OSM job downloads the Geofabrik extract for the region (about 900 MB across both zones), skips the download if the local copy is under 20 hours old, and streams it through osmium to cut out about 20 layers: metro stations, hospitals, parks, landfills, runways and so on.
The store is plain files: one folder per city, one folder per source, one JSON file per dataset. There is no database. A collector writes in one of two ways, depending on whether the source keeps its own history.
tpddl_outages/log.jsonl is de-duplicated by district, start time and area.data/ ├── store/ # 72 MB · what reports read │ ├── bengaluru/ │ │ ├── osm/ hospital.json metro_station.json … manifest.json │ │ ├── cpcb_air/ stations.json snapshots.jsonl manifest.json │ │ └── rera_karnataka/ projects.json manifest.json │ └── delhi/ │ └── delhi_flood/ stages.json floodplain.json … ├── cache/ # 888 MB · raw downloads, kept so re-parsing is cheap │ └── osm/ asia_india_southern-zone.osm.pbf … └── collect.log # one JSON line per source per run cities/ ├── bengaluru.toml # sources, declared gaps, sample points └── bengaluru/infra_registry.json # hand-curated, checked into git
Two sources are too large to fetch in one run without hammering the portal. They keep a list of what is already done in the store and take the next slice each run. The areas we use for sample points go first.
A click on the map sends one request. The engine works out which city the point is in from the city bounding boxes, asks every enabled adapter for its metrics, adds the declared gaps, and returns one JSON report. Nothing in this path touches the network.
errors such as "not collected yet". The search box suggests RERA-registered projects as you type, by project or builder name, from our own store (/api/suggest), so search makes no outside calls.Spatial queries are fast because each layer is projected once from latitude and longitude into metres around the city centre, then kept as a numpy array of shapely geometries. One lookup is a vectorised distance or containment test over that array.
The API keeps parsed layers in memory, so a new collection has to replace them without a restart. The cache key is the file's path plus its last-modified time. When cron rewrites a file, its mtime changes and the next request misses the cache and loads the new data.
as_of, the date of the observation itself, which is separate from when we fetched it. A five-year-old school census fetched yesterday still shows its census year.Every number in a report carries a reliability verdict alongside its value, source and date. The verdict is set by the adapter, based on audits of how well that source holds up at that place.
Measured at the point, or a place-level fact we've verified against the ground. Listed under verified_metrics in the city's TOML.
Shown with a sentence saying what the number can't tell you.
Declared as [[gaps]], so the report says so plainly instead of guessing.
Each metric also says how local it really is, so a city-wide average isn't mistaken for a fact about one street.
{
"key": "metro_nearest",
"label": "Nearest metro station",
"value": 640,
"unit": "m",
"source": "OpenStreetMap",
"source_url": "https://download.geofabrik.de/…",
"granularity": "point",
"as_of": "2026-10-16",
"verdict": "show",
"caveat": "",
"topic": "connectivity",
"points": [{ "lat", "lon", "name", "distance_m" }]
}
null for a gapExample values are illustrative.
Government portals fail often. The pipeline is built so that one failure makes one source's data older and leaves everything else alone.
collect() runs inside its own try block. A failure is logged with status error and the next source runs. The previous file stays in the store, so users keep seeing the last good data with its date.
Files download to *.part and are renamed only when complete. A half-finished file never replaces a good one.
The report still returns. That adapter's metrics are missing and the errors list says "not collected yet".
Metrics for areas not yet crawled stay hidden until the crawl reaches them.
Every request identifies itself as neighborhood-intel/0.2 (non-commercial research). Large crawls are sliced across runs. Sources behind captchas, tokens or personal data (KSP FIR search, BMRCL API) are deliberately not automated.