Methodology

How we discover, verify, structure, and publish human events. Full transparency on every step.

🎯Core Principles

The Event is the Central Object

An article is only a presentation layer around an event. The event itself—structured, sourced, versioned—is the primary data entity.

No Number Without Provenance

Every figure—deaths, injuries, participants, displaced—must trace to one or more source records. We never store a number without its evidence chain.

AI is a Tool, Not a Source

AI-generated statements are never evidence. AI assists with classification, extraction, clustering, and synthesis—but never invents facts or fills missing numbers.

Traceable Claims

Every factual claim links to a source quote, a source URL, and a confidence assessment. You can always verify what we say.

Uncertainty Over False Precision

We distinguish: reported, verified, corroborated, disputed, estimated, unknown. We never turn uncertain information into false precision.

📡Source Acquisition

Layer 1: Broad Discovery (GDELT)

We use the Global Database of Events, Language, and Tone (GDELT) for global, multilingual news discovery across 65+ languages. GDELT provides structured event records, article URLs, and metadata for initial candidate detection.

Why GDELT: Unparalleled global coverage, structured output, free access. Limitations: Noisy, requires heavy filtering, not a verified source itself.

Layer 2: Structured Datasets

UCDP (Uppsala Conflict Data Program): Authoritative conflict event data, battle-related deaths
ReliefWeb / UN OCHA: Humanitarian reports, displacement figures, disaster data
ACLED: Armed Conflict Location & Event Data (when available)
UNHCR / IOM: Official displacement and refugee statistics
WHO: Health emergency and disease outbreak data

Layer 3: RSS & Official Feeds

Configurable source registry (sources.yaml) with global/regional news, government feeds, NGO reports, and UN agencies. Each source has priority, poll interval, and robots.txt compliance.

Principle: Respect robots.txt, rate limits, copyright. Prefer RSS and public APIs. Never scrape indiscriminately. Store metadata and link to original.

⚙️Discovery & Processing Pipeline

Automated Pipeline Stages

1.DISCOVER – Fetch from GDELT, RSS, datasets
2.FETCH METADATA – Title, URL, date, source
3.NORMALIZE URL – Canonicalize, deduplicate
4.DEDUPLICATE – Exact URL, content hash, title similarity
5.EXTRACT CANDIDATE FACTS – LLM-structured extraction (Gemini)
6.CLASSIFY EVENT – Type, location, date, severity
7.CLUSTER RELATED ARTICLES – Multi-source → single event
8.CREATE EVENT CANDIDATE – Structured event with claims
9.VERIFY & SCORE – Cross-source corroboration, importance scoring
10.PUBLISH DECISION – Auto-publish / Auto-publish with warning / Hold
11.GENERATE – Event page, map marker, summary, share card

🔗Event Clustering (Multi-Source → Single Event)

Ten news articles describing one protest become ONE event with multiple sources.

Clustering Signals

✓ Exact URL canonicalization
✓ Title similarity (fuzzy matching)
✓ Named entity overlap (actors, organizations)
✓ Location proximity (within 50km)
✓ Temporal proximity (within 48 hours)
✓ Semantic similarity (embeddings)

Example Cluster

Reuters: "Protests in Paris over pension reform"
AP: "French unions mobilize against retirement age"
Le Monde: "Manifestations à Paris : 15 000 selon la CGT"
Gov statement: "Ministry confirms 7 arrests"
GDELT: Event code 1401, Paris, 2026-10-08
→ ONE EVENT with 5 sources

🤖AI Role & Constraints

AI Performs

✓ Event classification
✓ Entity extraction (actors, locations, dates)
✓ Fact/claim extraction with quotes
✓ Event clustering assistance
✓ Importance & novelty scoring
✓ Source comparison & conflict detection
✓ Summary & headline generation
✓ Daily brief & social card copy

AI Must NOT

✗ Invent facts or casualty figures
✗ Fill missing numbers
✗ Pretend verification
✗ Merge incompatible numbers
✗ Create fake citations
✗ Publish unsourced claims
✗ Generate content without structured output

Structured Output Only

We never ask for free-form prose. We request JSON with strict schema validation (Pydantic). Malformed output is rejected and retried.

{"event_type": "PROTEST", "claims": [{"type": "participants", "min": 10000, "max": 20000, "source_id": "..."}], "confidence_score": 82}

📊Scoring & Publishing Policy

Interestingness Engine (Event Score)

Weighted formula (configurable in scoring.yaml):

score =
  0.25 × importance
+ 0.20 × human_impact
+ 0.15 × novelty
+ 0.15 × corroboration
+ 0.10 × velocity
+ 0.10 × public_interest
+ 0.05 × unusualness
− penalties (duplicate, low confidence, single source, old event)

Three-Tier Publishing

✅ AUTO-PUBLISH

High confidence
Multiple independent sources
Clear location & date
No major contradictions

⚠️ AUTO-PUBLISH WITH WARNING

Sources disagree on numbers
Early/developing reports
Page clearly states: "Reports differ"

🔴 HOLD

Single weak source
Location/date uncertain
Major contradictions
Possible misinformation

💔Human Cost Metrics

We never combine incompatible datasets. Each metric specifies exactly what is counted, by whom, and when.

Supported Metric Types

reported_deaths
reported_injuries
civilian_deaths
combatant_deaths
displaced_people
reported_airstrikes
reported_attacks
reported_missiles

Every Metric Includes

value + unit
source + source date
methodology
confidence level
cumulative vs period-specific
geography scope

🔄Revision & Contradiction Handling

Event Revisions

Events evolve. We store historical claims, never overwrite.

00 → Report: 5 killed (Source A)
00 → Official: 8 killed (Source B)
00 → UN: 7 killed (Source C)

UI shows: "Earlier reports: 5 (A), 8 (B). Latest: 7 (UN)."

Contradiction Engine

Detects conflicting reports and flags them explicitly.

Source A: 23 deaths (UN)

Source B: 21 deaths (Gov)

System: value_min=21, value_max=23, status=conflicting_reports

🛠️Technology Stack

Backend (Python)

httpx – Async HTTP client
feedparser – RSS/Atom parsing
BeautifulSoup – HTML parsing
Pydantic – Data validation
RapidFuzz – String similarity
Google Generative AI – Gemini API
Tavily – Search API (optional)

Frontend (Astro)

Astro – Static site generation
MapLibre GL JS – Interactive maps
Tailwind CSS – Styling
ECharts – Visualizations
TypeScript – Type safety

Deployment (Free Tier)

GitHub Actions – Scheduled ingestion
Cloudflare Pages – Static hosting
Cloudflare D1 – Database (later)
GitHub Secrets – API keys

Data Storage (MVP)

Local JSON – Events, sources, claims
Generated JSON – Map, timeline, stats
Static files – Event pages, briefs
→ Migrate to D1 when JSON becomes cumbersome

⏰Schedule & Resource Budget

Autonomous Schedule

Every 5 min: Best-effort source discovery, processing, and publishing
Every minute: Open news feeds check for newly published data; source ingestion runs on its own schedule
Every 2–4 hrs: Recheck developing stories, update casualties
Every 6 hrs: Rebuild daily summaries, refresh rankings
Daily: Historical aggregation, sitemap, cleanup

Economic Model (Free Tier)

1000 discovered articles
→ 200 candidates (deterministic filter)
→ 50 meaningful events (clustering)
→ 20 verified (LLM extraction)
→ 5–15 published

Gemini: 80% extraction, 15% verification, 5% editorial. Tavily: only high-interest cases.

🚫What We Don't Do (Yet)

No chatbot, no mobile app, no user accounts
No comments, no social network, no subscriptions
No AI-generated images or videos
No full-text news republishing (copyright)
No paid APIs, paid scraping, paid LLMs (unless required later)
No browser extensions, no premium features

Focus: Prove that autonomous event discovery + verification + visualization + SEO + publication creates useful traffic.

Open Source & Transparent

This methodology is a living document. Code, configuration, and data pipelines are open source.

View on GitHub Browse Events