Automated Pipeline

Data Refresh Pipeline

Monitors authoritative data sources for new releases and flags ENV/SCI variables that may need re-validation. Sources are checked against their expected refresh cycles with automated API connectivity tests.

Pipeline Control

Trigger manual checks or review automated results

ENVAPI
World Bank Open Data

World Bank Group

Check Failed
Refresh CycleQuarterly (90d)
Last CheckedApr 7, 2026, 11:15 AM
Next Check DueIn 78 days

Affected Variables

water stress ratiogroundwater depletionarable land loss
ENVHTTP
FAO AQUASTAT

Food and Agriculture Organization of the United Nations

Fresh
Refresh CycleSemi-Annual (180d)
Last CheckedApr 7, 2026, 11:15 AM
Next Check DueIn 168 days

Affected Variables

groundwater depletionwater stress ratioflood drought frequency
Source ↗
SCIHTTP
USGS Mineral Commodity Summaries

U.S. Geological Survey

Fresh
Refresh CycleAnnual (365d)
Last CheckedApr 7, 2026, 11:15 AM
Next Check DueIn 353 days

Affected Variables

critical mineral hhirare earth dependencysupplier concentration
Source ↗
ENVHTTP
ND-GAIN Country Index

University of Notre Dame

Fresh
Refresh CycleAnnual (365d)
Last CheckedApr 7, 2026, 11:15 AM
Next Check DueIn 353 days

Affected Variables

nd gain vulnerabilityclimate exposure
Source ↗
ENVHTTP
WRI Aqueduct Water Risk Atlas

World Resources Institute

Fresh
Refresh CycleAnnual (365d)
Last CheckedApr 7, 2026, 11:15 AM
Next Check DueIn 353 days

Affected Variables

water stress ratioflood drought frequency
Source ↗
Check History

Recent automated and manual data source checks

SourceCategoryStatusDetailsChecked AtNext Due
WRI Aqueduct Water Risk AtlasENVFreshLast-Modified: Mon, 06 Apr 2026 22:47:33 GMTApr 7, 2026, 11:15 AMApr 7, 2027, 11:15 AM
ND-GAIN Country IndexENVFreshSource accessibleApr 7, 2026, 11:15 AMApr 7, 2027, 11:15 AM
USGS Mineral Commodity SummariesSCIFreshLast-Modified: Mon, 03 Mar 2025 15:53:59 GMTApr 7, 2026, 11:15 AMApr 7, 2027, 11:15 AM
FAO AQUASTATENVFreshLast-Modified: Fri, 16 Jan 2026 09:45:50 GMTApr 7, 2026, 11:15 AMOct 4, 2026, 11:15 AM
World Bank Open DataENVCheck FailedAPI not respondingApr 7, 2026, 11:15 AMJul 6, 2026, 11:15 AM
Pipeline Methodology

The Data Refresh Pipeline monitors five authoritative sources that feed into the ARGOS engine's ENV and SCI sub-indices. Each source has a defined refresh cycle based on its publication frequency:

API-Based Checks

World Bank Open Data is checked via the DataBank API, verifying indicator accessibility and checking for new data vintages. The API returns metadata about available indicators and their latest update timestamps.

HTTP-Based Checks

FAO AQUASTAT, USGS MCS, ND-GAIN, and WRI Aqueduct are checked via HTTP HEAD requests, verifying source accessibility and checking Last-Modified headers for content changes since the last validation.

When a source check indicates potential new data, the affected ENV/SCI variables are flagged for re-validation. The pipeline does not automatically update values — all changes require manual cross-validation through the Audit Trail process to ensure data integrity and provenance tracking.