# DataPulse MY

<!-- BEGIN catalog-summary -->
> DataPulse MY publishes a machine-readable manifest of 389 official datasets.

Each manifest entry retains its human-readable `steward` and stable `custodian` publisher ID. Health anomaly fields explain freshness-delta outliers without adding statuses.
<!-- END catalog-summary -->

For focused human-readable NPRA coverage, use the [NPRA engine page](https://www.data-pulse.my/npra.html).

## Model Context Protocol (MCP)

DataPulse MY exposes a **read-only MCP server** so AI agents (Claude Desktop, Cursor, Cline, etc.) can query the 389-dataset catalogue natively.

- **MCP endpoint:** https://mcp.data-pulse.my/mcp
- **Transport:** Streamable HTTP (POST-only, single endpoint)
- **Auth:** none required (all advertised tools are read-only)
- **Source:** reads the same `datapulse.json` + `health/latest.json` + `health/trends.json` + `health/drift.json` + `health/reconciliation.json` served from this repository; no separate API

### Agent discovery endpoints

For AI agents that look up canonical discovery paths before fetching:

<!-- BEGIN public-discovery -->
- [LLM index](https://www.data-pulse.my/llms.txt)
- [Agent manifest](https://www.data-pulse.my/agent.json)
- [MCP advertisement](https://www.data-pulse.my/mcp.json)
- [Sitemap](https://www.data-pulse.my/sitemap.xml)
- [MCP endpoint](https://mcp.data-pulse.my/mcp)
<!-- END public-discovery -->

> Note on path: we use the root (`/llms.txt`, `/agent.json`, `/mcp.json`)
> rather than `/.well-known/...` because (a) the llmstxt.org spec and the
> proposed MCP-discovery convention both use the root, (b) RFC 8615 reserves
> `/.well-known/` for IETF-mandated endpoints (security.txt, etc.), and
> (c) GitHub Pages' default deploy pipeline strips dotfile-prefixed
> paths from the artifact, so `/.well-known/` would silently not deploy
> without a workflow change.

<!-- BEGIN mcp-tools -->
### Tools

| Tool | Use when |
|---|---|
| `search_datasets(query, licence?, source?, limit?)` | Search DataPulse MY's 389 Malaysian public datasets by natural-language query. Filter by licence (e.g. 'CC BY 4.0', 'Open Government Licence (Malaysia)') or source ('OpenDOSM', 'data.gov.my', 'MET Malaysia', etc.). Returns ranked matches: id, title, source, licence, status, score. Use when an agent needs to find datasets covering a topic, by an agency, or under a specific licence. |
| `get_dataset(dataset_id)` | Return full detail for one dataset id, including its latest health status and last-verified timestamp, content_freshness_date, and freshness_signal_source (last_modified, content_parse, or none). Use to fetch the provenance/citation metadata for a dataset found via search_datasets and distinguish unknown-freshness from proven stale data. |
| `find_stale(max_age_hours?)` | Return datasets whose status is aging, stale, or degraded, plus datasets missing from the latest health snapshot. Use when an agent needs to know which data has a freshness or schema-validity risk. |
| `find_anomalies(limit?, mode?, min_reliability?)` | Return datasets flagged by the latest published anomaly detection (anomalies), ranked by how far the observed update interval exceeds its threshold. Optionally require a minimum publish-reliability grade; includes pipeline-computed anomaly and reliability evidence so agents do not recompute it. |
| `find_deteriorating(limit?, min_anomaly_rate?)` | Return datasets whose published freshness trend is deteriorating, ranked by staleness slope. Optionally require a minimum historical anomaly rate; includes pipeline-computed trend and reliability evidence so agents do not recompute it. |
| `find_recovering(limit?)` | Return datasets whose published freshness trend is recovering, with the fastest staleness reductions first. Includes pipeline-computed trend and publish-reliability evidence. |
| `find_unreliable(limit?, at_or_below_grade?)` | Return datasets whose evaluated publish-reliability grade is at or below a threshold (the unreliable ones), with the worst grades and lowest on-time percentages first. Reliability measures timeliness of successful freshness observations, not uptime; sample days are included so agents can judge evidence depth. |
| `find_schema_drift(limit?, min_change_count?)` | Return datasets with published structural or record-count drift evidence, ranked with structural changes first. Optionally require a minimum number of structural transitions; includes pipeline-computed evidence so agents do not infer drift from freshness alone. |
| `check_reconciliation(dataset_name)` | Return the published cross-source reconciliation group for a dataset name or id, including per-member counts, dates, statuses, tolerances, and contextual deltas. A discrepancy requires human review and does not prove either source is wrong. |
| `get_provenance(dataset_ids[])` | Return citation-ready provenance metadata for the listed dataset ids, plus compact pipeline-published evidence receipts: row probe time, HTTP status, request URL, access dependency, freshness source, content date, record count, shape fingerprint, anomaly flag, and status. Use when an agent must cite data and show the evidence behind the trust claim without recomputing it. |
| `get_evidence(dataset_id)` | Return the complete pipeline-published evidence receipt for one dataset id, including probe time, transport, access dependency, freshness, record-count, shape, tolerance, status, and anomaly fields. Use for a deep audit, e.g. get_evidence('fuelprice'); values are presented without MCP-side recomputation. |
| `verify_evidence(dataset_id)` | Perform a rate-limited live streamed GET for one direct-access dataset and compare transport receipts with the latest published evidence, e.g. verify_evidence('fuelprice'). Content dates, row counts, and shape fingerprints remain pipeline-only and are explicitly reported as unverified; results are ephemeral and never update health artifacts. Returns a dict with transport receipt fields and a `verdict` for downstream trust checks without re-fetching. |
| `trust_verdict(dataset_id)` | Return published attestation facts, the unsigned methodology-versioned trust score, numeric components, and component_availability reasons, plus existing health/trend/drift/reconciliation evidence for one canonical dataset id, e.g. 'fuelprice'. This tool does not re-probe or verify the signature; call verify_attestation separately. |
| `verify_attestation(reference, replay_chain?)` | Verify a published Ed25519 probe attestation by canonical dataset id or safe relative digest reference, e.g. 'fuelprice' or 'attestations/2026-08-15/fuelprice.json'. L1 checks signature/key validity; optional L2 replays daily heads to a Git-tag anchor; L3 is provided by verify_evidence. Returns `levels.L1.signature_valid` and `levels.L2.satisfied` for signature and replay status. |
| `find_by_licence(licence)` | Return all datasets with the given licence, summarised. Use to enumerate what's available under a specific licence for compliance/reuse scoping. |
| `usage_summary(buyer_id, since, until)` | Aggregate one buyer's audit-ledger usage for an inclusive ISO date range, e.g. 2026-08-01 to 2026-08-07. Returns `total_calls`, `by_tool`, `by_dataset`, `trust_distribution` (per-status counts of cited datasets) for the inclusive range. |
<!-- END mcp-tools -->

### Resources (read-only context)

- `datapulse://index` — lightweight list of every manifest dataset with namespace (read first)
- `datapulse://anomalies` — anomaly summary with pipeline-computed evidence
- `datapulse://trends` — published freshness trends and publish reliability
- `datapulse://reliability` — publish-reliability grade counts (timeliness, not uptime).
- `datapulse://drift` — published structural and record-count drift evidence
- `datapulse://reconciliation` — published cross-source reconciliation evidence; differences are not proof either source is wrong
- `datapulse://licences` — licence summary (154 CC BY 4.0 + 12 OGL)
- `datapulse://{dataset_id}` — full manifest entry for one dataset (on-demand)

### Connect from Claude Desktop

```json
{
  "mcpServers": {
    "datapulse-my": {
      "transport": "streamable-http",
      "url": "https://mcp.data-pulse.my/mcp"
    }
  }
}
```

### GTFS-aware discovery

The `transport` namespace contains 16 static schedule ZIPs and 14 realtime
vehicle-position protobuf feeds. Use `search_datasets("ktmb")` or another operator
or place name to discover feeds, then `get_dataset(dataset_id)` for static calendar
freshness and route/stop/trip counts or realtime vehicle counts and feed timestamps.
The `datapulse://index` resource exposes each dataset's namespace for lightweight
transport filtering.

The public endpoint is live at `https://mcp.data-pulse.my/mcp` and serves all 16 read-only tools over the 389-dataset catalogue. `trust_verdict` joins published signed facts and unsigned score rows; `verify_attestation` performs L1 signature/key checks and optional L2 tag-anchor replay. Deployment and service details for `datapulse-mcp` are in [`mcp-deploy.md`](https://www.data-pulse.my/mcp-deploy.md).

## Datasets

<!-- BEGIN featured-datasets -->
- [Malaysian Fuel Prices](https://www.data-pulse.my/data/fuelprice.md): Open Government Licence (Malaysia); weekly.
- [ePerolehan Tender Notices (DIIKLANKAN)](https://www.data-pulse.my/data/eperolehan-diklankan.md): Open Government Licence (Malaysia); hourly.
- [PriceCatcher (Grocery Prices)](https://www.data-pulse.my/data/pricecatcher.md): Open Government Licence (Malaysia); monthly.
- [BNM Daily Exchange Rates (0900)](https://www.data-pulse.my/data/exchangerates_daily_0900.md): Open Government Licence (Malaysia); daily (weekdays, 0900 MYT).
- [BNM Daily Exchange Rates (1130)](https://www.data-pulse.my/data/exchangerates_daily_1130.md): Open Government Licence (Malaysia); daily (weekdays, 1130 MYT).
- [BNM Daily Exchange Rates (1200)](https://www.data-pulse.my/data/exchangerates_daily_1200.md): Open Government Licence (Malaysia); daily (weekdays, 1200 MYT).
- [BNM Daily Exchange Rates (1700)](https://www.data-pulse.my/data/exchangerates_daily_1700.md): Open Government Licence (Malaysia); daily (weekdays, 1700 MYT).
- [MET Malaysia Weather Forecast](https://www.data-pulse.my/data/met_weather.md): Open Government Licence (Malaysia); daily.
- [DOE APIMS Air Quality (Hourly API)](https://www.data-pulse.my/data/doe_apims.md): Open Government Licence (Malaysia); hourly.
- [DOE RQIMS River Water Quality (Continuous)](https://www.data-pulse.my/data/doe_rqims.md): Open Government Licence (Malaysia); hourly.
- [DOE MQIMS Marine Water Quality (Manual)](https://www.data-pulse.my/data/doe_mqims.md): Open Government Licence (Malaysia); monthly.
- [KKM iDengue Weekly Dengue Cases](https://www.data-pulse.my/data/kkm_idengue.md): Open Government Licence (Malaysia); daily.
- [OpenDOSM Crime by District & Type (Annual)](https://www.data-pulse.my/data/dosm_crime_district.md): Creative Commons Attribution 4.0; annual.
- [OpenDOSM Monthly CPI by State & Division](https://www.data-pulse.my/data/dosm_cpi_state.md): Creative Commons Attribution 4.0; monthly.
- [OpenDOSM Annual Real GDP by State & Sector](https://www.data-pulse.my/data/dosm_gdp_state_real_supply.md): Creative Commons Attribution 4.0; annual.
<!-- END featured-datasets -->

## Current public artifacts

<!-- BEGIN public-artifacts -->
- [Live dashboard](https://www.data-pulse.my/): Human-readable dataset status cards with embedded health and manifest data.
- [Landing page](https://www.data-pulse.my/landing.html): Public DataPulse MY overview.
- [NPRA page](https://www.data-pulse.my/npra.html): Public NPRA dataset surface.
- [Health methodology](https://www.data-pulse.my/health-methodology.html): Published status and freshness methodology.
- [Buyer API reference](https://www.data-pulse.my/buyer-api-reference.md): Read-only buyer API contract and examples.
- [LLM index](https://www.data-pulse.my/llms.txt): Machine-readable discovery index for agents.
- [Dataset manifest](https://www.data-pulse.my/datapulse.json): Full machine-readable dataset manifest.
- [Manifest JSON Schema](https://www.data-pulse.my/datapulse.schema.json): Machine-readable schema for the dataset manifest.
- [Latest health snapshot](https://www.data-pulse.my/health/latest.json): Current published dataset health evidence.
- [Published trends](https://www.data-pulse.my/health/trends.json): Per-dataset freshness trend and reliability evidence.
- [Published drift](https://www.data-pulse.my/health/drift.json): Published structural and record-count drift evidence.
- [Published reconciliation](https://www.data-pulse.my/health/reconciliation.json): Cross-source publication differences for human review.
- [RSS feed](https://www.data-pulse.my/feed.xml): Dataset health changes with status-tagged entries.
- [Changelog](https://www.data-pulse.my/changelog.json): Machine-readable release-by-release summary.
- [Agent manifest](https://www.data-pulse.my/agent.json): Machine-readable agent capability manifest.
- [MCP advertisement](https://www.data-pulse.my/mcp.json): Machine-readable MCP server advertisement.
- [JSON-LD catalog](https://www.data-pulse.my/data/jsonld/catalog.json): Schema.org JSON-LD dataset catalog.
- [Status badges](https://www.data-pulse.my/badges/): Per-dataset SVG health badges.
<!-- END public-artifacts -->

## Trust posture

DataPulse MY distinguishes between:

- HTTP reachability (URL returns any response)
- Browser-required access (Camofox needed to render the page)
- Schema validity (record count within expected tolerance)
- Freshness (Last-Modified header or parsed content date within declared cadence)
- Discontinued data (`discontinued`) when the source has stopped publishing and the data is frozen at its last known content date; this is a publisher decision, not a freshness failure
- Unknown freshness (`unknown-freshness`) when HTTP and content shape pass but neither freshness signal is available
- Reference data (`reference`) when a versioned lookup table is reachable and countable but date-based freshness does not apply

A time-series dataset is marked `fresh` only when the applicable direct-access,
schema, and freshness signals pass. See `health/latest.json` `_trust_summary` for the current
distribution and the count of signals that could not be extracted.