Documents, observations and revisions

Current document contract#

The documents dataset retains one record per source URL. id is the public record identifier. title can be translated; original_title preserves the retained original. title_lang and is_translated tell you how the title is presented. source and source_url provide provenance. published_at can be unknown; collected_at and first_seen are first collection and last_seen the latest. publisher and acquisition_route (registry_listing, rss_feed, news_proxy, api) say who published it and how we reached it; translated_title is null when the original is English.

Deduplication#

The public coverage contract says one retained record per source URL. It does not promise semantic deduplication across mirrors or publishers. Two URLs can describe the same event; repeated retrieval of one ID does not imply a new publication.

Original text#

text_status distinguishes available, source_only and not_extracted. Approved detail can contain original_text, text_truncated and rights_basis_url. It is an original-language extraction, not a promised full translation. Rights approval is bound to the current extraction; later changes can withhold it.

Observations and revisions#

Every collection of a source URL is an observation: observation_count on the list row, observations[] (observed_at, acquisition_route; capped, with observations_total) on the detail. revisions[] lists retained content versions by version_id, content_hash and observed_at, and version_count counts them; the private body of each version is never served. change_type is revised when a different content hash was observed after first_seen, else newly_discovered when the publisher date is more than two days before first_seen (Beijing date), else new. Rolling out: the change feed, the document observation fields and revisions[] answer once database migration 043 is applied on a deployment. Before that, /api/changes and mode=changes answer HTTP 503 with code unavailable ("not yet enabled"), and list rows return the pending fields as null, named in unavailable_fields.

Coming soon: the text of each historical version. Indicator releases and editorial topic versions are different objects.

Fields#