# Controlled Market Intelligence API — private beta The [developer interface guide](API_ACCESS_UI.md) describes console routes, admin review and the limits of the current workflow. Public HTML documentation is served at `/data-access/api-docs/`; use `?format=text` for this reference. Customer endpoints (GET only): - `/api/v1/market-intelligence/products/` — paginated Product ID/name. - `/api/v1/market-intelligence/markets/` — paginated Location ID/name/region. - `/api/v1/market-intelligence/prices/` — existing observation schema below. - `/api/v1/market-intelligence/trends/` — daily decimal average/minimum/maximum, samples and latest_update, preserving exact observation context. Create an application at `/accounts/api-applications/` using your existing login. Name it, describe its use case and select sandbox evaluation. Save the returned key immediately: its secret is shown once, never emailed or recoverable. Keys expire after 90 days. Rotation atomically revokes the old key and shows a new one while preserving staff-assigned plans and scopes; revocation is immediate even for cached responses. Separate test/live keys belong to the application, not the login session. Maximum ten applications per account. Sandbox uses real observations, not simulated data. Production requires accepting agreement 2026-11-06 in the portal and staff approval in Django admin → API applications. Staff must review the use case and upstream rights before checking production_approved. The owner then creates a live key. Staff can change key plans/scopes/expiry or revoke keys in admin. Removing approval disables every live key. Admin review buttons can approve, reject or revoke production access. Rejection records production_rejected_at (migration pricing.0006) and clears approval; owners may resubmit for another review. Sandbox credentials remain separate. Changing the agreement version in settings requires renewed acceptance. Scopes are `prices:read` (observations and lookups) and `trends:read`. Unknown scopes do not grant permissions. Use only `Authorization: Bearer `, over HTTPS. Never put keys in URLs. Key format is `tb_test_.` or `tb_live_.`. The 24-character public ID is indexed; the random secret has 32 bytes entropy. Only its keyed HMAC-SHA256 digest is stored, using dedicated API_KEY_PEPPER. Losing/changing the pepper invalidates existing keys; rotate all keys after such a change. Keep it separate from SECRET_KEY / DJANGO_SECRET_KEY. Missing pepper disables issuance and authentication, including in production. Never log Authorization headers. The credential portal suppresses third-party scripts and sends no-store. ## Customer filters, pagination and beta limits Prices require both `from` and `to` (inclusive Kigali dates), at most 366 days. Other supported filters are commodity, market, district, updated_since and page_size, as described below. Default page_size is the plan maximum. Follow the returned relative `next` URL alone. Cursors expire after one hour, bind to the key, and preserve initial maximum observation ID and filters. Products/markets accept no initial filters; follow their signed `next` links. Metadata is reserved for internal sync; customers use the paginated lookups. Trends require from/to, commodity, market, source (positive source ID) and **all** of unit, currency, channel, variety and grade (explicit empty string means unknown). No mixed-context averages. Range is limited to the plan's page maximum in days (25 or 100), and returns at most one row per observed day, with context and next=null. No trend cursor: request subsequent bounded date windows. updated_since is rejected for trends to avoid presenting a changed-row subset as a full daily average. If a smaller page_size is supplied, the date span must fit that size. Product/source names can change independently of observation timestamps; refresh lookups regularly. | Plan | Requests/minute | Requests/UTC day | Maximum records/page | | --- | --- | --- | --- | | Sandbox | 10 | 500 | 25 | | Approved pilot | 30 | 5000 | 100 | Beta limits are configurable policies, not measured VPS capacity guarantees. Institutional requirements require manual review; no institutional billing or unbounded export plan exists. Defaults: two concurrent requests per key, eight globally, 256 KiB JSON response, 10-second PostgreSQL statement timeout, 60-second response cache. API_LIMITS / API_* settings and matching environment variables in .env.example control these. Sandbox environment always uses sandbox limits, even if staff changes its plan. Cache hits still consume quota and authenticate. 401 invalid/expired/revoked key; 403 missing scope; 400 invalid filters, oversized response or expired cursor; 405 unsupported method; 429 quota/capacity exceeded with Retry-After; 503 unavailable Redis/database. No full-dataset synchronous customer response. Existing public CSV exports and their behavior are unchanged. ```bash # Supply TANGABID_API_KEY securely in the client environment. curl --fail-with-body -H "Authorization: Bearer $TANGABID_API_KEY" \ 'https://tangabid.rw/api/v1/market-intelligence/prices/?from=2026-10-01&to=2026-10-31&page_size=25' curl --fail-with-body -H "Authorization: Bearer $TANGABID_API_KEY" \ 'https://tangabid.rw/api/v1/market-intelligence/trends/?from=2026-10-01&to=2026-10-07&commodity=1&market=2&source=1&unit=kg¤cy=RWF&channel=wholesale&variety=&grade=' ``` ## Operations, privacy and deployment gates Apply pricing.0005 after review; it adds only API applications and keys. It does not change observations. Set API_KEY_PEPPER and API_REDIS_URL. Docker includes a separate internal api-redis service: AOF with appendfsync always, noeviction, 64 MiB cap and persistent volume. Do not point API_REDIS_URL at the existing allkeys-lru Redis cache: requests reject unsafe eviction configuration. Redis CONFIG GET permission is required for the policy check. No local-memory fallback exists. Monitor persistence, restarts, memory and 503s. Lost Redis state loses quotas; keep API disabled during recovery until counters are restored or the UTC quota day passes. Lua atomically checks minute/day quotas and key/global leases; leases recover after worker crashes. Do not increase DB statement timeouts beyond lease duration (at least 60 seconds). Use PostgreSQL in production; SQLite cannot enforce statement_timeout and is for development/tests only. Queries run in a read-only PostgreSQL transaction with statement_timeout. The shared connection role still has wider privileges for the Django application; set API_DB_USER and API_DB_PASSWORD to use the optional api_read connection for customer data queries. Provision this PostgreSQL role with SELECT only on pricing_marketprice, market_product, market_location and market_source (plus schema USAGE/connection privileges). Authentication and portal writes stay on default. Role grants are not provisioned automatically; without API_DB_USER the shared role is used inside a read-only transaction. Existing product/location/time and observation-time indexes match the allowlisted filters; no speculative index added. EXPLAIN and load-test production-scale data before expanding limits. Redis keeps aggregate requests/status/endpoint/latency/row counts and last-used at per key for 30 days after last use, alongside expiring quota counters. One accepted request costs one unit, including cache hits and validation failures. No request bodies, IPs, full query strings or secrets are stored by this usage tracker. Owners see usage in the portal; administrators can inspect application ownership and credentials in Django admin. Existing account records remain. Nginx API access logs omit query strings and headers, but infrastructure error logs and Cloudflare policies require operator review. Portal pages have no-store; configure proxies/APM to redact headers, form bodies and credential responses. Nginx configuration adds API connection/request ceilings and short timeouts. Edge rejections include Retry-After: 1; application quota responses provide the quota-window retry interval. Per-IP limits see proxy addresses unless trusted real-IP handling is configured, so review topology before enabling that config. No Cloudflare config exists in the repository and no remote change is made: operators should apply broad abuse rules, bypass caching for authenticated API and credential portal, and avoid logging authorization. Cloudflare rules are not a substitute for Redis. Data-use agreement: cite TangaBid, upstream source, extraction date and filters; preserve units/context/freshness; do not claim completeness or guaranteed accuracy; keep credentials private and respect quotas. API access grants no additional upstream rights and does not restrict data already available via public CSV. Source licensing and commercial redistribution rights need explicit confirmation before production approval. Institutional access is reviewed case by case. No new claim of exclusive ownership, certified quality, or freshness SLA. Prices are DecimalField(10,2), not display text; no ambiguous text coercion was introduced. Quality flags and raw source values are not persisted in MarketPrice; raw ingestion files remain the audit source. Unknown unit/context stays null. Observation dates, updated_at and source attribution are returned; latest_update on a trend describes Tangabid edits, not source revisions. See limitations below. ## Existing internal forecasting compatibility The following section documents the pre-existing shared-token internal sync. It is **not** the customer contract. Keep MARKET_PRICE_SYNC_TOKEN unset on public serving environments; if retained on an internal deployment, restrict it at the network/edge because it intentionally retains its historical backfill behavior and is outside customer Redis quotas. Never distribute it to customers. # Read-only market-price sync API (v1) `GET /api/v1/market-intelligence/prices/` `GET /api/v1/market-intelligence/metadata/` ## Data provenance and limits The API reads `pricing.MarketPrice`, the normalized observations already queried by Market Intelligence, using existing Product, Location and Source relationships. The active Esoko pipeline cleans raw records and loads these observations; the legacy normalizer can also populate the same table. This is not a separately certified clean dataset: cleaner warnings/outlier flags and source update times are not persisted on MarketPrice. No quality label is inferred or returned. No ingestion, price calculation or historical unit values are changed. `district` returns Location.region verbatim (null if blank). The loader stores a reviewed district there, but can fall back to province when district is missing; there is no district entity, ID, or reliable district-only classification. Markets use Location IDs; different location types can share a display name. Observation unit is returned as stored, never filled from Product.unit. Migration `pricing.0004` adds `updated_at` and a sync index to the existing table. Existing rows receive the migration time, because historical modification times are unavailable. Database triggers track corrections through save, update, bulk_update and existing ingestion save(update_fields=...). Inserts receive their creation timestamp. SQLite and PostgreSQL are supported. Timestamps describe Tangabid record changes, not source-system revisions. Related product/market/source name or region edits do not touch price timestamps: refresh metadata each sync and join commodity/market names by ID locally. Source-name changes require a full price refresh if that denormalized name matters. Deletions have no tombstones; periodic full reconciliation is necessary to detect deleted observations. ## Setup and authentication There is no existing DRF installation or machine API authentication. This API uses Django JSON views and a dedicated read-only bearer token, independent of user sessions. It cannot access accounts or write data. 1. Generate a random credential locally, e.g. `openssl rand -hex 32`. 2. Set `MARKET_PRICE_SYNC_TOKEN` in the server environment and restart workers. Keep it out of version control. Unset/empty disables API access. 3. Apply migrations with `python manage.py migrate` before serving requests. 4. Set the same token in the forecasting workspace's environment. Send `Authorization: Bearer `. Missing/invalid tokens return JSON 401; unsupported authenticated methods return 405 (CSRF middleware may reject unsafe methods earlier). GET is the only supported method. HTTPS is required outside loopback. Tokens never belong in query strings. Rotate by replacing the server and client environment values. Responses use `Cache-Control: no-store`. ## Filters and pagination | Parameter | Meaning | | --- | --- | | `from`, `to` | Inclusive observation dates, strictly YYYY-MM-DD, Africa/Kigali | | `commodity` | Positive Product ID | | `market` | Positive Location ID | | `district` | Case-insensitive exact match on stored Location.region | | `updated_since` | Inclusive record-update lower bound, ISO 8601 with timezone | | `page_size` | 1–2000, default 500 | | `cursor` | Opaque signed continuation; use alone, preferably follow `next` | Filters combine with AND and run in the database. Malformed dates/timestamps, reversed ranges, invalid IDs, repeated/unknown parameters, invalid cursors and out-of-range page sizes return JSON 400. Well-formed unmatched IDs return empty results. Metadata accepts no parameters and lists only values represented by observations; blank units, regions and currencies are omitted from its lists. Rows are ordered by ascending immutable observation ID. Cursors bind filters, page size, last ID, starting maximum ID and starting timestamp. `next` is a relative URL, or null when complete. No offset/count scan is used. Cursors remain valid while the Django signing key is unchanged. The scan is bounded, not a transactional snapshot: corrections can occur during it. Newly inserted higher IDs appear in the next incremental run. ## Client synchronization For the initial backfill, request prices without `updated_since` or observation date filters, then follow every `next` URL. Upsert by observation `id`, parsing `price` as a decimal. Persist the first response's `sync_started_at` only after all pages have been successfully stored. A failed page can be retried with the same URL; replaying a run is safe with upserts. For the next run, supply the last successfully stored `sync_started_at` as `updated_since`, follow all pages, then advance the checkpoint to this run's `sync_started_at`. Use the run start rather than its end or the largest timestamp seen: corrections to already-read IDs during pagination must be fetched again. Use an overlap (at least one second for SQLite timestamp precision, and longer than the maximum expected writer transaction duration) when computing the lower bound. Long-running transactions may commit writes whose timestamp predates a checkpoint; no timestamp API can guarantee those without bounded transaction durations or periodic overlapping/full reconciliation. Apply a consistent filter scope across runs; changing it requires a new backfill. Refresh metadata each run. ```bash export TANGABID_URL=http://127.0.0.1:8000 # TANGABID_SYNC_TOKEN is supplied through your local environment. curl --fail-with-body -H "Authorization: Bearer $TANGABID_SYNC_TOKEN" \ "$TANGABID_URL/api/v1/market-intelligence/prices/?page_size=500" curl --fail-with-body -G -H "Authorization: Bearer $TANGABID_SYNC_TOKEN" \ --data-urlencode 'updated_since=2026-10-29T08:00:00+00:00' \ --data-urlencode 'page_size=500' \ "$TANGABID_URL/api/v1/market-intelligence/prices/" curl --fail-with-body -G -H "Authorization: Bearer $TANGABID_SYNC_TOKEN" \ --data-urlencode 'from=2025-01-01' --data-urlencode 'to=2025-12-31' \ --data-urlencode 'commodity=1' --data-urlencode 'market=2' \ "$TANGABID_URL/api/v1/market-intelligence/prices/" curl --fail-with-body -H "Authorization: Bearer $TANGABID_SYNC_TOKEN" \ "$TANGABID_URL/api/v1/market-intelligence/metadata/" ``` Illustrative response (IDs and values are examples): ```json { "sync_started_at": "2026-10-29T08:00:01.123456+00:00", "next": null, "results": [{ "id": 123, "observation_date": "2025-06-01", "observed_at": "2025-05-31T22:00:00+00:00", "commodity": {"id": 1, "name": "Beans"}, "market": {"id": 2, "name": "Example market"}, "district": "GASABO", "price": "1234.56", "unit": "kg", "currency": "RWF", "source": {"id": 1, "name": "Esoko"}, "variety": null, "grade": null, "channel": "wholesale", "updated_at": "2026-10-29T07:30:00+00:00" }] } ``` `observed_at` preserves the original observation time. `observation_date` is its Kigali calendar date. `updated_at` is timezone-aware. IDs are database primary keys. Price is a string preserving the model's two decimal places; absent observations are absent rows, never synthesized zeros. Unknown context is null. Metadata returns `commodities` (id/name), `markets` (id/name/district), and string arrays `districts`, `units`, `currencies`. No user or account fields are exposed.