adam-acs/docs/deployment/tr069-exchange-viewer.md

Deployment Guide: TR-069 Exchange Viewer

Overview

The TR-069 Exchange Viewer stores CWMP protocol exchanges in MongoDB and streams them in real-time via SSE (Server-Sent Events) to a frontend sequence diagram. This guide covers the infrastructure requirements, configuration, feature flags, and monitoring needed for deployment.

MongoDB Setup

Requirements

• MongoDB version: 4.4+ (tested with 5.0 and 6.0)
• Replica set: Not required. The system uses an in-process EventEmitter for real-time notifications instead of MongoDB Change Streams.
• Authentication: Standard MongoDB credentials (same as the existing adam-acs database)

Collections

The feature creates two MongoDB collections:

Collection	Purpose
cwmp_exchanges	Stores structured CWMP exchange documents
cwmp_exchange_counters	Atomic counters for per-device sequence number generation

Indexes

Three indexes are created on cwmp_exchanges:

Name	Fields	Options	Purpose
idx_deviceId_timestamp	{ deviceId: 1, timestamp: 1 }	—	Efficient retrieval by device, ordered by time
idx_deviceId_sequenceNumber	{ deviceId: 1, sequenceNumber: 1 }	unique: true	Atomic sequence numbering, Last-Event-ID queries
idx_createdAt_ttl	{ createdAt: 1 }	expireAfterSeconds: 86400	Auto-remove exchanges older than 24 hours

Running the Migration

Indexes and collections are created via migration 011_create_cwmp_exchanges_collection.ts:

# Run all pending migrations
node ace migration:run

# Verify migration applied
node ace migration:status

Index Verification

On startup, the application checks that all expected indexes exist and logs warnings if any are missing. The health check endpoint also reports index status (see Health Check below).

To manually verify indexes in the MongoDB shell:

db.cwmp_exchanges.getIndexes()
// Expected: _id, idx_deviceId_timestamp, idx_deviceId_sequenceNumber, idx_createdAt_ttl

Storage Considerations

• TTL: Documents are automatically removed after 24 hours (configurable via EXCHANGE_STORE_TTL_SECONDS)
• Message cap: Maximum 5000 exchanges per device (configurable via EXCHANGE_STORE_MAX_MESSAGES)
• Document size: Each exchange document stores raw XML (max 2 MB per message)
• Estimated storage: For 100 traced devices with 5000 messages averaging 10 KB each → ~5 GB peak

---

Environment Variables

Feature Flags

Variable	Type	Default	Description
FEATURE_EXCHANGE_PERSISTENCE	Boolean	true	Enable/disable MongoDB persistence of exchanges
FEATURE_SSE_ENABLED	Boolean	true	Enable/disable the SSE streaming endpoint
FEATURE_EXCHANGE_API_SOURCE	String	mongodb	Data source for the Exchange API: mongodb or file

Exchange Store Configuration

Variable	Type	Default	Description
EXCHANGE_STORE_MAX_MESSAGES	Number	5000	Max exchanges stored per device
EXCHANGE_STORE_TTL_SECONDS	Number	86400	TTL in seconds for auto-removal (24h)
EXCHANGE_STORE_READ_TIMEOUT_MS	Number	5000	Query timeout before returning partial results

SSE Configuration

Variable	Type	Default	Description
EXCHANGE_STORE_SSE_INACTIVITY_TIMEOUT_MS	Number	300000	Close SSE connection after 5 min of inactivity
EXCHANGE_STORE_SSE_ABSOLUTE_TIMEOUT_MS	Number	7200000	Close SSE connection after 2 hours regardless
EXCHANGE_STORE_SSE_MAX_CONNECTIONS_PER_DEVICE	Number	5	Max concurrent SSE connections per device
EXCHANGE_STORE_SSE_RETRY_INTERVAL_MS	Number	3000	Retry interval sent to clients for reconnection

Retry Policy

Variable	Type	Default	Description
EXCHANGE_STORE_RETRY_MAX_ATTEMPTS	Number	3	Max retries for critical message persistence
EXCHANGE_STORE_RETRY_BASE_DELAY_MS	Number	1000	Base delay for exponential backoff (1s, 2s, 4s)

Example .env Addition

# TR-069 Exchange Viewer - Feature Flags
FEATURE_EXCHANGE_PERSISTENCE=true
FEATURE_SSE_ENABLED=true
FEATURE_EXCHANGE_API_SOURCE=mongodb

# TR-069 Exchange Viewer - Storage
EXCHANGE_STORE_MAX_MESSAGES=5000
EXCHANGE_STORE_TTL_SECONDS=86400
EXCHANGE_STORE_READ_TIMEOUT_MS=5000

# TR-069 Exchange Viewer - SSE
EXCHANGE_STORE_SSE_INACTIVITY_TIMEOUT_MS=300000
EXCHANGE_STORE_SSE_ABSOLUTE_TIMEOUT_MS=7200000
EXCHANGE_STORE_SSE_MAX_CONNECTIONS_PER_DEVICE=5
EXCHANGE_STORE_SSE_RETRY_INTERVAL_MS=3000

# TR-069 Exchange Viewer - Retry Policy
EXCHANGE_STORE_RETRY_MAX_ATTEMPTS=3
EXCHANGE_STORE_RETRY_BASE_DELAY_MS=1000

---

Feature Flags Usage and Toggle Procedures

The system uses three feature flags for gradual rollout and safe rollback. All flags can be toggled at runtime by changing environment variables and restarting the application.

FEATURE_EXCHANGE_PERSISTENCE

Controls whether CWMP exchanges are written to MongoDB.

Value	Behavior
true (default)	Exchanges are persisted to MongoDB for traced devices
false	No writes to MongoDB; CWMP sessions continue unaffected

When to disable:

• MongoDB is experiencing high latency or unavailability
• Storage exhaustion on the MongoDB server
• You need to temporarily reduce load on the database

Toggle procedure:

# Disable persistence
export FEATURE_EXCHANGE_PERSISTENCE=false
# Restart the application
docker compose restart app

Note: Disabling persistence means no new exchanges are stored. Existing exchanges remain queryable until TTL expires.

FEATURE_SSE_ENABLED

Controls the real-time streaming endpoint.

Value	Behavior
true (default)	SSE endpoint is available at /api/v1/tr069-trace/:deviceId/exchanges/stream
false	SSE endpoint returns 503; frontend falls back to REST polling every 5 seconds

When to disable:

• High number of concurrent SSE connections causing resource exhaustion
• Network infrastructure (proxy/load balancer) not supporting long-lived connections
• Debugging connectivity issues

Toggle procedure:

# Disable SSE streaming
export FEATURE_SSE_ENABLED=false
# Restart the application
docker compose restart app

Note: The frontend automatically falls back to REST polling (5-second interval) when SSE is unavailable. No user action required.

FEATURE_EXCHANGE_API_SOURCE

Controls where the Exchange API reads data from.

Value	Behavior
mongodb (default)	API reads from the cwmp_exchanges MongoDB collection
file	API reads from the legacy file-based trace system

When to switch to file:

• MongoDB data is corrupted or inconsistent
• Rollback to the previous system during migration
• Comparing outputs between old and new systems

Toggle procedure:

# Fallback to file-based system
export FEATURE_EXCHANGE_API_SOURCE=file
# Restart the application
docker compose restart app

Recommended Rollout Sequence

1. Phase 1 — Dual-write (default state):

   • FEATURE_EXCHANGE_PERSISTENCE=true
   • FEATURE_SSE_ENABLED=true
   • FEATURE_EXCHANGE_API_SOURCE=mongodb
   • Old file-based system continues writing in parallel (dual-write)

2. Phase 2 — Validate:

   • Compare file-based and MongoDB exchange counts
   • Monitor health check for discrepancies
   • Verify SSE connections are stable

3. Phase 3 — Full cutover:

   • Remove file-based writes from CwmpLoggerMiddleware
   • FEATURE_EXCHANGE_API_SOURCE=mongodb (permanently)

Emergency Rollback

To fully disable the exchange viewer feature without stopping the application:

export FEATURE_EXCHANGE_PERSISTENCE=false
export FEATURE_SSE_ENABLED=false
export FEATURE_EXCHANGE_API_SOURCE=file
docker compose restart app

This restores pre-feature behavior: no MongoDB writes, no SSE connections, and the API reads from the legacy file system.

---

Health Check Endpoint

Endpoint

GET /api/v1/health

The existing health check endpoint (/api/v1/health) reports on the overall application status including MongoDB connectivity and cwmp_exchanges index status.

Response Format

{
  "status": "OK",
  "version": "1.0.0",
  "indexes": {
    "cwmp_exchanges": {
      "allPresent": true,
      "indexes": [
        { "name": "idx_deviceId_timestamp", "present": true },
        { "name": "idx_deviceId_sequenceNumber", "present": true },
        { "name": "idx_createdAt_ttl", "present": true }
      ]
    }
  }
}

Status Codes

Status	Code	Meaning
OK	200	MongoDB connected, all indexes present
KO	503	MongoDB unreachable or critical failure

Monitoring Recommendations

Metric	Threshold	Action
Health check status KO	Any occurrence	Investigate MongoDB connectivity
Missing indexes (allPresent: false)	Any occurrence	Re-run migration: node ace migration:run
MongoDB ping latency	> 100ms p95	Check MongoDB server load
Active SSE connections	> 100 total	Consider scaling or limiting connections

Integration with Monitoring Tools

The health check endpoint can be used with standard monitoring tools:

# Simple liveness check (Docker HEALTHCHECK)
curl -f http://localhost:8080/api/v1/health || exit 1

# Kubernetes readiness probe
livenessProbe:
  httpGet:
    path: /api/v1/health
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 30

# Nagios/Zabbix check
curl -s http://localhost:8080/api/v1/health | jq -e '.status == "OK"'

---

API Endpoints

The feature exposes three endpoints under the /api/v1 prefix:

Method	Path	Description
GET	/api/v1/tr069-trace/:deviceId/exchanges	Retrieve exchanges (paginated)
DELETE	/api/v1/tr069-trace/:deviceId/exchanges	Purge all exchanges for a device
GET	/api/v1/tr069-trace/:deviceId/exchanges/stream	SSE endpoint for real-time streaming

---

Reverse Proxy Configuration

If running behind a reverse proxy (nginx, Traefik, etc.), configure it to support SSE:

Nginx

location /api/v1/tr069-trace/ {
    proxy_pass http://app:8080;
    proxy_http_version 1.1;
    proxy_set_header Connection '';
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 7200s;  # Match absolute timeout (2h)
    chunked_transfer_encoding off;
}

Traefik

http:
  middlewares:
    sse-headers:
      headers:
        customResponseHeaders:
          X-Accel-Buffering: "no"
  services:
    adam-acs:
      loadBalancer:
        responseForwarding:
          flushInterval: "0ms"

---

Docker Deployment

No additional Docker services are needed. The feature uses the existing MongoDB instance.

# docker-compose.yml (relevant section)
services:
  app:
    environment:
      - FEATURE_EXCHANGE_PERSISTENCE=true
      - FEATURE_SSE_ENABLED=true
      - FEATURE_EXCHANGE_API_SOURCE=mongodb
    depends_on:
      - mongodb

Ensure the MongoDB container has sufficient storage for exchange data (see Storage Considerations).