adam-acs/docs/deployment/tr069-exchange-viewer.md

# Deployment Guide: TR-069 Exchange Viewer

## Overview

The TR-069 Exchange Viewer stores CWMP protocol exchanges in MongoDB and streams them in real-time via SSE (Server-Sent Events) to a frontend sequence diagram. This guide covers the infrastructure requirements, configuration, feature flags, and monitoring needed for deployment.

## MongoDB Setup

### Requirements

- **MongoDB version**: 4.4+ (tested with 5.0 and 6.0)
- **Replica set**: Not required. The system uses an in-process EventEmitter for real-time notifications instead of MongoDB Change Streams.
- **Authentication**: Standard MongoDB credentials (same as the existing adam-acs database)

### Collections

The feature creates two MongoDB collections:

| Collection | Purpose |
|-----------|---------|
| `cwmp_exchanges` | Stores structured CWMP exchange documents |
| `cwmp_exchange_counters` | Atomic counters for per-device sequence number generation |

### Indexes

Three indexes are created on `cwmp_exchanges`:

| Name | Fields | Options | Purpose |
|------|--------|---------|---------|
| `idx_deviceId_timestamp` | `{ deviceId: 1, timestamp: 1 }` | — | Efficient retrieval by device, ordered by time |
| `idx_deviceId_sequenceNumber` | `{ deviceId: 1, sequenceNumber: 1 }` | `unique: true` | Atomic sequence numbering, Last-Event-ID queries |
| `idx_createdAt_ttl` | `{ createdAt: 1 }` | `expireAfterSeconds: 86400` | Auto-remove exchanges older than 24 hours |

### Running the Migration

Indexes and collections are created via migration `011_create_cwmp_exchanges_collection.ts`:

```bash
# Run all pending migrations
node ace migration:run

# Verify migration applied
node ace migration:status
```

### Index Verification

On startup, the application checks that all expected indexes exist and logs warnings if any are missing. The health check endpoint also reports index status (see [Health Check](#health-check-endpoint) below).

To manually verify indexes in the MongoDB shell:

```javascript
db.cwmp_exchanges.getIndexes()
// Expected: _id, idx_deviceId_timestamp, idx_deviceId_sequenceNumber, idx_createdAt_ttl
```

### Storage Considerations

- **TTL**: Documents are automatically removed after 24 hours (configurable via `EXCHANGE_STORE_TTL_SECONDS`)
- **Message cap**: Maximum 5000 exchanges per device (configurable via `EXCHANGE_STORE_MAX_MESSAGES`)
- **Document size**: Each exchange document stores raw XML (max 2 MB per message)
- **Estimated storage**: For 100 traced devices with 5000 messages averaging 10 KB each → ~5 GB peak

---

## Environment Variables

### Feature Flags

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `FEATURE_EXCHANGE_PERSISTENCE` | Boolean | `true` | Enable/disable MongoDB persistence of exchanges |
| `FEATURE_SSE_ENABLED` | Boolean | `true` | Enable/disable the SSE streaming endpoint |
| `FEATURE_EXCHANGE_API_SOURCE` | String | `mongodb` | Data source for the Exchange API: `mongodb` or `file` |

### Exchange Store Configuration

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `EXCHANGE_STORE_MAX_MESSAGES` | Number | `5000` | Max exchanges stored per device |
| `EXCHANGE_STORE_TTL_SECONDS` | Number | `86400` | TTL in seconds for auto-removal (24h) |
| `EXCHANGE_STORE_READ_TIMEOUT_MS` | Number | `5000` | Query timeout before returning partial results |

### SSE Configuration

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `EXCHANGE_STORE_SSE_INACTIVITY_TIMEOUT_MS` | Number | `300000` | Close SSE connection after 5 min of inactivity |
| `EXCHANGE_STORE_SSE_ABSOLUTE_TIMEOUT_MS` | Number | `7200000` | Close SSE connection after 2 hours regardless |
| `EXCHANGE_STORE_SSE_MAX_CONNECTIONS_PER_DEVICE` | Number | `5` | Max concurrent SSE connections per device |
| `EXCHANGE_STORE_SSE_RETRY_INTERVAL_MS` | Number | `3000` | Retry interval sent to clients for reconnection |

### Retry Policy

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `EXCHANGE_STORE_RETRY_MAX_ATTEMPTS` | Number | `3` | Max retries for critical message persistence |
| `EXCHANGE_STORE_RETRY_BASE_DELAY_MS` | Number | `1000` | Base delay for exponential backoff (1s, 2s, 4s) |

### Example .env Addition

```dotenv
# TR-069 Exchange Viewer - Feature Flags
FEATURE_EXCHANGE_PERSISTENCE=true
FEATURE_SSE_ENABLED=true
FEATURE_EXCHANGE_API_SOURCE=mongodb

# TR-069 Exchange Viewer - Storage
EXCHANGE_STORE_MAX_MESSAGES=5000
EXCHANGE_STORE_TTL_SECONDS=86400
EXCHANGE_STORE_READ_TIMEOUT_MS=5000

# TR-069 Exchange Viewer - SSE
EXCHANGE_STORE_SSE_INACTIVITY_TIMEOUT_MS=300000
EXCHANGE_STORE_SSE_ABSOLUTE_TIMEOUT_MS=7200000
EXCHANGE_STORE_SSE_MAX_CONNECTIONS_PER_DEVICE=5
EXCHANGE_STORE_SSE_RETRY_INTERVAL_MS=3000

# TR-069 Exchange Viewer - Retry Policy
EXCHANGE_STORE_RETRY_MAX_ATTEMPTS=3
EXCHANGE_STORE_RETRY_BASE_DELAY_MS=1000
```

---

## Feature Flags Usage and Toggle Procedures

The system uses three feature flags for gradual rollout and safe rollback. All flags can be toggled at runtime by changing environment variables and restarting the application.

### FEATURE_EXCHANGE_PERSISTENCE

Controls whether CWMP exchanges are written to MongoDB.

| Value | Behavior |
|-------|----------|
| `true` (default) | Exchanges are persisted to MongoDB for traced devices |
| `false` | No writes to MongoDB; CWMP sessions continue unaffected |

**When to disable:**
- MongoDB is experiencing high latency or unavailability
- Storage exhaustion on the MongoDB server
- You need to temporarily reduce load on the database

**Toggle procedure:**
```bash
# Disable persistence
export FEATURE_EXCHANGE_PERSISTENCE=false
# Restart the application
docker compose restart app
```

> Note: Disabling persistence means no new exchanges are stored. Existing exchanges remain queryable until TTL expires.

### FEATURE_SSE_ENABLED

Controls the real-time streaming endpoint.

| Value | Behavior |
|-------|----------|
| `true` (default) | SSE endpoint is available at `/api/v1/tr069-trace/:deviceId/exchanges/stream` |
| `false` | SSE endpoint returns 503; frontend falls back to REST polling every 5 seconds |

**When to disable:**
- High number of concurrent SSE connections causing resource exhaustion
- Network infrastructure (proxy/load balancer) not supporting long-lived connections
- Debugging connectivity issues

**Toggle procedure:**
```bash
# Disable SSE streaming
export FEATURE_SSE_ENABLED=false
# Restart the application
docker compose restart app
```

> Note: The frontend automatically falls back to REST polling (5-second interval) when SSE is unavailable. No user action required.

### FEATURE_EXCHANGE_API_SOURCE

Controls where the Exchange API reads data from.

| Value | Behavior |
|-------|----------|
| `mongodb` (default) | API reads from the `cwmp_exchanges` MongoDB collection |
| `file` | API reads from the legacy file-based trace system |

**When to switch to `file`:**
- MongoDB data is corrupted or inconsistent
- Rollback to the previous system during migration
- Comparing outputs between old and new systems

**Toggle procedure:**
```bash
# Fallback to file-based system
export FEATURE_EXCHANGE_API_SOURCE=file
# Restart the application
docker compose restart app
```

### Recommended Rollout Sequence

1. **Phase 1 — Dual-write** (default state):
   - `FEATURE_EXCHANGE_PERSISTENCE=true`
   - `FEATURE_SSE_ENABLED=true`
   - `FEATURE_EXCHANGE_API_SOURCE=mongodb`
   - Old file-based system continues writing in parallel (dual-write)

2. **Phase 2 — Validate**:
   - Compare file-based and MongoDB exchange counts
   - Monitor health check for discrepancies
   - Verify SSE connections are stable

3. **Phase 3 — Full cutover**:
   - Remove file-based writes from `CwmpLoggerMiddleware`
   - `FEATURE_EXCHANGE_API_SOURCE=mongodb` (permanently)

### Emergency Rollback

To fully disable the exchange viewer feature without stopping the application:

```bash
export FEATURE_EXCHANGE_PERSISTENCE=false
export FEATURE_SSE_ENABLED=false
export FEATURE_EXCHANGE_API_SOURCE=file
docker compose restart app
```

This restores pre-feature behavior: no MongoDB writes, no SSE connections, and the API reads from the legacy file system.

---

## Health Check Endpoint

### Endpoint

```
GET /api/v1/health
```

The existing health check endpoint (`/api/v1/health`) reports on the overall application status including MongoDB connectivity and `cwmp_exchanges` index status.

### Response Format

```json
{
  "status": "OK",
  "version": "1.0.0",
  "indexes": {
    "cwmp_exchanges": {
      "allPresent": true,
      "indexes": [
        { "name": "idx_deviceId_timestamp", "present": true },
        { "name": "idx_deviceId_sequenceNumber", "present": true },
        { "name": "idx_createdAt_ttl", "present": true }
      ]
    }
  }
}
```

### Status Codes

| Status | Code | Meaning |
|--------|------|---------|
| OK | 200 | MongoDB connected, all indexes present |
| KO | 503 | MongoDB unreachable or critical failure |

### Monitoring Recommendations

| Metric | Threshold | Action |
|--------|-----------|--------|
| Health check status `KO` | Any occurrence | Investigate MongoDB connectivity |
| Missing indexes (`allPresent: false`) | Any occurrence | Re-run migration: `node ace migration:run` |
| MongoDB ping latency | > 100ms p95 | Check MongoDB server load |
| Active SSE connections | > 100 total | Consider scaling or limiting connections |

### Integration with Monitoring Tools

The health check endpoint can be used with standard monitoring tools:

```bash
# Simple liveness check (Docker HEALTHCHECK)
curl -f http://localhost:8080/api/v1/health || exit 1

# Kubernetes readiness probe
livenessProbe:
  httpGet:
    path: /api/v1/health
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 30

# Nagios/Zabbix check
curl -s http://localhost:8080/api/v1/health | jq -e '.status == "OK"'
```

---

## API Endpoints

The feature exposes three endpoints under the `/api/v1` prefix:

| Method | Path | Description |
|--------|------|-------------|
| GET | `/api/v1/tr069-trace/:deviceId/exchanges` | Retrieve exchanges (paginated) |
| DELETE | `/api/v1/tr069-trace/:deviceId/exchanges` | Purge all exchanges for a device |
| GET | `/api/v1/tr069-trace/:deviceId/exchanges/stream` | SSE endpoint for real-time streaming |

---

## Reverse Proxy Configuration

If running behind a reverse proxy (nginx, Traefik, etc.), configure it to support SSE:

### Nginx

```nginx
location /api/v1/tr069-trace/ {
    proxy_pass http://app:8080;
    proxy_http_version 1.1;
    proxy_set_header Connection '';
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 7200s;  # Match absolute timeout (2h)
    chunked_transfer_encoding off;
}
```

### Traefik

```yaml
http:
  middlewares:
    sse-headers:
      headers:
        customResponseHeaders:
          X-Accel-Buffering: "no"
  services:
    adam-acs:
      loadBalancer:
        responseForwarding:
          flushInterval: "0ms"
```

---

## Docker Deployment

No additional Docker services are needed. The feature uses the existing MongoDB instance.

```yaml
# docker-compose.yml (relevant section)
services:
  app:
    environment:
      - FEATURE_EXCHANGE_PERSISTENCE=true
      - FEATURE_SSE_ENABLED=true
      - FEATURE_EXCHANGE_API_SOURCE=mongodb
    depends_on:
      - mongodb
```

Ensure the MongoDB container has sufficient storage for exchange data (see [Storage Considerations](#storage-considerations)).