feat: TR-069 Exchange Viewer - structured MongoDB storage with SSE streaming and sequence diagram UI
- MongoDB storage: cwmp_exchanges collection with indexes (deviceId+timestamp, deviceId+sequenceNumber unique, TTL 24h) - ExchangeStoreService: CWMP method extraction, per-device queue for ordering, retry logic, message cap (5000/device) - Exchange API: GET/DELETE/SSE stream endpoints with Last-Event-ID reconnection - CwmpLoggerMiddleware: dual-write (MongoDB + legacy files) with fire-and-forget - Frontend: SequenceViewer (CPE↔ACS lifelines, arrows, auto-scroll, virtual scroll) + InspectorPanel (formatted XML, copy, syntax highlighting) - Tr069Trace page: integrated new viewer with col-6/col-6 layout - DeviceDetail page: collapsible exchange section when device is traced - 14 property-based tests, 40+ unit tests, 5 integration tests - Health check includes index verification - Config: feature flags, SSE timeouts, retry policy
This commit is contained in:
Kevin L'Hereec committed
1 parent
3fefbf6c40
commit
0c83a4ccb2
50 files changed
+15731
-43
No files matched your search
@@ -0,0 +1,475 @@
|
||||
openapi: 3.0.3
|
||||
info:
|
||||
title: TR-069 Exchange Viewer API
|
||||
description: |
|
||||
API REST pour la récupération, la suppression et le streaming en temps réel
|
||||
des échanges CWMP/TR-069 capturés lors d'une session de trace sur un device.
|
||||
|
||||
## Architecture
|
||||
|
||||
Les échanges SOAP sont capturés par le `CwmpLoggerMiddleware`, persistés dans
|
||||
MongoDB via `ExchangeStoreService`, et diffusés en temps réel via Server-Sent Events (SSE).
|
||||
|
||||
## Authentification
|
||||
|
||||
Tous les endpoints nécessitent une session authentifiée (cookie-based via AdonisJS).
|
||||
|
||||
## Feature Flags
|
||||
|
||||
| Variable d'environnement | Effet |
|
||||
|--------------------------|-------|
|
||||
| `FEATURE_EXCHANGE_PERSISTENCE=false` | Désactive l'écriture MongoDB |
|
||||
| `FEATURE_SSE_ENABLED=false` | Désactive le SSE (endpoint renvoie 503) |
|
||||
| `FEATURE_EXCHANGE_API_SOURCE=file` | Lecture depuis l'ancien système fichier |
|
||||
version: 1.0.0
|
||||
contact:
|
||||
name: adam-acs Team
|
||||
|
||||
servers:
|
||||
- url: http://localhost:8080/api/v1
|
||||
description: Local development
|
||||
|
||||
paths:
|
||||
/tr069-trace/{deviceId}/exchanges:
|
||||
get:
|
||||
operationId: getExchanges
|
||||
summary: Récupérer les échanges CWMP d'un device
|
||||
description: |
|
||||
Retourne les échanges CWMP structurés pour un device tracé, ordonnés par
|
||||
timestamp ascending. Inclut des métadonnées sur le statut de la trace et
|
||||
la fenêtre d'enregistrement.
|
||||
|
||||
**Pagination cursor-based** : utiliser `after` pour charger les messages plus
|
||||
récents, `before` pour charger les messages plus anciens, ou `latest=true`
|
||||
pour obtenir les N messages les plus récents (chargement initial).
|
||||
tags:
|
||||
- Exchanges
|
||||
parameters:
|
||||
- $ref: '#/components/parameters/deviceId'
|
||||
- name: limit
|
||||
in: query
|
||||
required: false
|
||||
description: |
|
||||
Nombre maximum de messages retournés.
|
||||
- Défaut : 200
|
||||
- Minimum : 1
|
||||
- Maximum : 1000
|
||||
- Retourne HTTP 400 si non-numérique ou < 1
|
||||
schema:
|
||||
type: integer
|
||||
minimum: 1
|
||||
maximum: 1000
|
||||
default: 200
|
||||
example: 50
|
||||
- name: after
|
||||
in: query
|
||||
required: false
|
||||
description: |
|
||||
Timestamp ISO 8601 pour la pagination cursor-based (messages plus récents).
|
||||
Seuls les messages avec un timestamp strictement supérieur à cette valeur
|
||||
sont retournés. Retourne HTTP 400 si le format est invalide.
|
||||
schema:
|
||||
type: string
|
||||
format: date-time
|
||||
example: '2025-01-15T10:30:00.000Z'
|
||||
- name: before
|
||||
in: query
|
||||
required: false
|
||||
description: |
|
||||
Timestamp ISO 8601 pour la pagination inverse (messages plus anciens).
|
||||
Seuls les messages avec un timestamp strictement inférieur à cette valeur
|
||||
sont retournés. Retourne HTTP 400 si le format est invalide.
|
||||
schema:
|
||||
type: string
|
||||
format: date-time
|
||||
example: '2025-01-15T10:35:00.000Z'
|
||||
- name: latest
|
||||
in: query
|
||||
required: false
|
||||
description: |
|
||||
Si `true`, retourne les N messages les plus récents (pour le chargement
|
||||
initial). Combiné avec `limit` pour contrôler le nombre de messages.
|
||||
schema:
|
||||
type: string
|
||||
enum:
|
||||
- 'true'
|
||||
- 'false'
|
||||
example: 'true'
|
||||
responses:
|
||||
'200':
|
||||
description: Liste des échanges avec métadonnées
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/ExchangeListResponse'
|
||||
examples:
|
||||
activeTrace:
|
||||
summary: Device actuellement tracé avec des échanges
|
||||
value:
|
||||
exchanges:
|
||||
- sequenceNumber: 1
|
||||
timestamp: '2025-01-15T10:30:00.123Z'
|
||||
direction: 'CPE→ACS'
|
||||
cwmpMethod: 'Inform'
|
||||
sessionId: '10.0.0.1:54321->192.168.1.1:8080'
|
||||
rawXml: '<soap-env:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/" xmlns:cwmp="urn:dslforum-org:cwmp-1-0"><soap-env:Body><cwmp:Inform>...</cwmp:Inform></soap-env:Body></soap-env:Envelope>'
|
||||
- sequenceNumber: 2
|
||||
timestamp: '2025-01-15T10:30:00.456Z'
|
||||
direction: 'ACS→CPE'
|
||||
cwmpMethod: 'InformResponse'
|
||||
sessionId: '10.0.0.1:54321->192.168.1.1:8080'
|
||||
rawXml: '<soap-env:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/" xmlns:cwmp="urn:dslforum-org:cwmp-1-0"><soap-env:Body><cwmp:InformResponse><MaxEnvelopes>1</MaxEnvelopes></cwmp:InformResponse></soap-env:Body></soap-env:Envelope>'
|
||||
metadata:
|
||||
traceStatus: 'ACTIVE'
|
||||
traceStoppedAt: null
|
||||
recordedFrom: '2025-01-15T10:30:00.123Z'
|
||||
recordedUntil: '2025-01-15T10:35:42.567Z'
|
||||
total: 42
|
||||
stoppedTrace:
|
||||
summary: Trace arrêtée avec historique conservé
|
||||
value:
|
||||
exchanges:
|
||||
- sequenceNumber: 1
|
||||
timestamp: '2025-01-15T10:30:00.123Z'
|
||||
direction: 'CPE→ACS'
|
||||
cwmpMethod: 'Inform'
|
||||
sessionId: '10.0.0.1:54321->192.168.1.1:8080'
|
||||
rawXml: '<soap-env:Envelope>...</soap-env:Envelope>'
|
||||
metadata:
|
||||
traceStatus: 'STOPPED'
|
||||
traceStoppedAt: '2025-01-15T10:40:00.000Z'
|
||||
recordedFrom: '2025-01-15T10:30:00.123Z'
|
||||
recordedUntil: '2025-01-15T10:35:42.567Z'
|
||||
total: 15
|
||||
emptyTrace:
|
||||
summary: Device tracé mais aucun échange encore enregistré
|
||||
value:
|
||||
exchanges: []
|
||||
metadata:
|
||||
traceStatus: 'ACTIVE'
|
||||
traceStoppedAt: null
|
||||
recordedFrom: null
|
||||
recordedUntil: null
|
||||
total: 0
|
||||
truncatedResult:
|
||||
summary: Résultat partiel (timeout de lecture dépassé)
|
||||
value:
|
||||
exchanges: []
|
||||
metadata:
|
||||
traceStatus: 'ACTIVE'
|
||||
traceStoppedAt: null
|
||||
recordedFrom: '2025-01-15T10:30:00.123Z'
|
||||
recordedUntil: '2025-01-15T10:35:42.567Z'
|
||||
total: 5000
|
||||
isTruncated: true
|
||||
'400':
|
||||
description: Paramètre de requête invalide
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/ErrorResponse'
|
||||
examples:
|
||||
invalidLimit:
|
||||
summary: Paramètre limit invalide
|
||||
value:
|
||||
error: 'Invalid limit parameter. Must be an integer between 1 and 1000.'
|
||||
invalidAfter:
|
||||
summary: Paramètre after invalide
|
||||
value:
|
||||
error: 'Invalid after parameter. Must be a valid ISO 8601 timestamp.'
|
||||
invalidBefore:
|
||||
summary: Paramètre before invalide
|
||||
value:
|
||||
error: 'Invalid before parameter. Must be a valid ISO 8601 timestamp.'
|
||||
'404':
|
||||
description: Le device n'a jamais été tracé (aucun échange historique)
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/ErrorResponse'
|
||||
example:
|
||||
error: 'No trace exists for device F8B7FF-Livebox4-LB4-TEST-001'
|
||||
|
||||
delete:
|
||||
operationId: deleteExchanges
|
||||
summary: Supprimer tous les échanges d'un device
|
||||
description: |
|
||||
Supprime tous les échanges CWMP stockés pour le device spécifié.
|
||||
Retourne 204 sans corps de réponse en cas de succès.
|
||||
|
||||
Cette opération est irréversible. Si le device est actuellement tracé,
|
||||
les nouveaux échanges continueront d'être enregistrés après la suppression.
|
||||
tags:
|
||||
- Exchanges
|
||||
parameters:
|
||||
- $ref: '#/components/parameters/deviceId'
|
||||
responses:
|
||||
'204':
|
||||
description: Échanges supprimés avec succès (pas de corps de réponse)
|
||||
|
||||
/tr069-trace/{deviceId}/exchanges/stream:
|
||||
get:
|
||||
operationId: streamExchanges
|
||||
summary: Flux SSE temps réel des échanges CWMP
|
||||
description: |
|
||||
Ouvre une connexion HTTP persistante avec `Content-Type: text/event-stream`
|
||||
qui pousse chaque nouvel échange CWMP dès qu'il est persisté.
|
||||
|
||||
## Protocole SSE
|
||||
|
||||
La connexion utilise le protocole Server-Sent Events (SSE) standard :
|
||||
|
||||
- **Content-Type** : `text/event-stream`
|
||||
- **Cache-Control** : `no-cache`
|
||||
- **Connection** : `keep-alive`
|
||||
- **X-Accel-Buffering** : `no` (pour nginx)
|
||||
|
||||
## Format des événements
|
||||
|
||||
Chaque événement SSE contient :
|
||||
```
|
||||
id: <sequenceNumber>
|
||||
data: <JSON-encoded CwmpExchange>
|
||||
|
||||
```
|
||||
|
||||
Exemple d'événement brut :
|
||||
```
|
||||
id: 42
|
||||
data: {"sequenceNumber":42,"timestamp":"2025-01-15T10:30:00.123Z","direction":"CPE→ACS","cwmpMethod":"Inform","sessionId":"10.0.0.1:54321->192.168.1.1:8080","rawXml":"<soap-env:Envelope>...</soap-env:Envelope>"}
|
||||
|
||||
```
|
||||
|
||||
## Reconnexion (Last-Event-ID)
|
||||
|
||||
Lors d'une reconnexion, le client envoie l'en-tête `Last-Event-ID` avec le
|
||||
`sequenceNumber` du dernier événement reçu. Le serveur reprend alors à partir
|
||||
du message suivant (sequenceNumber > Last-Event-ID), sans duplication.
|
||||
|
||||
Le serveur envoie un champ `retry` au début de la connexion (défaut: 3000ms)
|
||||
pour configurer l'intervalle de reconnexion automatique du client.
|
||||
|
||||
## Timeouts serveur
|
||||
|
||||
Le serveur ferme la connexion dans les cas suivants :
|
||||
- **Inactivité** : aucun nouvel échange poussé pendant 5 minutes
|
||||
- **Durée absolue** : connexion ouverte depuis plus de 2 heures
|
||||
|
||||
Dans les deux cas, le client devrait se reconnecter automatiquement
|
||||
(mécanisme natif EventSource avec Last-Event-ID).
|
||||
|
||||
## Feature Flag
|
||||
|
||||
Si `FEATURE_SSE_ENABLED=false`, cet endpoint retourne HTTP 503.
|
||||
tags:
|
||||
- Exchanges
|
||||
- SSE
|
||||
parameters:
|
||||
- $ref: '#/components/parameters/deviceId'
|
||||
- name: Last-Event-ID
|
||||
in: header
|
||||
required: false
|
||||
description: |
|
||||
Sequence number du dernier événement reçu par le client.
|
||||
Utilisé pour reprendre le flux après une déconnexion sans duplication.
|
||||
schema:
|
||||
type: string
|
||||
example: '42'
|
||||
responses:
|
||||
'200':
|
||||
description: Flux SSE ouvert avec succès
|
||||
headers:
|
||||
Content-Type:
|
||||
schema:
|
||||
type: string
|
||||
example: text/event-stream
|
||||
Cache-Control:
|
||||
schema:
|
||||
type: string
|
||||
example: no-cache
|
||||
Connection:
|
||||
schema:
|
||||
type: string
|
||||
example: keep-alive
|
||||
X-Accel-Buffering:
|
||||
schema:
|
||||
type: string
|
||||
example: 'no'
|
||||
content:
|
||||
text/event-stream:
|
||||
schema:
|
||||
type: string
|
||||
description: Flux d'événements SSE (voir description pour le format)
|
||||
example: |
|
||||
retry: 3000
|
||||
|
||||
id: 1
|
||||
data: {"sequenceNumber":1,"timestamp":"2025-01-15T10:30:00.123Z","direction":"CPE→ACS","cwmpMethod":"Inform","sessionId":"10.0.0.1:54321->192.168.1.1:8080","rawXml":"<soap-env:Envelope>...</soap-env:Envelope>"}
|
||||
|
||||
id: 2
|
||||
data: {"sequenceNumber":2,"timestamp":"2025-01-15T10:30:00.456Z","direction":"ACS→CPE","cwmpMethod":"InformResponse","sessionId":"10.0.0.1:54321->192.168.1.1:8080","rawXml":"<soap-env:Envelope>...</soap-env:Envelope>"}
|
||||
|
||||
'404':
|
||||
description: Le device n'est pas actuellement tracé
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/ErrorResponse'
|
||||
example:
|
||||
error: 'Device is not being traced'
|
||||
'503':
|
||||
description: Le streaming SSE est désactivé (feature flag)
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: '#/components/schemas/ErrorResponse'
|
||||
example:
|
||||
error: 'SSE streaming is currently disabled'
|
||||
|
||||
components:
|
||||
parameters:
|
||||
deviceId:
|
||||
name: deviceId
|
||||
in: path
|
||||
required: true
|
||||
description: |
|
||||
Identifiant du device (format: `OUI-ProductClass-SerialNumber`
|
||||
ou hash MongoDB `_id`).
|
||||
schema:
|
||||
type: string
|
||||
maxLength: 128
|
||||
example: 'F8B7FF-Livebox4-LB4-TEST-001'
|
||||
|
||||
schemas:
|
||||
CwmpExchange:
|
||||
type: object
|
||||
description: Un échange SOAP CWMP individuel capturé
|
||||
required:
|
||||
- sequenceNumber
|
||||
- timestamp
|
||||
- direction
|
||||
- cwmpMethod
|
||||
- sessionId
|
||||
- rawXml
|
||||
properties:
|
||||
sequenceNumber:
|
||||
type: integer
|
||||
description: |
|
||||
Numéro de séquence monotoniquement croissant par device (commence à 1).
|
||||
Utilisé comme identifiant d'événement SSE pour la reconnexion.
|
||||
example: 42
|
||||
timestamp:
|
||||
type: string
|
||||
format: date-time
|
||||
description: Horodatage UTC de la capture (ISO 8601)
|
||||
example: '2025-01-15T10:30:00.123Z'
|
||||
direction:
|
||||
type: string
|
||||
enum:
|
||||
- 'CPE→ACS'
|
||||
- 'ACS→CPE'
|
||||
description: |
|
||||
Direction du message :
|
||||
- `CPE→ACS` : requête envoyée par le CPE vers l'ACS
|
||||
- `ACS→CPE` : réponse envoyée par l'ACS vers le CPE
|
||||
example: 'CPE→ACS'
|
||||
cwmpMethod:
|
||||
type: string
|
||||
description: |
|
||||
Méthode CWMP extraite du body SOAP :
|
||||
- Nom de la méthode (ex: `Inform`, `GetParameterValues`, `SetParameterValuesResponse`)
|
||||
- `"Empty"` si le body XML est vide (empty POST du CPE)
|
||||
- `"Unknown"` si aucun élément `cwmp:` n'est trouvé dans le body
|
||||
example: 'Inform'
|
||||
sessionId:
|
||||
type: string
|
||||
description: Identifiant de session TCP (format `IP:port->IP:port`)
|
||||
example: '10.0.0.1:54321->192.168.1.1:8080'
|
||||
rawXml:
|
||||
type: string
|
||||
description: |
|
||||
Contenu XML SOAP brut du message (maximum 2 Mo).
|
||||
Peut être vide pour les messages de type "Empty".
|
||||
maxLength: 2097152
|
||||
example: '<soap-env:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/">...</soap-env:Envelope>'
|
||||
|
||||
ExchangeMetadata:
|
||||
type: object
|
||||
description: Métadonnées sur la session de trace du device
|
||||
required:
|
||||
- traceStatus
|
||||
- traceStoppedAt
|
||||
- recordedFrom
|
||||
- recordedUntil
|
||||
- total
|
||||
properties:
|
||||
traceStatus:
|
||||
type: string
|
||||
enum:
|
||||
- ACTIVE
|
||||
- STOPPED
|
||||
description: |
|
||||
Statut actuel de la trace :
|
||||
- `ACTIVE` : le device est en cours de trace
|
||||
- `STOPPED` : la trace a été arrêtée (historique conservé)
|
||||
example: 'ACTIVE'
|
||||
traceStoppedAt:
|
||||
type: string
|
||||
format: date-time
|
||||
nullable: true
|
||||
description: Timestamp d'arrêt de la trace (null si active)
|
||||
example: null
|
||||
recordedFrom:
|
||||
type: string
|
||||
format: date-time
|
||||
nullable: true
|
||||
description: Timestamp du plus ancien échange enregistré
|
||||
example: '2025-01-15T10:30:00.123Z'
|
||||
recordedUntil:
|
||||
type: string
|
||||
format: date-time
|
||||
nullable: true
|
||||
description: Timestamp du plus récent échange enregistré
|
||||
example: '2025-01-15T10:35:42.567Z'
|
||||
total:
|
||||
type: integer
|
||||
description: Nombre total d'échanges stockés pour ce device
|
||||
example: 42
|
||||
isTruncated:
|
||||
type: boolean
|
||||
description: |
|
||||
Présent uniquement si le résultat est partiel (timeout de lecture dépassé).
|
||||
Le client devrait réessayer avec des paramètres de pagination plus restreints.
|
||||
example: true
|
||||
|
||||
ExchangeListResponse:
|
||||
type: object
|
||||
description: Réponse de l'endpoint GET /exchanges
|
||||
required:
|
||||
- exchanges
|
||||
- metadata
|
||||
properties:
|
||||
exchanges:
|
||||
type: array
|
||||
items:
|
||||
$ref: '#/components/schemas/CwmpExchange'
|
||||
description: Liste des échanges ordonnés par timestamp ascending
|
||||
metadata:
|
||||
$ref: '#/components/schemas/ExchangeMetadata'
|
||||
|
||||
ErrorResponse:
|
||||
type: object
|
||||
description: Réponse d'erreur standard
|
||||
required:
|
||||
- error
|
||||
properties:
|
||||
error:
|
||||
type: string
|
||||
description: Message d'erreur décrivant le problème
|
||||
example: 'Invalid limit parameter. Must be an integer between 1 and 1000.'
|
||||
|
||||
tags:
|
||||
- name: Exchanges
|
||||
description: Gestion des échanges CWMP capturés
|
||||
- name: SSE
|
||||
description: Streaming temps réel via Server-Sent Events
|
||||
@@ -0,0 +1,363 @@
|
||||
# Deployment Guide: TR-069 Exchange Viewer
|
||||
|
||||
## Overview
|
||||
|
||||
The TR-069 Exchange Viewer stores CWMP protocol exchanges in MongoDB and streams them in real-time via SSE (Server-Sent Events) to a frontend sequence diagram. This guide covers the infrastructure requirements, configuration, feature flags, and monitoring needed for deployment.
|
||||
|
||||
## MongoDB Setup
|
||||
|
||||
### Requirements
|
||||
|
||||
- **MongoDB version**: 4.4+ (tested with 5.0 and 6.0)
|
||||
- **Replica set**: Not required. The system uses an in-process EventEmitter for real-time notifications instead of MongoDB Change Streams.
|
||||
- **Authentication**: Standard MongoDB credentials (same as the existing adam-acs database)
|
||||
|
||||
### Collections
|
||||
|
||||
The feature creates two MongoDB collections:
|
||||
|
||||
| Collection | Purpose |
|
||||
|-----------|---------|
|
||||
| `cwmp_exchanges` | Stores structured CWMP exchange documents |
|
||||
| `cwmp_exchange_counters` | Atomic counters for per-device sequence number generation |
|
||||
|
||||
### Indexes
|
||||
|
||||
Three indexes are created on `cwmp_exchanges`:
|
||||
|
||||
| Name | Fields | Options | Purpose |
|
||||
|------|--------|---------|---------|
|
||||
| `idx_deviceId_timestamp` | `{ deviceId: 1, timestamp: 1 }` | — | Efficient retrieval by device, ordered by time |
|
||||
| `idx_deviceId_sequenceNumber` | `{ deviceId: 1, sequenceNumber: 1 }` | `unique: true` | Atomic sequence numbering, Last-Event-ID queries |
|
||||
| `idx_createdAt_ttl` | `{ createdAt: 1 }` | `expireAfterSeconds: 86400` | Auto-remove exchanges older than 24 hours |
|
||||
|
||||
### Running the Migration
|
||||
|
||||
Indexes and collections are created via migration `011_create_cwmp_exchanges_collection.ts`:
|
||||
|
||||
```bash
|
||||
# Run all pending migrations
|
||||
node ace migration:run
|
||||
|
||||
# Verify migration applied
|
||||
node ace migration:status
|
||||
```
|
||||
|
||||
### Index Verification
|
||||
|
||||
On startup, the application checks that all expected indexes exist and logs warnings if any are missing. The health check endpoint also reports index status (see [Health Check](#health-check-endpoint) below).
|
||||
|
||||
To manually verify indexes in the MongoDB shell:
|
||||
|
||||
```javascript
|
||||
db.cwmp_exchanges.getIndexes()
|
||||
// Expected: _id, idx_deviceId_timestamp, idx_deviceId_sequenceNumber, idx_createdAt_ttl
|
||||
```
|
||||
|
||||
### Storage Considerations
|
||||
|
||||
- **TTL**: Documents are automatically removed after 24 hours (configurable via `EXCHANGE_STORE_TTL_SECONDS`)
|
||||
- **Message cap**: Maximum 5000 exchanges per device (configurable via `EXCHANGE_STORE_MAX_MESSAGES`)
|
||||
- **Document size**: Each exchange document stores raw XML (max 2 MB per message)
|
||||
- **Estimated storage**: For 100 traced devices with 5000 messages averaging 10 KB each → ~5 GB peak
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables
|
||||
|
||||
### Feature Flags
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `FEATURE_EXCHANGE_PERSISTENCE` | Boolean | `true` | Enable/disable MongoDB persistence of exchanges |
|
||||
| `FEATURE_SSE_ENABLED` | Boolean | `true` | Enable/disable the SSE streaming endpoint |
|
||||
| `FEATURE_EXCHANGE_API_SOURCE` | String | `mongodb` | Data source for the Exchange API: `mongodb` or `file` |
|
||||
|
||||
### Exchange Store Configuration
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `EXCHANGE_STORE_MAX_MESSAGES` | Number | `5000` | Max exchanges stored per device |
|
||||
| `EXCHANGE_STORE_TTL_SECONDS` | Number | `86400` | TTL in seconds for auto-removal (24h) |
|
||||
| `EXCHANGE_STORE_READ_TIMEOUT_MS` | Number | `5000` | Query timeout before returning partial results |
|
||||
|
||||
### SSE Configuration
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `EXCHANGE_STORE_SSE_INACTIVITY_TIMEOUT_MS` | Number | `300000` | Close SSE connection after 5 min of inactivity |
|
||||
| `EXCHANGE_STORE_SSE_ABSOLUTE_TIMEOUT_MS` | Number | `7200000` | Close SSE connection after 2 hours regardless |
|
||||
| `EXCHANGE_STORE_SSE_MAX_CONNECTIONS_PER_DEVICE` | Number | `5` | Max concurrent SSE connections per device |
|
||||
| `EXCHANGE_STORE_SSE_RETRY_INTERVAL_MS` | Number | `3000` | Retry interval sent to clients for reconnection |
|
||||
|
||||
### Retry Policy
|
||||
|
||||
| Variable | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `EXCHANGE_STORE_RETRY_MAX_ATTEMPTS` | Number | `3` | Max retries for critical message persistence |
|
||||
| `EXCHANGE_STORE_RETRY_BASE_DELAY_MS` | Number | `1000` | Base delay for exponential backoff (1s, 2s, 4s) |
|
||||
|
||||
### Example .env Addition
|
||||
|
||||
```dotenv
|
||||
# TR-069 Exchange Viewer - Feature Flags
|
||||
FEATURE_EXCHANGE_PERSISTENCE=true
|
||||
FEATURE_SSE_ENABLED=true
|
||||
FEATURE_EXCHANGE_API_SOURCE=mongodb
|
||||
|
||||
# TR-069 Exchange Viewer - Storage
|
||||
EXCHANGE_STORE_MAX_MESSAGES=5000
|
||||
EXCHANGE_STORE_TTL_SECONDS=86400
|
||||
EXCHANGE_STORE_READ_TIMEOUT_MS=5000
|
||||
|
||||
# TR-069 Exchange Viewer - SSE
|
||||
EXCHANGE_STORE_SSE_INACTIVITY_TIMEOUT_MS=300000
|
||||
EXCHANGE_STORE_SSE_ABSOLUTE_TIMEOUT_MS=7200000
|
||||
EXCHANGE_STORE_SSE_MAX_CONNECTIONS_PER_DEVICE=5
|
||||
EXCHANGE_STORE_SSE_RETRY_INTERVAL_MS=3000
|
||||
|
||||
# TR-069 Exchange Viewer - Retry Policy
|
||||
EXCHANGE_STORE_RETRY_MAX_ATTEMPTS=3
|
||||
EXCHANGE_STORE_RETRY_BASE_DELAY_MS=1000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Feature Flags Usage and Toggle Procedures
|
||||
|
||||
The system uses three feature flags for gradual rollout and safe rollback. All flags can be toggled at runtime by changing environment variables and restarting the application.
|
||||
|
||||
### FEATURE_EXCHANGE_PERSISTENCE
|
||||
|
||||
Controls whether CWMP exchanges are written to MongoDB.
|
||||
|
||||
| Value | Behavior |
|
||||
|-------|----------|
|
||||
| `true` (default) | Exchanges are persisted to MongoDB for traced devices |
|
||||
| `false` | No writes to MongoDB; CWMP sessions continue unaffected |
|
||||
|
||||
**When to disable:**
|
||||
- MongoDB is experiencing high latency or unavailability
|
||||
- Storage exhaustion on the MongoDB server
|
||||
- You need to temporarily reduce load on the database
|
||||
|
||||
**Toggle procedure:**
|
||||
```bash
|
||||
# Disable persistence
|
||||
export FEATURE_EXCHANGE_PERSISTENCE=false
|
||||
# Restart the application
|
||||
docker compose restart app
|
||||
```
|
||||
|
||||
> Note: Disabling persistence means no new exchanges are stored. Existing exchanges remain queryable until TTL expires.
|
||||
|
||||
### FEATURE_SSE_ENABLED
|
||||
|
||||
Controls the real-time streaming endpoint.
|
||||
|
||||
| Value | Behavior |
|
||||
|-------|----------|
|
||||
| `true` (default) | SSE endpoint is available at `/api/v1/tr069-trace/:deviceId/exchanges/stream` |
|
||||
| `false` | SSE endpoint returns 503; frontend falls back to REST polling every 5 seconds |
|
||||
|
||||
**When to disable:**
|
||||
- High number of concurrent SSE connections causing resource exhaustion
|
||||
- Network infrastructure (proxy/load balancer) not supporting long-lived connections
|
||||
- Debugging connectivity issues
|
||||
|
||||
**Toggle procedure:**
|
||||
```bash
|
||||
# Disable SSE streaming
|
||||
export FEATURE_SSE_ENABLED=false
|
||||
# Restart the application
|
||||
docker compose restart app
|
||||
```
|
||||
|
||||
> Note: The frontend automatically falls back to REST polling (5-second interval) when SSE is unavailable. No user action required.
|
||||
|
||||
### FEATURE_EXCHANGE_API_SOURCE
|
||||
|
||||
Controls where the Exchange API reads data from.
|
||||
|
||||
| Value | Behavior |
|
||||
|-------|----------|
|
||||
| `mongodb` (default) | API reads from the `cwmp_exchanges` MongoDB collection |
|
||||
| `file` | API reads from the legacy file-based trace system |
|
||||
|
||||
**When to switch to `file`:**
|
||||
- MongoDB data is corrupted or inconsistent
|
||||
- Rollback to the previous system during migration
|
||||
- Comparing outputs between old and new systems
|
||||
|
||||
**Toggle procedure:**
|
||||
```bash
|
||||
# Fallback to file-based system
|
||||
export FEATURE_EXCHANGE_API_SOURCE=file
|
||||
# Restart the application
|
||||
docker compose restart app
|
||||
```
|
||||
|
||||
### Recommended Rollout Sequence
|
||||
|
||||
1. **Phase 1 — Dual-write** (default state):
|
||||
- `FEATURE_EXCHANGE_PERSISTENCE=true`
|
||||
- `FEATURE_SSE_ENABLED=true`
|
||||
- `FEATURE_EXCHANGE_API_SOURCE=mongodb`
|
||||
- Old file-based system continues writing in parallel (dual-write)
|
||||
|
||||
2. **Phase 2 — Validate**:
|
||||
- Compare file-based and MongoDB exchange counts
|
||||
- Monitor health check for discrepancies
|
||||
- Verify SSE connections are stable
|
||||
|
||||
3. **Phase 3 — Full cutover**:
|
||||
- Remove file-based writes from `CwmpLoggerMiddleware`
|
||||
- `FEATURE_EXCHANGE_API_SOURCE=mongodb` (permanently)
|
||||
|
||||
### Emergency Rollback
|
||||
|
||||
To fully disable the exchange viewer feature without stopping the application:
|
||||
|
||||
```bash
|
||||
export FEATURE_EXCHANGE_PERSISTENCE=false
|
||||
export FEATURE_SSE_ENABLED=false
|
||||
export FEATURE_EXCHANGE_API_SOURCE=file
|
||||
docker compose restart app
|
||||
```
|
||||
|
||||
This restores pre-feature behavior: no MongoDB writes, no SSE connections, and the API reads from the legacy file system.
|
||||
|
||||
---
|
||||
|
||||
## Health Check Endpoint
|
||||
|
||||
### Endpoint
|
||||
|
||||
```
|
||||
GET /api/v1/health
|
||||
```
|
||||
|
||||
The existing health check endpoint (`/api/v1/health`) reports on the overall application status including MongoDB connectivity and `cwmp_exchanges` index status.
|
||||
|
||||
### Response Format
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "OK",
|
||||
"version": "1.0.0",
|
||||
"indexes": {
|
||||
"cwmp_exchanges": {
|
||||
"allPresent": true,
|
||||
"indexes": [
|
||||
{ "name": "idx_deviceId_timestamp", "present": true },
|
||||
{ "name": "idx_deviceId_sequenceNumber", "present": true },
|
||||
{ "name": "idx_createdAt_ttl", "present": true }
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Status Codes
|
||||
|
||||
| Status | Code | Meaning |
|
||||
|--------|------|---------|
|
||||
| OK | 200 | MongoDB connected, all indexes present |
|
||||
| KO | 503 | MongoDB unreachable or critical failure |
|
||||
|
||||
### Monitoring Recommendations
|
||||
|
||||
| Metric | Threshold | Action |
|
||||
|--------|-----------|--------|
|
||||
| Health check status `KO` | Any occurrence | Investigate MongoDB connectivity |
|
||||
| Missing indexes (`allPresent: false`) | Any occurrence | Re-run migration: `node ace migration:run` |
|
||||
| MongoDB ping latency | > 100ms p95 | Check MongoDB server load |
|
||||
| Active SSE connections | > 100 total | Consider scaling or limiting connections |
|
||||
|
||||
### Integration with Monitoring Tools
|
||||
|
||||
The health check endpoint can be used with standard monitoring tools:
|
||||
|
||||
```bash
|
||||
# Simple liveness check (Docker HEALTHCHECK)
|
||||
curl -f http://localhost:8080/api/v1/health || exit 1
|
||||
|
||||
# Kubernetes readiness probe
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /api/v1/health
|
||||
port: 8080
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 30
|
||||
|
||||
# Nagios/Zabbix check
|
||||
curl -s http://localhost:8080/api/v1/health | jq -e '.status == "OK"'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Endpoints
|
||||
|
||||
The feature exposes three endpoints under the `/api/v1` prefix:
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/v1/tr069-trace/:deviceId/exchanges` | Retrieve exchanges (paginated) |
|
||||
| DELETE | `/api/v1/tr069-trace/:deviceId/exchanges` | Purge all exchanges for a device |
|
||||
| GET | `/api/v1/tr069-trace/:deviceId/exchanges/stream` | SSE endpoint for real-time streaming |
|
||||
|
||||
---
|
||||
|
||||
## Reverse Proxy Configuration
|
||||
|
||||
If running behind a reverse proxy (nginx, Traefik, etc.), configure it to support SSE:
|
||||
|
||||
### Nginx
|
||||
|
||||
```nginx
|
||||
location /api/v1/tr069-trace/ {
|
||||
proxy_pass http://app:8080;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection '';
|
||||
proxy_buffering off;
|
||||
proxy_cache off;
|
||||
proxy_read_timeout 7200s; # Match absolute timeout (2h)
|
||||
chunked_transfer_encoding off;
|
||||
}
|
||||
```
|
||||
|
||||
### Traefik
|
||||
|
||||
```yaml
|
||||
http:
|
||||
middlewares:
|
||||
sse-headers:
|
||||
headers:
|
||||
customResponseHeaders:
|
||||
X-Accel-Buffering: "no"
|
||||
services:
|
||||
adam-acs:
|
||||
loadBalancer:
|
||||
responseForwarding:
|
||||
flushInterval: "0ms"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Docker Deployment
|
||||
|
||||
No additional Docker services are needed. The feature uses the existing MongoDB instance.
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml (relevant section)
|
||||
services:
|
||||
app:
|
||||
environment:
|
||||
- FEATURE_EXCHANGE_PERSISTENCE=true
|
||||
- FEATURE_SSE_ENABLED=true
|
||||
- FEATURE_EXCHANGE_API_SOURCE=mongodb
|
||||
depends_on:
|
||||
- mongodb
|
||||
```
|
||||
|
||||
Ensure the MongoDB container has sufficient storage for exchange data (see [Storage Considerations](#storage-considerations)).
|
||||
@@ -0,0 +1,592 @@
|
||||
# Guide de migration : TR-069 Exchange Viewer
|
||||
|
||||
## Résumé
|
||||
|
||||
Ce guide décrit la migration du système de trace TR-069 depuis le stockage fichier (logs `.log` rotatifs) vers le stockage structuré MongoDB (`cwmp_exchanges`). La migration suit une approche en 3 phases pour garantir zéro perte de données et un rollback possible à chaque étape.
|
||||
|
||||
| Phase | Durée | Description | Risque |
|
||||
|-------|-------|-------------|--------|
|
||||
| Phase 1 — Dual-Write | Semaines 1-2 | Écriture simultanée fichiers + MongoDB | Faible |
|
||||
| Phase 2 — Read from MongoDB | Semaines 3-4 | API lit depuis MongoDB, fichiers en backup | Moyen |
|
||||
| Phase 3 — Deprecate Files | Semaine 5+ | Suppression du système fichier | Irréversible |
|
||||
|
||||
---
|
||||
|
||||
## Prérequis
|
||||
|
||||
### Infrastructure
|
||||
|
||||
- MongoDB 5.0+ (pas besoin de replica set — le système utilise un EventEmitter in-process, pas de change streams)
|
||||
- Espace disque MongoDB : prévoir ~500 Mo pour 10 000 devices actifs avec 5 000 messages max chacun
|
||||
- L'application adam-acs doit être redémarrée après la migration de base
|
||||
|
||||
### Variables d'environnement
|
||||
|
||||
```bash
|
||||
# Feature flags (déjà présents dans config/exchange_store.ts)
|
||||
FEATURE_EXCHANGE_PERSISTENCE=true # Active les écritures MongoDB
|
||||
FEATURE_SSE_ENABLED=true # Active le streaming SSE
|
||||
FEATURE_EXCHANGE_API_SOURCE=mongodb # Source des données API (mongodb | file)
|
||||
|
||||
# Configuration optionnelle
|
||||
EXCHANGE_STORE_MAX_MESSAGES=5000 # Cap messages par device
|
||||
EXCHANGE_STORE_TTL_SECONDS=86400 # TTL 24h
|
||||
EXCHANGE_STORE_READ_TIMEOUT_MS=5000 # Timeout requêtes API
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 : Dual-Write (Semaines 1-2)
|
||||
|
||||
### Objectif
|
||||
|
||||
Écrire simultanément dans les deux systèmes (fichiers ET MongoDB) pour valider la fiabilité du nouveau système sans risque de perte.
|
||||
|
||||
### 1.1 Exécuter la migration MongoDB
|
||||
|
||||
La migration crée la collection `cwmp_exchanges` et ses index :
|
||||
|
||||
```bash
|
||||
# Exécuter la migration
|
||||
node ace migration:run --connection=mongodb
|
||||
```
|
||||
|
||||
Cela exécute `database/mongo_migrations/011_create_cwmp_exchanges_collection.ts` qui crée :
|
||||
- Collection `cwmp_exchanges` avec 3 index :
|
||||
- `{ deviceId: 1, timestamp: 1 }` — requêtes par device
|
||||
- `{ deviceId: 1, sequenceNumber: 1 }` (unique) — numérotation séquentielle et reconnexion SSE
|
||||
- `{ createdAt: 1 }` (TTL: 86400s) — nettoyage automatique après 24h
|
||||
- Collection `cwmp_exchange_counters` — compteurs atomiques de séquence
|
||||
|
||||
### 1.2 Vérifier les index
|
||||
|
||||
```bash
|
||||
# Via mongosh
|
||||
mongosh --eval "
|
||||
db.cwmp_exchanges.getIndexes().forEach(idx => {
|
||||
print(idx.name + ': ' + JSON.stringify(idx.key))
|
||||
})
|
||||
"
|
||||
```
|
||||
|
||||
Résultat attendu :
|
||||
```
|
||||
_id_: {"_id":1}
|
||||
idx_deviceId_timestamp: {"deviceId":1,"timestamp":1}
|
||||
idx_deviceId_sequenceNumber: {"deviceId":1,"sequenceNumber":1}
|
||||
idx_createdAt_ttl: {"createdAt":1}
|
||||
```
|
||||
|
||||
### 1.3 Activer le dual-write
|
||||
|
||||
Le dual-write est déjà implémenté dans `CwmpLoggerMiddleware`. Vérifier les feature flags :
|
||||
|
||||
```bash
|
||||
# .env — Phase 1
|
||||
FEATURE_EXCHANGE_PERSISTENCE=true
|
||||
FEATURE_SSE_ENABLED=true
|
||||
FEATURE_EXCHANGE_API_SOURCE=file # GARDER 'file' en Phase 1
|
||||
```
|
||||
|
||||
Le middleware écrit dans les deux systèmes :
|
||||
|
||||
```typescript
|
||||
// app/middleware/cwmp_logger_middleware.ts (extrait)
|
||||
if (isTraced && deviceId) {
|
||||
// Nouveau système : MongoDB (fire-and-forget)
|
||||
exchangeStore.persistExchange({ deviceId, direction, sessionId, rawXml })
|
||||
.catch(() => {})
|
||||
|
||||
// Ancien système : fichiers (gardé pour rollback)
|
||||
traceService.writeExchange(deviceId, direction, sessionId, prettyXml)
|
||||
}
|
||||
```
|
||||
|
||||
### 1.4 Redémarrer l'application
|
||||
|
||||
```bash
|
||||
docker compose build app
|
||||
docker compose up -d app
|
||||
```
|
||||
|
||||
### 1.5 Vérifier le fonctionnement
|
||||
|
||||
Activer le trace sur un device et déclencher un Inform :
|
||||
|
||||
```bash
|
||||
# Vérifier que MongoDB reçoit les données
|
||||
mongosh --eval "db.cwmp_exchanges.countDocuments({})"
|
||||
|
||||
# Vérifier les derniers échanges
|
||||
mongosh --eval "
|
||||
db.cwmp_exchanges.find({}).sort({timestamp: -1}).limit(3).forEach(doc => {
|
||||
print(doc.deviceId + ' | ' + doc.direction + ' | ' + doc.cwmpMethod)
|
||||
})
|
||||
"
|
||||
```
|
||||
|
||||
### 1.6 Vérification de cohérence (Dual-Write)
|
||||
|
||||
Pendant la Phase 1, vérifier régulièrement que les deux systèmes restent synchronisés :
|
||||
|
||||
#### Script de comparaison
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# scripts/verify_dual_write.sh
|
||||
# Compare le nombre d'échanges entre fichiers et MongoDB pour chaque device tracé
|
||||
|
||||
DEVICE_ID=$1
|
||||
|
||||
if [ -z "$DEVICE_ID" ]; then
|
||||
echo "Usage: $0 <deviceId>"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Compteur fichier (nombre de séparateurs '═' = nombre d'échanges)
|
||||
FILE_COUNT=$(grep -c '═════' "logs/tr069/${DEVICE_ID}.log" 2>/dev/null || echo "0")
|
||||
|
||||
# Compteur MongoDB
|
||||
MONGO_COUNT=$(mongosh --quiet --eval "db.cwmp_exchanges.countDocuments({deviceId: '${DEVICE_ID}'})")
|
||||
|
||||
echo "Device: ${DEVICE_ID}"
|
||||
echo " Fichiers : ${FILE_COUNT} échanges"
|
||||
echo " MongoDB : ${MONGO_COUNT} échanges"
|
||||
|
||||
if [ "$FILE_COUNT" != "$MONGO_COUNT" ]; then
|
||||
DIFF=$((FILE_COUNT - MONGO_COUNT))
|
||||
echo " ⚠️ Écart : ${DIFF} messages"
|
||||
if [ "${DIFF#-}" -gt $((FILE_COUNT / 20)) ]; then
|
||||
echo " 🚨 Écart > 5% — Investigation requise !"
|
||||
fi
|
||||
else
|
||||
echo " ✅ Cohérent"
|
||||
fi
|
||||
```
|
||||
|
||||
#### Critères d'acceptation Phase 1
|
||||
|
||||
| Critère | Seuil | Action si dépassé |
|
||||
|---------|-------|-------------------|
|
||||
| Écart fichier/MongoDB | < 5% | Vérifier les logs d'erreur ExchangeStore |
|
||||
| Erreurs de persistance | < 0.1% | Vérifier la connectivité MongoDB |
|
||||
| Latence MongoDB p95 | < 100ms | Vérifier la charge et les index |
|
||||
| Messages perdus (critiques) | 0 après retries | Vérifier les logs de retry exhaustion |
|
||||
|
||||
#### Durée recommandée
|
||||
|
||||
Maintenir le dual-write pendant **minimum 1 semaine** avec des vérifications quotidiennes. Passer en Phase 2 uniquement si les critères ci-dessus sont respectés de manière constante.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 : Read from MongoDB (Semaines 3-4)
|
||||
|
||||
### Objectif
|
||||
|
||||
Basculer la lecture des données vers MongoDB tout en gardant les fichiers en écriture backup.
|
||||
|
||||
### 2.1 Changer la source de lecture API
|
||||
|
||||
```bash
|
||||
# .env — Phase 2
|
||||
FEATURE_EXCHANGE_PERSISTENCE=true
|
||||
FEATURE_SSE_ENABLED=true
|
||||
FEATURE_EXCHANGE_API_SOURCE=mongodb # ← Changer de 'file' à 'mongodb'
|
||||
```
|
||||
|
||||
### 2.2 Redémarrer l'application
|
||||
|
||||
```bash
|
||||
docker compose restart app
|
||||
```
|
||||
|
||||
### 2.3 Vérifier le fonctionnement
|
||||
|
||||
```bash
|
||||
# Vérifier que l'API retourne des données depuis MongoDB
|
||||
curl -s "http://localhost:8080/api/v1/tr069-trace/<DEVICE_ID>/exchanges?limit=5" | \
|
||||
python3 -c "
|
||||
import sys, json
|
||||
data = json.load(sys.stdin)
|
||||
print(f\"Source: MongoDB\")
|
||||
print(f\"Exchanges: {len(data.get('exchanges', []))} messages\")
|
||||
print(f\"Metadata: {data.get('metadata', {})}\")
|
||||
"
|
||||
```
|
||||
|
||||
### 2.4 Vérifier le SSE
|
||||
|
||||
```bash
|
||||
# Ouvrir une connexion SSE (ctrl+C pour arrêter)
|
||||
curl -N -H "Accept: text/event-stream" \
|
||||
"http://localhost:8080/api/v1/tr069-trace/<DEVICE_ID>/exchanges/stream"
|
||||
```
|
||||
|
||||
Déclencher un Inform sur le device tracé et vérifier qu'un événement SSE arrive en temps réel.
|
||||
|
||||
### 2.5 Vérifier le fallback
|
||||
|
||||
Si MongoDB est indisponible, le frontend doit retomber sur les fichiers :
|
||||
|
||||
```typescript
|
||||
// Logique dans DeviceDetail.vue
|
||||
const exchanges = await api.getExchanges(deviceId)
|
||||
.catch(() => legacyService.readExchanges(deviceId))
|
||||
```
|
||||
|
||||
### 2.6 Critères de passage Phase 2 → Phase 3
|
||||
|
||||
| Critère | Seuil | Vérifié |
|
||||
|---------|-------|---------|
|
||||
| API MongoDB fonctionne sans erreur | 48h continues | ☐ |
|
||||
| SSE pousse les événements en < 500ms | p95 vérifié | ☐ |
|
||||
| Aucune perte de message critique | 0 sur 48h | ☐ |
|
||||
| Reconnexion SSE (Last-Event-ID) valide | Testé manuellement | ☐ |
|
||||
| Purge (DELETE) fonctionne | 204 retourné | ☐ |
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 : Deprecate Old System (Semaine 5+)
|
||||
|
||||
### Objectif
|
||||
|
||||
Supprimer le système fichier une fois que MongoDB est validé en production.
|
||||
|
||||
### 3.1 Supprimer le dual-write du middleware
|
||||
|
||||
Dans `app/middleware/cwmp_logger_middleware.ts`, retirer les appels au système fichier :
|
||||
|
||||
```typescript
|
||||
// AVANT (dual-write)
|
||||
if (isTraced && deviceId) {
|
||||
exchangeStore.persistExchange({ deviceId, direction, sessionId, rawXml }).catch(() => {})
|
||||
traceService.writeExchange(deviceId, direction, sessionId, prettyXml) // ← SUPPRIMER
|
||||
}
|
||||
|
||||
// APRÈS (MongoDB uniquement)
|
||||
if (isTraced && deviceId) {
|
||||
exchangeStore.persistExchange({ deviceId, direction, sessionId, rawXml }).catch(() => {})
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 Supprimer les dépendances obsolètes
|
||||
|
||||
1. Retirer l'import `xml-formatter` du middleware (le formatage n'est plus nécessaire côté persistance)
|
||||
2. Supprimer les appels à `Tr069TraceService.writeExchange()` dans le middleware
|
||||
3. **NE PAS** supprimer `Tr069TraceService` lui-même — il est encore utilisé pour gérer l'état trace (start/stop)
|
||||
|
||||
### 3.3 Nettoyer les fichiers de logs
|
||||
|
||||
```bash
|
||||
# Archiver les anciens logs avant suppression
|
||||
tar -czf logs/tr069-archive-$(date +%Y%m%d).tar.gz logs/tr069/
|
||||
rm -rf logs/tr069/*.log*
|
||||
```
|
||||
|
||||
### 3.4 Mise à jour de la configuration
|
||||
|
||||
```bash
|
||||
# .env — Phase 3 (état final)
|
||||
FEATURE_EXCHANGE_PERSISTENCE=true
|
||||
FEATURE_SSE_ENABLED=true
|
||||
FEATURE_EXCHANGE_API_SOURCE=mongodb
|
||||
```
|
||||
|
||||
### 3.5 Vérification finale
|
||||
|
||||
```bash
|
||||
# Vérifier qu'aucun fichier .log n'est créé
|
||||
ls -la logs/tr069/
|
||||
|
||||
# Vérifier que le système fonctionne uniquement via MongoDB
|
||||
curl -s "http://localhost:8080/api/v1/health/tr069-exchanges" | python3 -m json.tool
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Procédures de Rollback
|
||||
|
||||
### Rollback Phase 1 → Système fichier seul
|
||||
|
||||
**Situation :** MongoDB pose problème, il faut revenir au système fichier uniquement.
|
||||
|
||||
```bash
|
||||
# 1. Désactiver la persistance MongoDB
|
||||
FEATURE_EXCHANGE_PERSISTENCE=false
|
||||
|
||||
# 2. Redémarrer
|
||||
docker compose restart app
|
||||
|
||||
# 3. Vérifier que les fichiers continuent d'être écrits
|
||||
ls -la logs/tr069/
|
||||
```
|
||||
|
||||
**Impact :** Les données déjà en MongoDB restent accessibles mais ne sont plus alimentées. Le SSE ne fonctionne plus. L'UI retombe sur les fichiers.
|
||||
|
||||
**Temps de rollback :** < 2 minutes (changement d'env + restart)
|
||||
|
||||
### Rollback Phase 2 → Phase 1 (Dual-Write)
|
||||
|
||||
**Situation :** L'API MongoDB a des problèmes, revenir au dual-write avec lecture fichier.
|
||||
|
||||
```bash
|
||||
# 1. Changer la source API
|
||||
FEATURE_EXCHANGE_API_SOURCE=file
|
||||
|
||||
# 2. Redémarrer
|
||||
docker compose restart app
|
||||
```
|
||||
|
||||
**Impact :** L'API lit les fichiers au lieu de MongoDB. Le dual-write continue. Aucune perte de données.
|
||||
|
||||
**Temps de rollback :** < 2 minutes
|
||||
|
||||
### Rollback Phase 3 → Phase 2
|
||||
|
||||
**Situation :** Après suppression du dual-write, MongoDB pose problème.
|
||||
|
||||
**⚠️ ATTENTION : Ce rollback nécessite un redéploiement du code.**
|
||||
|
||||
1. Revert du commit qui a supprimé le dual-write dans `CwmpLoggerMiddleware`
|
||||
2. Redéployer l'application
|
||||
3. Vérifier que les fichiers sont à nouveau écrits
|
||||
|
||||
```bash
|
||||
# Identifier le commit de suppression du dual-write
|
||||
git log --oneline -- app/middleware/cwmp_logger_middleware.ts
|
||||
|
||||
# Revert
|
||||
git revert <commit-hash>
|
||||
docker compose build app
|
||||
docker compose up -d app
|
||||
```
|
||||
|
||||
**Impact :** Nécessite un redéploiement (~5 minutes). Les échanges entre la suppression et le rollback ne sont PAS dans les fichiers.
|
||||
|
||||
**Temps de rollback :** 5-10 minutes
|
||||
|
||||
### Rollback SSE uniquement
|
||||
|
||||
**Situation :** Le SSE cause des problèmes (connexions, mémoire) mais MongoDB est stable.
|
||||
|
||||
```bash
|
||||
# Désactiver uniquement le SSE
|
||||
FEATURE_SSE_ENABLED=false
|
||||
|
||||
# Redémarrer
|
||||
docker compose restart app
|
||||
```
|
||||
|
||||
**Impact :** Le frontend bascule automatiquement en REST polling (toutes les 5 secondes). Les données MongoDB sont toujours disponibles. Pas de perte.
|
||||
|
||||
### Rollback complet (retour système fichier)
|
||||
|
||||
**Situation :** Problème majeur, tout désactiver.
|
||||
|
||||
```bash
|
||||
# Tout désactiver
|
||||
FEATURE_EXCHANGE_PERSISTENCE=false
|
||||
FEATURE_SSE_ENABLED=false
|
||||
FEATURE_EXCHANGE_API_SOURCE=file
|
||||
|
||||
# Redémarrer
|
||||
docker compose restart app
|
||||
```
|
||||
|
||||
**Impact :** Retour complet au système fichier. MongoDB n'est plus utilisé. L'UI affiche les logs comme avant la migration.
|
||||
|
||||
---
|
||||
|
||||
## Migration des données historiques (optionnel)
|
||||
|
||||
Si vous souhaitez importer les logs fichier existants dans MongoDB, utilisez le script ci-dessous. Cette étape est **optionnelle** car les anciens logs disparaîtront naturellement via le TTL 24h.
|
||||
|
||||
### Script d'import
|
||||
|
||||
```typescript
|
||||
// scripts/migrate_file_logs_to_mongodb.ts
|
||||
import { readFileSync, readdirSync } from 'node:fs'
|
||||
import { join } from 'node:path'
|
||||
import { getMongoDb } from '#providers/mongodb_provider'
|
||||
|
||||
const LOG_DIR = join(process.cwd(), 'logs', 'tr069')
|
||||
const SEPARATOR = '═'.repeat(80)
|
||||
|
||||
interface ParsedEntry {
|
||||
deviceId: string
|
||||
timestamp: string
|
||||
direction: 'CPE→ACS' | 'ACS→CPE'
|
||||
sessionId: string
|
||||
rawXml: string
|
||||
}
|
||||
|
||||
function parseLogFile(deviceId: string, content: string): ParsedEntry[] {
|
||||
const entries: ParsedEntry[] = []
|
||||
const blocks = content.split(SEPARATOR).filter((b) => b.trim())
|
||||
|
||||
for (let i = 0; i < blocks.length - 1; i += 2) {
|
||||
const header = blocks[i].trim()
|
||||
const xml = blocks[i + 1]?.trim() || ''
|
||||
|
||||
// Parse header: [2025-01-15T10:30:00.123Z] CPE→ACS | 10.0.0.1:54321->192.168.1.1:8080
|
||||
const headerMatch = header.match(
|
||||
/\[(.+?)\]\s+(CPE→ACS|ACS→CPE)\s+\|\s+(.+)/
|
||||
)
|
||||
if (!headerMatch) continue
|
||||
|
||||
entries.push({
|
||||
deviceId,
|
||||
timestamp: headerMatch[1],
|
||||
direction: headerMatch[2] as 'CPE→ACS' | 'ACS→CPE',
|
||||
sessionId: headerMatch[3],
|
||||
rawXml: xml,
|
||||
})
|
||||
}
|
||||
|
||||
return entries
|
||||
}
|
||||
|
||||
async function migrateAll(): Promise<void> {
|
||||
const db = getMongoDb()
|
||||
const collection = db.collection('cwmp_exchanges')
|
||||
const counters = db.collection('cwmp_exchange_counters')
|
||||
|
||||
const files = readdirSync(LOG_DIR).filter((f) => f.endsWith('.log'))
|
||||
let totalMigrated = 0
|
||||
|
||||
for (const file of files) {
|
||||
const deviceId = file.replace('.log', '')
|
||||
const content = readFileSync(join(LOG_DIR, file), 'utf-8')
|
||||
const entries = parseLogFile(deviceId, content)
|
||||
|
||||
if (entries.length === 0) continue
|
||||
|
||||
// Get or initialize sequence counter
|
||||
const counterResult = await counters.findOneAndUpdate(
|
||||
{ _id: deviceId as any },
|
||||
{ $inc: { seq: entries.length } },
|
||||
{ upsert: true, returnDocument: 'after' }
|
||||
)
|
||||
const startSeq = (counterResult!.seq as number) - entries.length + 1
|
||||
|
||||
// Build documents
|
||||
const docs = entries.map((entry, idx) => ({
|
||||
...entry,
|
||||
sequenceNumber: startSeq + idx,
|
||||
cwmpMethod: extractMethod(entry.rawXml),
|
||||
createdAt: new Date(entry.timestamp),
|
||||
}))
|
||||
|
||||
// Bulk insert
|
||||
await collection.insertMany(docs, { ordered: false }).catch((err) => {
|
||||
console.warn(` ⚠️ Duplicates ignored for ${deviceId}: ${err.message}`)
|
||||
})
|
||||
|
||||
totalMigrated += docs.length
|
||||
console.log(` ✅ ${deviceId}: ${docs.length} échanges migrés`)
|
||||
}
|
||||
|
||||
console.log(`\n📊 Total: ${totalMigrated} échanges migrés depuis ${files.length} fichiers`)
|
||||
}
|
||||
|
||||
function extractMethod(xml: string): string {
|
||||
if (!xml || xml.trim() === '') return 'Empty'
|
||||
const match = xml.match(/<cwmp:(\w+)[\s/>]/)
|
||||
return match ? match[1] : 'Unknown'
|
||||
}
|
||||
|
||||
migrateAll().catch(console.error)
|
||||
```
|
||||
|
||||
### Exécution
|
||||
|
||||
```bash
|
||||
# Depuis la racine du projet
|
||||
npx ts-node scripts/migrate_file_logs_to_mongodb.ts
|
||||
```
|
||||
|
||||
**Notes :**
|
||||
- Le script est idempotent : les doublons sont ignorés (grâce à l'index unique deviceId + sequenceNumber)
|
||||
- Les documents importés auront un `createdAt` basé sur le timestamp du log, donc ils seront supprimés par le TTL 24h si ils datent de plus de 24 heures
|
||||
- Pour conserver les données historiques plus longtemps, ajustez temporairement `EXCHANGE_STORE_TTL_SECONDS`
|
||||
|
||||
---
|
||||
|
||||
## Vérifications post-migration
|
||||
|
||||
### Checklist
|
||||
|
||||
| # | Vérification | Commande | Attendu |
|
||||
|---|-------------|----------|---------|
|
||||
| 1 | Collection existe | `db.cwmp_exchanges.stats()` | Collection présente |
|
||||
| 2 | Index créés | `db.cwmp_exchanges.getIndexes()` | 4 index (dont _id) |
|
||||
| 3 | TTL actif | `db.cwmp_exchanges.getIndexes()` | `expireAfterSeconds: 86400` |
|
||||
| 4 | Compteurs fonctionnent | `db.cwmp_exchange_counters.find()` | Entrées par device |
|
||||
| 5 | API accessible | `GET /api/v1/tr069-trace/:id/exchanges` | 200 avec données |
|
||||
| 6 | SSE connecte | `GET /api/v1/tr069-trace/:id/exchanges/stream` | Event stream ouvert |
|
||||
| 7 | Health check | `GET /api/v1/health/tr069-exchanges` | `status: "healthy"` |
|
||||
|
||||
### Commandes de diagnostic
|
||||
|
||||
```bash
|
||||
# Statistiques collection
|
||||
mongosh --eval "
|
||||
const stats = db.cwmp_exchanges.stats()
|
||||
print('Documents:', stats.count)
|
||||
print('Taille:', (stats.size / 1024 / 1024).toFixed(2), 'Mo')
|
||||
print('Index size:', (stats.totalIndexSize / 1024 / 1024).toFixed(2), 'Mo')
|
||||
"
|
||||
|
||||
# Vérifier le TTL monitor
|
||||
mongosh --eval "
|
||||
db.cwmp_exchanges.aggregate([
|
||||
{ \$group: { _id: null, oldest: { \$min: '\$createdAt' }, newest: { \$max: '\$createdAt' } } }
|
||||
]).forEach(doc => {
|
||||
print('Plus ancien:', doc.oldest)
|
||||
print('Plus récent:', doc.newest)
|
||||
const ageH = (Date.now() - new Date(doc.oldest).getTime()) / 3600000
|
||||
print('Âge max:', ageH.toFixed(1), 'heures (devrait être < 24h)')
|
||||
})
|
||||
"
|
||||
|
||||
# Vérifier les compteurs de séquence
|
||||
mongosh --eval "
|
||||
db.cwmp_exchange_counters.find().sort({seq: -1}).limit(5).forEach(doc => {
|
||||
print(doc._id + ': seq=' + doc.seq)
|
||||
})
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Résumé des feature flags
|
||||
|
||||
| Flag | Phase 1 | Phase 2 | Phase 3 | Rollback |
|
||||
|------|---------|---------|---------|----------|
|
||||
| `FEATURE_EXCHANGE_PERSISTENCE` | `true` | `true` | `true` | `false` |
|
||||
| `FEATURE_SSE_ENABLED` | `true` | `true` | `true` | `false` |
|
||||
| `FEATURE_EXCHANGE_API_SOURCE` | `file` | `mongodb` | `mongodb` | `file` |
|
||||
|
||||
---
|
||||
|
||||
## FAQ
|
||||
|
||||
### Que se passe-t-il si MongoDB tombe pendant le dual-write ?
|
||||
|
||||
Les écritures MongoDB échouent silencieusement (fire-and-forget). Le système fichier continue normalement. Les messages critiques sont retentés 3 fois avec backoff exponentiel (1s, 2s, 4s) avant d'être abandonnés. La session CWMP n'est jamais interrompue.
|
||||
|
||||
### Le TTL va-t-il supprimer des données pendant la migration ?
|
||||
|
||||
Oui. Le TTL de 24h s'applique dès la création de l'index. Si vous importez des logs historiques de plus de 24h, ils seront supprimés par le TTL monitor MongoDB (qui tourne ~toutes les 60 secondes). Augmentez temporairement `EXCHANGE_STORE_TTL_SECONDS` si nécessaire.
|
||||
|
||||
### Puis-je migrer les données fichier vers MongoDB ?
|
||||
|
||||
Oui, voir la section "Migration des données historiques". C'est optionnel car le système fonctionne immédiatement avec les nouveaux échanges. Les anciens logs fichier restent lisibles via `FEATURE_EXCHANGE_API_SOURCE=file`.
|
||||
|
||||
### Combien d'espace MongoDB est nécessaire ?
|
||||
|
||||
Estimation : ~1 Ko par échange (métadonnées) + taille du XML brut (typiquement 2-50 Ko). Avec le cap de 5 000 messages par device et le TTL 24h, la croissance est bornée. Pour 100 devices tracés simultanément : ~500 Mo maximum.
|
||||
|
||||
### Le dual-write a-t-il un impact sur les performances CWMP ?
|
||||
|
||||
Non. Les écritures MongoDB sont fire-and-forget (`catch(() => {})`) et ne bloquent jamais le traitement CWMP. En cas de lenteur MongoDB, le middleware continue sans attendre.
|
||||
@@ -0,0 +1,396 @@
|
||||
# Alerting Rules — TR-069 Exchange Viewer
|
||||
|
||||
## Overview
|
||||
|
||||
Ce document décrit les règles d'alerte configurées pour le système TR-069 Exchange Viewer. Chaque règle définit un seuil critique, les conditions de déclenchement, l'impact attendu et la procédure d'escalade associée.
|
||||
|
||||
Les métriques sont exposées via le endpoint `/api/v1/health/tr069-exchanges` et les logs structurés JSON.
|
||||
|
||||
---
|
||||
|
||||
## Alert 1: SSE Active Connections > 100
|
||||
|
||||
### Seuil
|
||||
|
||||
| Métrique | Condition | Sévérité |
|
||||
|----------|-----------|----------|
|
||||
| `sse.active_connections` | > 100 connexions simultanées | **Warning** |
|
||||
| `sse.active_connections` | > 200 connexions simultanées | **Critical** |
|
||||
|
||||
### Description
|
||||
|
||||
Un nombre élevé de connexions SSE simultanées indique une charge serveur importante. Chaque connexion SSE maintient un socket HTTP ouvert et un listener EventEmitter en mémoire. Au-delà de 100 connexions, le serveur risque une dégradation de performance (latence accrue, consommation mémoire excessive).
|
||||
|
||||
### Causes possibles
|
||||
|
||||
- Pic d'activité : de nombreux opérateurs tracent des devices simultanément
|
||||
- Fuite de connexions : les clients ne ferment pas correctement les connexions EventSource (navigation sans cleanup)
|
||||
- Reconnection storm : perte réseau transitoire provoquant la reconnexion simultanée de tous les clients
|
||||
- Bug client : boucle de reconnexion infinie (EventSource retry sans backoff)
|
||||
|
||||
### Impact
|
||||
|
||||
- Augmentation de la consommation mémoire du processus Node.js
|
||||
- Latence accrue sur les push SSE (chaque nouveau message est dispatché à N listeners)
|
||||
- Risque d'épuisement des file descriptors (limite OS)
|
||||
- Dégradation possible du traitement CWMP si le thread principal est saturé
|
||||
|
||||
### Détection
|
||||
|
||||
```json
|
||||
// GET /api/v1/health/tr069-exchanges
|
||||
{
|
||||
"sse": {
|
||||
"active_connections": 142,
|
||||
"reconnections_1h": 56
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Logs structurés :
|
||||
```json
|
||||
{ "level": "warn", "msg": "SSE connection count high", "active_connections": 142, "threshold": 100 }
|
||||
```
|
||||
|
||||
### Procédure de résolution
|
||||
|
||||
1. **Vérifier** le nombre de connexions actives via le health check endpoint
|
||||
2. **Identifier** les devices avec le plus de connexions (logs : `sse.connect` events par deviceId)
|
||||
3. **Vérifier** si des connexions zombies existent (connexions ouvertes sans activité > 5 minutes)
|
||||
4. **Appliquer** les timeouts serveur : les connexions inactives sont fermées après 5 min, les absolues après 2h
|
||||
5. **Si le problème persiste** : redémarrer le service pour forcer la fermeture de toutes les connexions SSE
|
||||
|
||||
### Escalade
|
||||
|
||||
| Durée | Action |
|
||||
|-------|--------|
|
||||
| 0-5 min | Monitoring automatique, pas d'action |
|
||||
| 5-15 min (Warning) | Vérifier les logs, identifier la cause |
|
||||
| > 15 min (Warning) | Notifier l'équipe Dev via Slack #adam-acs-alerts |
|
||||
| Immédiat (Critical > 200) | Notifier l'équipe Dev + Ops, envisager un restart |
|
||||
| > 30 min (Critical) | Activer le feature flag `FEATURE_SSE_ENABLED=false` (fallback REST polling) |
|
||||
|
||||
---
|
||||
|
||||
## Alert 2: MongoDB Write Latency p95 > 100ms
|
||||
|
||||
### Seuil
|
||||
|
||||
| Métrique | Condition | Sévérité |
|
||||
|----------|-----------|----------|
|
||||
| `mongodb.write_latency_p95` | > 100ms | **Warning** |
|
||||
| `mongodb.write_latency_p95` | > 500ms | **Critical** |
|
||||
| `mongodb.write_latency_p99` | > 1000ms | **Critical** |
|
||||
|
||||
### Description
|
||||
|
||||
La latence d'écriture MongoDB au percentile 95 reflète la performance de persistance des échanges CWMP. Au-delà de 100ms (p95), les écritures commencent à s'accumuler, ce qui peut provoquer un retard dans la diffusion SSE et une pression mémoire sur le processus.
|
||||
|
||||
### Causes possibles
|
||||
|
||||
- Charge MongoDB élevée (autres opérations concurrentes, compaction, journalisation)
|
||||
- Index manquant ou dégradé sur la collection `cwmp_exchanges`
|
||||
- Disque I/O saturé sur le serveur MongoDB
|
||||
- Collection `cwmp_exchanges` trop volumineuse (TTL cleanup en retard)
|
||||
- Locks MongoDB : opérations bulk ou migrations en cours
|
||||
|
||||
### Impact
|
||||
|
||||
- Retard dans la diffusion SSE des nouveaux échanges (latence end-to-end augmentée)
|
||||
- Activation des retries pour les messages critiques (backoff 1s, 2s, 4s)
|
||||
- En cas de timeout persistant : perte potentielle de messages non-critiques (Empty, Unknown)
|
||||
- Le traitement CWMP n'est jamais impacté (fire-and-forget)
|
||||
|
||||
### Détection
|
||||
|
||||
Logs structurés :
|
||||
```json
|
||||
{ "level": "warn", "msg": "MongoDB write slow", "collection": "cwmp_exchanges", "duration_ms": 145, "deviceId": "abc123", "cwmpMethod": "Inform" }
|
||||
```
|
||||
|
||||
Métriques agrégées (via logs ou Prometheus) :
|
||||
```
|
||||
mongodb_write_latency_ms{collection="cwmp_exchanges", quantile="0.95"} 145
|
||||
mongodb_write_latency_ms{collection="cwmp_exchanges", quantile="0.99"} 320
|
||||
```
|
||||
|
||||
### Procédure de résolution
|
||||
|
||||
1. **Vérifier** l'état de MongoDB : `db.serverStatus()`, `db.currentOp()`
|
||||
2. **Vérifier** les index de `cwmp_exchanges` :
|
||||
```bash
|
||||
mongosh --eval "db.cwmp_exchanges.getIndexes()"
|
||||
```
|
||||
3. **Vérifier** la taille de la collection et le nombre de documents :
|
||||
```bash
|
||||
mongosh --eval "db.cwmp_exchanges.stats()"
|
||||
```
|
||||
4. **Vérifier** le TTL cleanup (voir Alert 4)
|
||||
5. **Si les index sont manquants** : recréer via la migration ou manuellement :
|
||||
```bash
|
||||
mongosh --eval "db.cwmp_exchanges.createIndex({deviceId:1, timestamp:1})"
|
||||
mongosh --eval "db.cwmp_exchanges.createIndex({deviceId:1, sequenceNumber:1}, {unique:true})"
|
||||
mongosh --eval "db.cwmp_exchanges.createIndex({createdAt:1}, {expireAfterSeconds:86400})"
|
||||
```
|
||||
6. **Si la charge est transitoire** : attendre la stabilisation (compaction terminée, etc.)
|
||||
|
||||
### Escalade
|
||||
|
||||
| Durée | Action |
|
||||
|-------|--------|
|
||||
| 0-5 min | Monitoring automatique |
|
||||
| 5-15 min (Warning) | Vérifier les index et la charge MongoDB |
|
||||
| > 15 min (Warning) | Notifier l'équipe Dev |
|
||||
| Immédiat (Critical > 500ms) | Notifier Dev + DBA |
|
||||
| > 30 min (Critical) | Envisager `FEATURE_EXCHANGE_PERSISTENCE=false` pour délester MongoDB |
|
||||
|
||||
---
|
||||
|
||||
## Alert 3: Message Loss Rate > 0.1%
|
||||
|
||||
### Seuil
|
||||
|
||||
| Métrique | Condition | Sévérité |
|
||||
|----------|-----------|----------|
|
||||
| `exchange.loss_rate` | > 0.1% (1 message sur 1000) | **Critical** |
|
||||
| `exchange.loss_rate` | > 1.0% (1 message sur 100) | **Urgent** |
|
||||
|
||||
### Description
|
||||
|
||||
Le taux de perte de messages représente le pourcentage de messages CWMP capturés par le middleware mais non persistés en MongoDB (après épuisement des retries pour les critiques, ou discard immédiat pour les non-critiques). Un taux > 0.1% indique des défaillances critiques du backend de persistance.
|
||||
|
||||
### Causes possibles
|
||||
|
||||
- Défaillance MongoDB : connexion perdue, timeout, disque plein
|
||||
- Retry exhaustion : 3 retries échoués consécutivement pour les messages critiques
|
||||
- Erreur dans le service `ExchangeStoreService` (bug, exception non catchée)
|
||||
- Saturation de la collection (cap à 5000 messages avec suppression simultanée)
|
||||
- Problème réseau entre l'application et MongoDB
|
||||
|
||||
### Impact
|
||||
|
||||
- Trous dans l'historique des échanges visibles dans le Sequence Viewer
|
||||
- Les opérateurs voient des séquences incomplètes, rendant le debug difficile
|
||||
- Les messages critiques (Inform, SetParameterValues) perdus représentent une perte d'observabilité sur les actions CPE
|
||||
|
||||
### Détection
|
||||
|
||||
Logs structurés :
|
||||
```json
|
||||
{ "level": "error", "msg": "Exchange persistence failed after retries", "deviceId": "abc123", "cwmpMethod": "Inform", "retries": 3, "error": "MongoServerError: connection timeout" }
|
||||
{ "level": "warn", "msg": "Non-critical exchange discarded", "deviceId": "abc123", "cwmpMethod": "Empty", "error": "MongoServerError: connection timeout" }
|
||||
```
|
||||
|
||||
Calcul du taux de perte :
|
||||
```
|
||||
loss_rate = (messages_discarded / messages_captured) * 100
|
||||
```
|
||||
|
||||
### Procédure de résolution
|
||||
|
||||
1. **Vérifier** la connectivité MongoDB : health check endpoint, `mongosh` ping
|
||||
2. **Vérifier** les logs d'erreur pour identifier le pattern de défaillance
|
||||
3. **Vérifier** l'espace disque disponible sur le serveur MongoDB
|
||||
4. **Vérifier** que le TTL cleanup fonctionne (pas d'accumulation illimitée)
|
||||
5. **Si MongoDB est injoignable** : vérifier le réseau, le service mongod, les limites de connexions
|
||||
6. **Si MongoDB est surchargé** : identifier les opérations concurrentes, envisager un throttle
|
||||
|
||||
### Escalade
|
||||
|
||||
| Durée | Action |
|
||||
|-------|--------|
|
||||
| Immédiat (> 0.1%) | **Alerte critique** — Notifier l'équipe Dev immédiatement |
|
||||
| 0-5 min | Diagnostic rapide : MongoDB up? Index OK? Disk space? |
|
||||
| 5-15 min | Notifier Dev + Ops, vérifier infrastructure MongoDB |
|
||||
| > 15 min | Escalade management, envisager rollback vers fichiers |
|
||||
| Immédiat (> 1.0%) | **Alerte urgente** — Activer `FEATURE_EXCHANGE_PERSISTENCE=false`, diagnostic prioritaire |
|
||||
|
||||
---
|
||||
|
||||
## Alert 4: TTL Cleanup Lag > 1 Hour
|
||||
|
||||
### Seuil
|
||||
|
||||
| Métrique | Condition | Sévérité |
|
||||
|----------|-----------|----------|
|
||||
| `exchange.ttl_cleanup_lag` | > 1 heure | **Warning** |
|
||||
| `exchange.ttl_cleanup_lag` | > 6 heures | **Critical** |
|
||||
|
||||
### Description
|
||||
|
||||
L'index TTL sur le champ `createdAt` (expire après 86400 secondes = 24h) est géré par un thread interne MongoDB qui s'exécute toutes les 60 secondes. Si le cleanup prend du retard, des documents expirés s'accumulent, augmentant la taille de la collection et dégradant les performances de requête.
|
||||
|
||||
Le "lag" est mesuré comme la différence entre le plus ancien document présent dans la collection et son heure d'expiration théorique (createdAt + 24h).
|
||||
|
||||
### Causes possibles
|
||||
|
||||
- MongoDB TTL monitor surchargé (collection très volumineuse, beaucoup de documents à supprimer)
|
||||
- Index TTL supprimé ou corrompu
|
||||
- MongoDB en mode de maintenance (locks, compaction)
|
||||
- Charge élevée sur le serveur empêchant le thread TTL de s'exécuter
|
||||
- Bug dans la configuration du TTL (mauvais champ, mauvaise valeur `expireAfterSeconds`)
|
||||
|
||||
### Impact
|
||||
|
||||
- Croissance non contrôlée de la collection `cwmp_exchanges`
|
||||
- Dégradation progressive des performances de lecture (scans plus larges)
|
||||
- Consommation disque accrue
|
||||
- Potentiel dépassement des limites de stockage
|
||||
|
||||
### Détection
|
||||
|
||||
Vérification manuelle du lag :
|
||||
```bash
|
||||
# Trouver le document le plus ancien
|
||||
mongosh --eval "
|
||||
const oldest = db.cwmp_exchanges.find().sort({createdAt: 1}).limit(1).toArray()[0];
|
||||
if (oldest) {
|
||||
const expectedExpiry = new Date(oldest.createdAt.getTime() + 86400000);
|
||||
const lag = Date.now() - expectedExpiry.getTime();
|
||||
print('Oldest createdAt:', oldest.createdAt);
|
||||
print('Expected expiry:', expectedExpiry);
|
||||
print('Lag (ms):', Math.max(0, lag));
|
||||
print('Lag (hours):', Math.max(0, lag / 3600000).toFixed(2));
|
||||
} else {
|
||||
print('Collection empty');
|
||||
}
|
||||
"
|
||||
```
|
||||
|
||||
Vérification de l'index TTL :
|
||||
```bash
|
||||
mongosh --eval "
|
||||
const indexes = db.cwmp_exchanges.getIndexes();
|
||||
const ttlIndex = indexes.find(i => i.expireAfterSeconds !== undefined);
|
||||
if (ttlIndex) {
|
||||
print('TTL index found:', JSON.stringify(ttlIndex));
|
||||
} else {
|
||||
print('WARNING: No TTL index found!');
|
||||
}
|
||||
"
|
||||
```
|
||||
|
||||
### Procédure de résolution
|
||||
|
||||
1. **Vérifier** que l'index TTL existe et est correctement configuré :
|
||||
```bash
|
||||
mongosh --eval "db.cwmp_exchanges.getIndexes()"
|
||||
```
|
||||
L'index attendu : `{ "createdAt": 1 }` avec `expireAfterSeconds: 86400`
|
||||
|
||||
2. **Si l'index TTL est manquant** — le recréer :
|
||||
```bash
|
||||
mongosh --eval "db.cwmp_exchanges.createIndex({createdAt: 1}, {expireAfterSeconds: 86400})"
|
||||
```
|
||||
|
||||
3. **Si l'index existe mais le cleanup est en retard** :
|
||||
- Vérifier la charge MongoDB : `db.serverStatus().metrics.ttl`
|
||||
- Vérifier si un grand nombre de documents doivent être supprimés d'un coup
|
||||
- Envisager un nettoyage manuel des plus anciens documents :
|
||||
```bash
|
||||
mongosh --eval "
|
||||
const cutoff = new Date(Date.now() - 86400000);
|
||||
const result = db.cwmp_exchanges.deleteMany({createdAt: {$lt: cutoff}});
|
||||
print('Deleted:', result.deletedCount);
|
||||
"
|
||||
```
|
||||
|
||||
4. **Si MongoDB est en maintenance** : attendre la fin de l'opération, surveiller la reprise du TTL
|
||||
|
||||
### Escalade
|
||||
|
||||
| Durée | Action |
|
||||
|-------|--------|
|
||||
| 0-1h | Monitoring automatique (TTL thread cycle = 60s, léger retard normal) |
|
||||
| 1-3h (Warning) | Vérifier l'existence de l'index TTL, vérifier la charge MongoDB |
|
||||
| 3-6h (Warning) | Notifier l'équipe Dev, envisager un nettoyage manuel |
|
||||
| > 6h (Critical) | Notifier Dev + DBA, nettoyage manuel obligatoire, diagnostic approfondi |
|
||||
| > 24h | Escalade management — la collection grandit sans limite |
|
||||
|
||||
---
|
||||
|
||||
## Résumé des alertes
|
||||
|
||||
| # | Alerte | Seuil Warning | Seuil Critical | Métrique |
|
||||
|---|--------|---------------|----------------|----------|
|
||||
| 1 | SSE Active Connections | > 100 | > 200 | `sse.active_connections` |
|
||||
| 2 | MongoDB Write Latency p95 | > 100ms | > 500ms | `mongodb.write_latency_p95` |
|
||||
| 3 | Message Loss Rate | > 0.1% | > 1.0% | `exchange.loss_rate` |
|
||||
| 4 | TTL Cleanup Lag | > 1 heure | > 6 heures | `exchange.ttl_cleanup_lag` |
|
||||
|
||||
---
|
||||
|
||||
## Configuration des alertes
|
||||
|
||||
### Variables d'environnement
|
||||
|
||||
| Variable | Défaut | Description |
|
||||
|----------|--------|-------------|
|
||||
| `ALERT_SSE_CONNECTIONS_WARNING` | `100` | Seuil warning connexions SSE |
|
||||
| `ALERT_SSE_CONNECTIONS_CRITICAL` | `200` | Seuil critical connexions SSE |
|
||||
| `ALERT_MONGO_LATENCY_P95_WARNING_MS` | `100` | Seuil warning latence MongoDB p95 (ms) |
|
||||
| `ALERT_MONGO_LATENCY_P95_CRITICAL_MS` | `500` | Seuil critical latence MongoDB p95 (ms) |
|
||||
| `ALERT_LOSS_RATE_CRITICAL_PCT` | `0.1` | Seuil critical taux de perte (%) |
|
||||
| `ALERT_TTL_LAG_WARNING_HOURS` | `1` | Seuil warning lag TTL (heures) |
|
||||
| `ALERT_TTL_LAG_CRITICAL_HOURS` | `6` | Seuil critical lag TTL (heures) |
|
||||
|
||||
### Intégration avec les systèmes d'alerte
|
||||
|
||||
Les alertes sont déclenchées via les logs structurés JSON, compatibles avec :
|
||||
|
||||
- **ELK Stack** : index pattern `adam-acs-*`, filtres sur `level: "error"` ou `level: "warn"` + tags spécifiques
|
||||
- **Datadog** : log-based monitors avec conditions sur les champs JSON
|
||||
- **Prometheus/Grafana** : si le endpoint `/metrics` est exposé, utiliser les alerting rules Prometheus
|
||||
|
||||
Exemple de règle Prometheus :
|
||||
```yaml
|
||||
groups:
|
||||
- name: tr069-exchange-viewer
|
||||
rules:
|
||||
- alert: HighSSEConnections
|
||||
expr: adam_acs_sse_active_connections > 100
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "High SSE connection count"
|
||||
description: "{{ $value }} SSE connections active (threshold: 100)"
|
||||
|
||||
- alert: MongoWriteLatencyHigh
|
||||
expr: histogram_quantile(0.95, adam_acs_mongodb_write_duration_seconds_bucket{collection="cwmp_exchanges"}) > 0.1
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "MongoDB write latency p95 > 100ms"
|
||||
description: "p95 latency: {{ $value }}s for cwmp_exchanges writes"
|
||||
|
||||
- alert: ExchangeMessageLoss
|
||||
expr: rate(adam_acs_exchange_messages_lost_total[5m]) / rate(adam_acs_exchange_messages_captured_total[5m]) > 0.001
|
||||
for: 1m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "Message loss rate exceeds 0.1%"
|
||||
description: "{{ $value | humanizePercentage }} messages lost in last 5 minutes"
|
||||
|
||||
- alert: TTLCleanupLag
|
||||
expr: adam_acs_exchange_ttl_lag_seconds > 3600
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "TTL cleanup lag exceeds 1 hour"
|
||||
description: "Oldest expired document is {{ $value | humanizeDuration }} past its TTL"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Contacts d'escalade
|
||||
|
||||
| Niveau | Contact | Canal |
|
||||
|--------|---------|-------|
|
||||
| L1 — Warning | Équipe Dev adam-acs | Slack #adam-acs-alerts |
|
||||
| L2 — Critical | Lead Dev + Ops | Slack #adam-acs-alerts + mention @oncall |
|
||||
| L3 — Urgent/Prolongé | Management technique | Escalade téléphonique |
|
||||
| DBA (MongoDB) | Équipe DBA | Slack #dba-support |
|
||||
@@ -0,0 +1,180 @@
|
||||
---
|
||||
name: Precision Protocol System
|
||||
colors:
|
||||
surface: '#f8f9ff'
|
||||
surface-dim: '#cbdbf5'
|
||||
surface-bright: '#f8f9ff'
|
||||
surface-container-lowest: '#ffffff'
|
||||
surface-container-low: '#eff4ff'
|
||||
surface-container: '#e5eeff'
|
||||
surface-container-high: '#dce9ff'
|
||||
surface-container-highest: '#d3e4fe'
|
||||
on-surface: '#0b1c30'
|
||||
on-surface-variant: '#45464d'
|
||||
inverse-surface: '#213145'
|
||||
inverse-on-surface: '#eaf1ff'
|
||||
outline: '#76777d'
|
||||
outline-variant: '#c6c6cd'
|
||||
surface-tint: '#565e74'
|
||||
primary: '#000000'
|
||||
on-primary: '#ffffff'
|
||||
primary-container: '#131b2e'
|
||||
on-primary-container: '#7c839b'
|
||||
inverse-primary: '#bec6e0'
|
||||
secondary: '#006a61'
|
||||
on-secondary: '#ffffff'
|
||||
secondary-container: '#86f2e4'
|
||||
on-secondary-container: '#006f66'
|
||||
tertiary: '#000000'
|
||||
on-tertiary: '#ffffff'
|
||||
tertiary-container: '#0f0069'
|
||||
on-tertiary-container: '#7671ff'
|
||||
error: '#ba1a1a'
|
||||
on-error: '#ffffff'
|
||||
error-container: '#ffdad6'
|
||||
on-error-container: '#93000a'
|
||||
primary-fixed: '#dae2fd'
|
||||
primary-fixed-dim: '#bec6e0'
|
||||
on-primary-fixed: '#131b2e'
|
||||
on-primary-fixed-variant: '#3f465c'
|
||||
secondary-fixed: '#89f5e7'
|
||||
secondary-fixed-dim: '#6bd8cb'
|
||||
on-secondary-fixed: '#00201d'
|
||||
on-secondary-fixed-variant: '#005049'
|
||||
tertiary-fixed: '#e2dfff'
|
||||
tertiary-fixed-dim: '#c3c0ff'
|
||||
on-tertiary-fixed: '#0f0069'
|
||||
on-tertiary-fixed-variant: '#3323cc'
|
||||
background: '#f8f9ff'
|
||||
on-background: '#0b1c30'
|
||||
surface-variant: '#d3e4fe'
|
||||
typography:
|
||||
headline-lg:
|
||||
fontFamily: Inter
|
||||
fontSize: 30px
|
||||
fontWeight: '600'
|
||||
lineHeight: 38px
|
||||
letterSpacing: -0.02em
|
||||
headline-md:
|
||||
fontFamily: Inter
|
||||
fontSize: 24px
|
||||
fontWeight: '600'
|
||||
lineHeight: 32px
|
||||
letterSpacing: -0.01em
|
||||
headline-sm:
|
||||
fontFamily: Inter
|
||||
fontSize: 18px
|
||||
fontWeight: '600'
|
||||
lineHeight: 26px
|
||||
body-lg:
|
||||
fontFamily: Inter
|
||||
fontSize: 16px
|
||||
fontWeight: '400'
|
||||
lineHeight: 24px
|
||||
body-md:
|
||||
fontFamily: Inter
|
||||
fontSize: 14px
|
||||
fontWeight: '400'
|
||||
lineHeight: 20px
|
||||
body-sm:
|
||||
fontFamily: Inter
|
||||
fontSize: 13px
|
||||
fontWeight: '400'
|
||||
lineHeight: 18px
|
||||
label-md:
|
||||
fontFamily: Inter
|
||||
fontSize: 12px
|
||||
fontWeight: '500'
|
||||
lineHeight: 16px
|
||||
letterSpacing: 0.05em
|
||||
mono-md:
|
||||
fontFamily: JetBrains Mono
|
||||
fontSize: 13px
|
||||
fontWeight: '400'
|
||||
lineHeight: 20px
|
||||
mono-sm:
|
||||
fontFamily: JetBrains Mono
|
||||
fontSize: 11px
|
||||
fontWeight: '400'
|
||||
lineHeight: 16px
|
||||
rounded:
|
||||
sm: 0.125rem
|
||||
DEFAULT: 0.25rem
|
||||
md: 0.375rem
|
||||
lg: 0.5rem
|
||||
xl: 0.75rem
|
||||
full: 9999px
|
||||
spacing:
|
||||
unit: 4px
|
||||
container-padding: 24px
|
||||
gutter: 16px
|
||||
sidebar-width: 260px
|
||||
stack-sm: 8px
|
||||
stack-md: 16px
|
||||
stack-lg: 32px
|
||||
---
|
||||
|
||||
## Brand & Style
|
||||
|
||||
This design system is engineered for high-performance network analysis and protocol monitoring. The brand personality is rooted in **precision, neutrality, and clarity**. It avoids decorative flourishes to prioritize the rapid interpretation of complex data structures.
|
||||
|
||||
The visual style is a blend of **Minimalism** and **Modern Corporate** aesthetics. It utilizes a spacious layout to reduce the cognitive load inherent in dense data environments. The interface conveys reliability through a systematic application of subtle borders and high-contrast typography, ensuring that critical network events are immediately visible.
|
||||
|
||||
## Colors
|
||||
|
||||
The palette is anchored in **Slate and Zinc** neutrals to provide a professional, calm environment for long-term monitoring.
|
||||
|
||||
* **Primary (Slate 950):** Used for core navigation, high-level headers, and text to ensure maximum contrast.
|
||||
* **Neutral (Slate/Zinc):** Used for structural elements, secondary text, and inactive states.
|
||||
* **Flow Accents:** Semantic colors are reserved for directional data. **Teal (ACS -> CPE)** and **Indigo (CPE -> ACS)** allow engineers to visually parse the direction of TR-069 traffic at a glance without reading text labels.
|
||||
* **Success/Error:** Traditional Green and Red are used sparingly for status indicators and connection states.
|
||||
|
||||
## Typography
|
||||
|
||||
This design system utilizes a dual-font strategy. **Inter** is the workhorse for the UI, chosen for its exceptional legibility in small sizes and high-density layouts. **JetBrains Mono** is utilized exclusively for technical data, including XML, JSON payloads, and packet hex dumps.
|
||||
|
||||
Headlines use tighter letter spacing for a more authoritative look, while labels utilize uppercase styling and increased tracking for clear categorization in sidebars and table headers. Monospaced text is rendered with a slightly smaller font size than body text to accommodate long lines of code without excessive wrapping.
|
||||
|
||||
## Layout & Spacing
|
||||
|
||||
The layout follows a **Hybrid Grid** model. Navigation and property inspectors use fixed-width sidebars, while the main content area—typically containing packet logs or traffic graphs—uses a **Fluid Grid** to maximize horizontal real estate.
|
||||
|
||||
### Breakpoints
|
||||
* **Desktop (1280px+):** 12-column fluid grid, 24px margins.
|
||||
* **Tablet (768px - 1279px):** 8-column fluid grid, 16px margins. Sidebar collapses to icons.
|
||||
* **Mobile (<767px):** 4-column fluid grid, 12px margins. Horizontal scrolling enabled for data tables.
|
||||
|
||||
Spacing is based on a **4px base unit**. Component internals use 8px or 12px increments to maintain a compact, "instrumental" feel without appearing cluttered.
|
||||
|
||||
## Elevation & Depth
|
||||
|
||||
To maintain a technical and minimalist aesthetic, this design system avoids heavy shadows. Instead, it relies on **Tonal Layers** and **Low-Contrast Outlines**.
|
||||
|
||||
* **Background (Level 0):** Slate-50. The base canvas.
|
||||
* **Surface (Level 1):** White. Primary cards and panels.
|
||||
* **Active/Overlay (Level 2):** White with a subtle 1px border in Slate-200 and a very soft, high-diffusion shadow (4px blur, 2% opacity) to indicate temporary elements like tooltips or dropdowns.
|
||||
|
||||
Visual separation between adjacent panels is achieved via 1px solid borders in Slate-200 rather than elevation, ensuring the UI remains crisp and "flat."
|
||||
|
||||
## Shapes
|
||||
|
||||
The shape language is **Soft (Level 1)**. A 4px radius is applied to buttons, input fields, and panels. This subtle rounding provides a modern touch without sacrificing the professional, rigid structure required for a technical tool.
|
||||
|
||||
Status indicators (pills) use a slightly larger 12px radius to distinguish them from interactive buttons, while data visualization bars use sharp 2px corners for maximum precision.
|
||||
|
||||
## Components
|
||||
|
||||
### Buttons
|
||||
Primary buttons use the Slate-950 background with white text. Ghost buttons (transparent background with Slate-200 borders) are preferred for secondary actions to keep the focus on the data.
|
||||
|
||||
### Packet Log Entries
|
||||
Logs must use a zebra-stripe pattern (White / Slate-50) for horizontal tracking. The "Flow Direction" is indicated by a 4px vertical accent bar on the far left of the row (Teal or Indigo).
|
||||
|
||||
### Input Fields
|
||||
Fields use a 1px Slate-200 border. On focus, the border transitions to Slate-900. Labels are always positioned above the input in the `label-md` style for clarity.
|
||||
|
||||
### Data Displays (Code Blocks)
|
||||
XML/JSON payloads are wrapped in containers with a Slate-50 background and a 1px border. They use `mono-md` typography with syntax highlighting that mirrors the flow colors (Teal for tags, Indigo for values).
|
||||
|
||||
### Chips & Badges
|
||||
Used for protocol methods (e.g., `Inform`, `GetParameterValues`). These are small, low-contrast components (Slate-100 background) to avoid competing with the flow direction accents.
|
||||
@@ -0,0 +1,367 @@
|
||||
<!DOCTYPE html>
|
||||
|
||||
<html class="light" lang="en"><head>
|
||||
<meta charset="utf-8"/>
|
||||
<meta content="width=device-width, initial-scale=1.0" name="viewport"/>
|
||||
<title>TR-069 Protocol Analyzer</title>
|
||||
<script src="https://cdn.tailwindcss.com?plugins=forms,container-queries"></script>
|
||||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&family=JetBrains+Mono:wght@400;500&family=Material+Symbols+Outlined:wght,FILL@100..700,0..1&display=swap" rel="stylesheet"/>
|
||||
<link href="https://fonts.googleapis.com/css2?family=Material+Symbols+Outlined:wght,FILL@100..700,0..1&display=swap" rel="stylesheet"/>
|
||||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@100..900&display=swap" rel="stylesheet"/>
|
||||
<style>
|
||||
.material-symbols-outlined {
|
||||
font-variation-settings: 'FILL' 0, 'wght' 400, 'GRAD' 0, 'opsz' 24;
|
||||
vertical-align: middle;
|
||||
}
|
||||
/* Custom scrollbar for technical look */
|
||||
::-webkit-scrollbar { width: 6px; height: 6px; }
|
||||
::-webkit-scrollbar-track { background: transparent; }
|
||||
::-webkit-scrollbar-thumb { background: #e2e8f0; border-radius: 10px; }
|
||||
::-webkit-scrollbar-thumb:hover { background: #94a3b8; }
|
||||
|
||||
.lifeline-gradient {
|
||||
background: linear-gradient(to bottom, #000 0%, #000 5%, transparent 5%, transparent 100%);
|
||||
background-size: 1px 20px;
|
||||
}
|
||||
</style>
|
||||
<script id="tailwind-config">
|
||||
tailwind.config = {
|
||||
darkMode: "class",
|
||||
theme: {
|
||||
extend: {
|
||||
"colors": {
|
||||
"tertiary-fixed-dim": "#c3c0ff",
|
||||
"surface-container-high": "#dce9ff",
|
||||
"surface-container-lowest": "#ffffff",
|
||||
"on-background": "#0b1c30",
|
||||
"on-tertiary": "#ffffff",
|
||||
"error": "#ba1a1a",
|
||||
"on-primary-fixed": "#131b2e",
|
||||
"surface-tint": "#565e74",
|
||||
"secondary-fixed-dim": "#6bd8cb",
|
||||
"secondary-container": "#86f2e4",
|
||||
"surface-container": "#e5eeff",
|
||||
"on-primary-fixed-variant": "#3f465c",
|
||||
"outline-variant": "#c6c6cd",
|
||||
"on-secondary-fixed": "#00201d",
|
||||
"on-surface-variant": "#45464d",
|
||||
"surface-bright": "#f8f9ff",
|
||||
"on-primary": "#ffffff",
|
||||
"tertiary": "#000000",
|
||||
"primary-fixed-dim": "#bec6e0",
|
||||
"on-tertiary-container": "#7671ff",
|
||||
"surface-container-low": "#eff4ff",
|
||||
"surface-container-highest": "#d3e4fe",
|
||||
"surface-variant": "#d3e4fe",
|
||||
"inverse-on-surface": "#eaf1ff",
|
||||
"secondary": "#006a61",
|
||||
"primary-fixed": "#dae2fd",
|
||||
"primary": "#000000",
|
||||
"tertiary-container": "#0f0069",
|
||||
"on-error-container": "#93000a",
|
||||
"on-tertiary-fixed-variant": "#3323cc",
|
||||
"on-surface": "#0b1c30",
|
||||
"on-error": "#ffffff",
|
||||
"on-secondary": "#ffffff",
|
||||
"surface-dim": "#cbdbf5",
|
||||
"error-container": "#ffdad6",
|
||||
"tertiary-fixed": "#e2dfff",
|
||||
"secondary-fixed": "#89f5e7",
|
||||
"background": "#f8f9ff",
|
||||
"primary-container": "#131b2e",
|
||||
"on-secondary-fixed-variant": "#005049",
|
||||
"on-tertiary-fixed": "#0f0069",
|
||||
"on-secondary-container": "#006f66",
|
||||
"surface": "#f8f9ff",
|
||||
"inverse-surface": "#213145",
|
||||
"inverse-primary": "#bec6e0",
|
||||
"outline": "#76777d"
|
||||
},
|
||||
"borderRadius": {
|
||||
"DEFAULT": "0.125rem",
|
||||
"lg": "0.25rem",
|
||||
"xl": "0.5rem",
|
||||
"full": "0.75rem"
|
||||
},
|
||||
"spacing": {
|
||||
"gutter": "16px",
|
||||
"stack-md": "16px",
|
||||
"stack-sm": "8px",
|
||||
"unit": "4px",
|
||||
"stack-lg": "32px",
|
||||
"sidebar-width": "260px",
|
||||
"container-padding": "24px"
|
||||
},
|
||||
"fontFamily": {
|
||||
"mono-sm": ["JetBrains Mono"],
|
||||
"headline-md": ["Inter"],
|
||||
"mono-md": ["JetBrains Mono"],
|
||||
"headline-lg": ["Inter"],
|
||||
"body-md": ["Inter"],
|
||||
"body-sm": ["Inter"],
|
||||
"body-lg": ["Inter"],
|
||||
"label-md": ["Inter"],
|
||||
"headline-sm": ["Inter"]
|
||||
},
|
||||
"fontSize": {
|
||||
"mono-sm": ["11px", { "lineHeight": "16px", "fontWeight": "400" }],
|
||||
"headline-md": ["24px", { "lineHeight": "32px", "letterSpacing": "-0.01em", "fontWeight": "600" }],
|
||||
"mono-md": ["13px", { "lineHeight": "20px", "fontWeight": "400" }],
|
||||
"headline-lg": ["30px", { "lineHeight": "38px", "letterSpacing": "-0.02em", "fontWeight": "600" }],
|
||||
"body-md": ["14px", { "lineHeight": "20px", "fontWeight": "400" }],
|
||||
"body-sm": ["13px", { "lineHeight": "18px", "fontWeight": "400" }],
|
||||
"body-lg": ["16px", { "lineHeight": "24px", "fontWeight": "400" }],
|
||||
"label-md": ["12px", { "lineHeight": "16px", "letterSpacing": "0.05em", "fontWeight": "500" }],
|
||||
"headline-sm": ["18px", { "lineHeight": "26px", "fontWeight": "600" }]
|
||||
}
|
||||
},
|
||||
},
|
||||
}
|
||||
</script>
|
||||
</head>
|
||||
<body class="bg-background text-on-background font-body-md overflow-hidden">
|
||||
<!-- Top Navigation Shell -->
|
||||
<header class="bg-surface text-on-surface font-headline-sm text-headline-sm docked full-width top-0 border-b border-outline-variant flex justify-between items-center px-gutter w-full h-16 z-50">
|
||||
<div class="flex items-center gap-stack-lg">
|
||||
<span class="font-headline-sm text-headline-sm font-bold text-on-surface">TR-069 Protocol Analyzer</span>
|
||||
<nav class="hidden md:flex items-center h-16 gap-6">
|
||||
<a class="text-on-surface-variant font-label-md text-label-md h-full flex items-center px-2 hover:bg-surface-container transition-colors duration-200" href="#">Dashboard</a>
|
||||
<a class="text-on-surface-variant font-label-md text-label-md h-full flex items-center px-2 hover:bg-surface-container transition-colors duration-200" href="#">Live Stream</a>
|
||||
<a class="text-primary font-bold border-b-2 border-primary h-full flex items-center px-2" href="#">Analysis</a>
|
||||
<a class="text-on-surface-variant font-label-md text-label-md h-full flex items-center px-2 hover:bg-surface-container transition-colors duration-200" href="#">Config</a>
|
||||
</nav>
|
||||
</div>
|
||||
<div class="flex items-center gap-4">
|
||||
<div class="relative group">
|
||||
<span class="material-symbols-outlined cursor-pointer p-2 hover:bg-surface-container-high rounded-full transition-colors duration-150" data-icon="filter_list">filter_list</span>
|
||||
</div>
|
||||
<span class="material-symbols-outlined cursor-pointer p-2 hover:bg-surface-container-high rounded-full transition-colors duration-150" data-icon="settings">settings</span>
|
||||
<span class="material-symbols-outlined cursor-pointer p-2 hover:bg-surface-container-high rounded-full transition-colors duration-150" data-icon="help">help</span>
|
||||
</div>
|
||||
</header>
|
||||
<div class="flex h-[calc(100vh-64px)] overflow-hidden">
|
||||
<!-- Side Navigation Shell -->
|
||||
<aside class="bg-surface-container-low text-primary font-label-md text-label-md docked left-0 h-full w-sidebar-width border-r border-outline-variant flex flex-col sticky top-0">
|
||||
<div class="p-6">
|
||||
<div class="flex items-center gap-3 mb-8">
|
||||
<div class="w-10 h-10 rounded-full bg-primary-container flex items-center justify-center">
|
||||
<span class="material-symbols-outlined text-surface" data-icon="router">router</span>
|
||||
</div>
|
||||
<div>
|
||||
<div class="font-headline-sm text-[14px] font-bold text-primary">Network Monitor</div>
|
||||
<div class="text-[11px] text-on-surface-variant">Active Nodes: 1,240</div>
|
||||
</div>
|
||||
</div>
|
||||
<button class="w-full bg-primary text-on-primary py-2.5 rounded-lg font-label-md text-label-md mb-8 hover:opacity-90 transition-opacity">
|
||||
New Capture
|
||||
</button>
|
||||
<nav class="space-y-1">
|
||||
<a class="flex items-center gap-3 px-4 py-3 bg-secondary-container text-on-secondary-container font-semibold rounded-lg transition-all duration-150" href="#">
|
||||
<span class="material-symbols-outlined" data-icon="history">history</span>
|
||||
<span>Sessions</span>
|
||||
</a>
|
||||
<a class="flex items-center gap-3 px-4 py-3 text-on-surface-variant hover:bg-surface-container-high rounded-lg transition-all duration-150" href="#">
|
||||
<span class="material-symbols-outlined" data-icon="router">router</span>
|
||||
<span>Devices</span>
|
||||
</a>
|
||||
<a class="flex items-center gap-3 px-4 py-3 text-on-surface-variant hover:bg-surface-container-high rounded-lg transition-all duration-150" href="#">
|
||||
<span class="material-symbols-outlined" data-icon="analytics">analytics</span>
|
||||
<span>Diagnostics</span>
|
||||
</a>
|
||||
<a class="flex items-center gap-3 px-4 py-3 text-on-surface-variant hover:bg-surface-container-high rounded-lg transition-all duration-150" href="#">
|
||||
<span class="material-symbols-outlined" data-icon="list_alt">list_alt</span>
|
||||
<span>Logs</span>
|
||||
</a>
|
||||
</nav>
|
||||
</div>
|
||||
<div class="mt-auto p-6 border-t border-outline-variant">
|
||||
<a class="flex items-center gap-3 px-4 py-3 text-on-surface-variant hover:bg-surface-container-high rounded-lg transition-all duration-150" href="#">
|
||||
<span class="material-symbols-outlined" data-icon="logout">logout</span>
|
||||
<span>Logout</span>
|
||||
</a>
|
||||
</div>
|
||||
</aside>
|
||||
<!-- Main Content Area -->
|
||||
<main class="flex-1 flex flex-col overflow-hidden bg-white">
|
||||
<!-- Session Status Header -->
|
||||
<div class="flex items-center justify-between px-8 py-4 bg-surface-container-low border-b border-outline-variant">
|
||||
<div class="flex gap-8">
|
||||
<div>
|
||||
<span class="text-label-md font-label-md text-on-surface-variant block uppercase tracking-wider mb-1">Device Serial</span>
|
||||
<span class="text-body-md font-mono-md font-medium text-on-surface">TR069-NODE-550291-B</span>
|
||||
</div>
|
||||
<div class="h-10 w-px bg-outline-variant"></div>
|
||||
<div>
|
||||
<span class="text-label-md font-label-md text-on-surface-variant block uppercase tracking-wider mb-1">IP Address</span>
|
||||
<span class="text-body-md font-mono-md font-medium text-on-surface">192.168.100.42</span>
|
||||
</div>
|
||||
<div class="h-10 w-px bg-outline-variant"></div>
|
||||
<div>
|
||||
<span class="text-label-md font-label-md text-on-surface-variant block uppercase tracking-wider mb-1">Connection Status</span>
|
||||
<div class="flex items-center gap-2">
|
||||
<span class="w-2 h-2 rounded-full bg-secondary"></span>
|
||||
<span class="text-body-md font-medium text-on-surface">CWMP Session Established</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="flex items-center gap-2">
|
||||
<span class="px-3 py-1 bg-surface-container text-on-surface-variant rounded-full text-label-md font-label-md border border-outline-variant">Session ID: 88241</span>
|
||||
<button class="material-symbols-outlined text-on-surface-variant p-1" data-icon="more_vert">more_vert</button>
|
||||
</div>
|
||||
</div>
|
||||
<!-- Sequence Diagram Content -->
|
||||
<div class="flex-1 flex overflow-hidden">
|
||||
<div class="flex-1 overflow-y-auto relative bg-slate-50 p-12">
|
||||
<!-- Lifeline Headers -->
|
||||
<div class="sticky top-0 z-10 flex justify-around mb-12 pointer-events-none">
|
||||
<div class="px-10 py-3 bg-primary text-on-primary rounded font-bold shadow-sm border border-primary min-w-[120px] text-center">CPE</div>
|
||||
<div class="px-10 py-3 bg-primary text-on-primary rounded font-bold shadow-sm border border-primary min-w-[120px] text-center">ACS</div>
|
||||
</div>
|
||||
<!-- Vertical Lifelines -->
|
||||
<div class="absolute top-0 bottom-0 left-1/4 w-px bg-outline pointer-events-none opacity-20"></div>
|
||||
<div class="absolute top-0 bottom-0 right-1/4 w-px bg-outline pointer-events-none opacity-20"></div>
|
||||
<!-- Messages -->
|
||||
<div class="relative space-y-10 pb-20">
|
||||
<!-- Row: Inform -->
|
||||
<div class="relative group cursor-pointer" onclick="selectMessage('inform')">
|
||||
<div class="absolute left-[25%] right-[25%] flex items-center">
|
||||
<div class="flex-1 h-px bg-secondary relative">
|
||||
<div class="absolute right-0 -top-[5px] border-y-[6px] border-y-transparent border-l-[10px] border-l-secondary"></div>
|
||||
</div>
|
||||
<div class="absolute left-1/2 -translate-x-1/2 -top-6 text-center">
|
||||
<span class="px-2 py-0.5 bg-secondary-container text-on-secondary-container text-label-md font-label-md rounded border border-secondary/20">Inform</span>
|
||||
<span class="block text-[10px] font-mono-sm text-on-surface-variant mt-1">10:05:22.450</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="h-8 invisible">spacer</div>
|
||||
</div>
|
||||
<!-- Row: InformResponse -->
|
||||
<div class="relative group cursor-pointer" onclick="selectMessage('inform-response')">
|
||||
<div class="absolute left-[25%] right-[25%] flex items-center">
|
||||
<div class="flex-1 h-px bg-tertiary-container relative">
|
||||
<div class="absolute left-0 -top-[5px] border-y-[6px] border-y-transparent border-r-[10px] border-r-tertiary-container"></div>
|
||||
</div>
|
||||
<div class="absolute left-1/2 -translate-x-1/2 -top-6 text-center">
|
||||
<span class="px-2 py-0.5 bg-surface-container-high text-on-tertiary-fixed-variant text-label-md font-label-md rounded border border-outline">InformResponse</span>
|
||||
<span class="block text-[10px] font-mono-sm text-on-surface-variant mt-1">10:05:22.612</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="h-8 invisible">spacer</div>
|
||||
</div>
|
||||
<!-- Selected Row: GetParameterValues -->
|
||||
<div class="relative group cursor-pointer" onclick="selectMessage('get-params')">
|
||||
<!-- Selection Highlight -->
|
||||
<div class="absolute -left-12 -right-12 h-16 bg-surface-container border-y border-outline-variant -mt-4 opacity-50 z-0"></div>
|
||||
<div class="absolute left-[25%] right-[25%] flex items-center z-10">
|
||||
<div class="flex-1 h-[2px] bg-primary relative">
|
||||
<div class="absolute left-0 -top-[6px] border-y-[7px] border-y-transparent border-r-[12px] border-r-primary"></div>
|
||||
</div>
|
||||
<div class="absolute left-1/2 -translate-x-1/2 -top-6 text-center">
|
||||
<span class="px-2 py-0.5 bg-primary text-on-primary text-label-md font-label-md rounded shadow-sm">GetParameterValues</span>
|
||||
<span class="block text-[10px] font-mono-sm font-bold text-primary mt-1">10:05:23.002</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="h-8 invisible">spacer</div>
|
||||
</div>
|
||||
<!-- Row: GetParameterValuesResponse -->
|
||||
<div class="relative group cursor-pointer" onclick="selectMessage('get-params-response')">
|
||||
<div class="absolute left-[25%] right-[25%] flex items-center">
|
||||
<div class="flex-1 h-px bg-secondary relative">
|
||||
<div class="absolute right-0 -top-[5px] border-y-[6px] border-y-transparent border-l-[10px] border-l-secondary"></div>
|
||||
</div>
|
||||
<div class="absolute left-1/2 -translate-x-1/2 -top-6 text-center">
|
||||
<span class="px-2 py-0.5 bg-secondary-container text-on-secondary-container text-label-md font-label-md rounded border border-secondary/20">GetParameterValuesResponse</span>
|
||||
<span class="block text-[10px] font-mono-sm text-on-surface-variant mt-1">10:05:23.189</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="h-8 invisible">spacer</div>
|
||||
</div>
|
||||
<!-- Row: SetParameterValues -->
|
||||
<div class="relative group cursor-pointer">
|
||||
<div class="absolute left-[25%] right-[25%] flex items-center">
|
||||
<div class="flex-1 h-px bg-tertiary-container relative">
|
||||
<div class="absolute left-0 -top-[5px] border-y-[6px] border-y-transparent border-r-[10px] border-r-tertiary-container"></div>
|
||||
</div>
|
||||
<div class="absolute left-1/2 -translate-x-1/2 -top-6 text-center">
|
||||
<span class="px-2 py-0.5 bg-surface-container-high text-on-tertiary-fixed-variant text-label-md font-label-md rounded border border-outline">SetParameterValues</span>
|
||||
<span class="block text-[10px] font-mono-sm text-on-surface-variant mt-1">10:05:23.441</span>
|
||||
</div>
|
||||
</div>
|
||||
<div class="h-8 invisible">spacer</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<!-- Inspector Panel (Right) -->
|
||||
<div class="w-[450px] bg-white border-l border-outline-variant flex flex-col shadow-xl z-20">
|
||||
<div class="p-4 border-b border-outline-variant bg-surface flex justify-between items-center">
|
||||
<div class="flex items-center gap-2">
|
||||
<span class="material-symbols-outlined text-on-surface-variant" data-icon="description">description</span>
|
||||
<span class="font-headline-sm text-[16px] font-bold text-on-surface">Message Detail</span>
|
||||
</div>
|
||||
<div class="flex gap-2">
|
||||
<button class="material-symbols-outlined text-on-surface-variant hover:text-primary transition-colors" data-icon="content_copy">content_copy</button>
|
||||
<button class="material-symbols-outlined text-on-surface-variant hover:text-primary transition-colors" data-icon="close">close</button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="flex-1 overflow-auto bg-surface-container-low p-4">
|
||||
<div class="flex items-center gap-2 mb-4">
|
||||
<span class="text-label-md font-label-md bg-primary-container text-on-primary-container px-2 py-1 rounded">CWMP:GetParameterValues</span>
|
||||
<span class="text-mono-sm text-on-surface-variant">ID: 1024_A</span>
|
||||
</div>
|
||||
<!-- Code Block Display -->
|
||||
<div class="rounded-lg border border-outline-variant bg-white p-4 font-mono-md text-mono-md leading-relaxed shadow-inner overflow-hidden">
|
||||
<pre class="whitespace-pre-wrap"><span class="text-secondary"><soap:Envelope</span> <span class="text-tertiary-fixed-variant">xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"</span><span class="text-secondary">></span>
|
||||
<span class="text-secondary"><soap:Header></span>
|
||||
<span class="text-secondary"><cwmp:ID</span> <span class="text-tertiary-fixed-variant">soap:mustUnderstand="1"</span><span class="text-secondary">></span>1024_A<span class="text-secondary"></cwmp:ID></span>
|
||||
<span class="text-secondary"></soap:Header></span>
|
||||
<span class="text-secondary"><soap:Body></span>
|
||||
<span class="text-secondary"><cwmp:GetParameterValues></span>
|
||||
<span class="text-secondary"><ParameterNames</span> <span class="text-tertiary-fixed-variant">soap:arrayType="xsd:string[3]"</span><span class="text-secondary">></span>
|
||||
<span class="text-secondary"><string></span>InternetGatewayDevice.DeviceInfo.HardwareVersion<span class="text-secondary"></string></span>
|
||||
<span class="text-secondary"><string></span>InternetGatewayDevice.DeviceInfo.SoftwareVersion<span class="text-secondary"></string></span>
|
||||
<span class="text-secondary"><string></span>InternetGatewayDevice.ManagementServer.ConnectionRequestURL<span class="text-secondary"></string></span>
|
||||
<span class="text-secondary"></ParameterNames></span>
|
||||
<span class="text-secondary"></cwmp:GetParameterValues></span>
|
||||
<span class="text-secondary"></soap:Body></span>
|
||||
<span class="text-secondary"></soap:Envelope></span></pre>
|
||||
</div>
|
||||
<div class="mt-6 space-y-4">
|
||||
<h4 class="text-label-md font-label-md text-on-surface-variant uppercase tracking-wider">Analysis Summary</h4>
|
||||
<div class="p-3 bg-secondary-container/20 border-l-4 border-secondary rounded-r">
|
||||
<p class="text-body-sm text-on-surface">ACS is requesting device hardware and software versioning information to determine compatible firmware upgrade paths.</p>
|
||||
</div>
|
||||
<div class="grid grid-cols-2 gap-4">
|
||||
<div class="bg-white p-3 border border-outline-variant rounded">
|
||||
<span class="text-label-md font-label-md text-on-surface-variant block mb-1">MTU Size</span>
|
||||
<span class="text-body-md font-medium">1500 Bytes</span>
|
||||
</div>
|
||||
<div class="bg-white p-3 border border-outline-variant rounded">
|
||||
<span class="text-label-md font-label-md text-on-surface-variant block mb-1">Latency</span>
|
||||
<span class="text-body-md font-medium text-secondary">42ms</span>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</main>
|
||||
</div>
|
||||
<!-- Micro-interaction script -->
|
||||
<script>
|
||||
function selectMessage(id) {
|
||||
// Placeholder logic for visual feedback in a prototype
|
||||
console.log('Selected message context:', id);
|
||||
}
|
||||
|
||||
// Add subtle hover tracking for interactive feel
|
||||
document.querySelectorAll('.group').forEach(item => {
|
||||
item.addEventListener('mouseenter', () => {
|
||||
item.style.transform = 'translateX(2px)';
|
||||
item.style.transition = 'transform 0.1s ease-out';
|
||||
});
|
||||
item.addEventListener('mouseleave', () => {
|
||||
item.style.transform = 'translateX(0)';
|
||||
});
|
||||
});
|
||||
</script>
|
||||
</body></html>
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 255 KiB |
@@ -0,0 +1,486 @@
|
||||
# Troubleshooting Guide: TR-069 Exchange Viewer
|
||||
|
||||
Ce guide documente les problèmes courants du système de visualisation des échanges TR-069/CWMP, leurs causes et solutions.
|
||||
|
||||
## Table des matières
|
||||
|
||||
1. [Déconnexions SSE](#1-déconnexions-sse)
|
||||
2. [Timeouts MongoDB et récupération](#2-timeouts-mongodb-et-récupération)
|
||||
3. [Diagnostic de perte de messages](#3-diagnostic-de-perte-de-messages)
|
||||
4. [Vérification des index et du TTL](#4-vérification-des-index-et-du-ttl)
|
||||
5. [Problèmes frontend (SequenceViewer)](#5-problèmes-frontend-sequenceviewer)
|
||||
6. [Feature flags et rollback](#6-feature-flags-et-rollback)
|
||||
|
||||
---
|
||||
|
||||
## 1. Déconnexions SSE
|
||||
|
||||
### 1.1 Causes courantes
|
||||
|
||||
| Cause | Symptôme | Fréquence |
|
||||
|-------|----------|-----------|
|
||||
| **Timeout d'inactivité (5 min)** | Déconnexion après 5 minutes sans nouvel échange | Normal — par design |
|
||||
| **Timeout absolu (2 h)** | Déconnexion après 2 heures de connexion continue | Normal — par design |
|
||||
| **Proxy/reverse-proxy buffering** | Messages SSE arrivent en burst au lieu de temps réel | Courant avec nginx |
|
||||
| **Perte réseau** | Indicateur "disconnected" dans le viewer | Variable |
|
||||
| **SSE désactivé (feature flag)** | Erreur 503 à la connexion SSE | Intentionnel |
|
||||
| **Device non tracé** | Erreur 404 à la connexion SSE | Erreur utilisateur |
|
||||
|
||||
### 1.2 Solutions
|
||||
|
||||
#### Timeout d'inactivité / absolu
|
||||
|
||||
Le système est conçu pour fermer les connexions inactives. Le client se reconnecte automatiquement via le mécanisme `Last-Event-ID` — aucun message n'est perdu.
|
||||
|
||||
Pour ajuster les timeouts :
|
||||
|
||||
```bash
|
||||
# Augmenter le timeout d'inactivité (défaut: 300000ms = 5 min)
|
||||
EXCHANGE_STORE_SSE_INACTIVITY_TIMEOUT_MS=600000
|
||||
|
||||
# Augmenter le timeout absolu (défaut: 7200000ms = 2h)
|
||||
EXCHANGE_STORE_SSE_ABSOLUTE_TIMEOUT_MS=14400000
|
||||
```
|
||||
|
||||
#### Proxy/reverse-proxy buffering
|
||||
|
||||
Si un reverse-proxy (nginx, HAProxy) est en place, il peut bufferiser les réponses SSE. Le header `X-Accel-Buffering: no` est déjà envoyé par le serveur, mais vérifiez aussi la configuration nginx :
|
||||
|
||||
```nginx
|
||||
location /api/v1/tr069-trace/ {
|
||||
proxy_buffering off;
|
||||
proxy_cache off;
|
||||
proxy_set_header Connection '';
|
||||
proxy_http_version 1.1;
|
||||
chunked_transfer_encoding off;
|
||||
}
|
||||
```
|
||||
|
||||
#### Client qui ne se reconnecte pas
|
||||
|
||||
Le composable `useExchangeStream` utilise l'`EventSource` natif avec retry automatique (3 secondes par défaut). Si la reconnexion ne fonctionne pas :
|
||||
|
||||
1. Vérifiez que le device est toujours en trace active
|
||||
2. Vérifiez les logs serveur pour des erreurs 404 ou 503
|
||||
3. Vérifiez la valeur `retry` dans la configuration SSE :
|
||||
|
||||
```bash
|
||||
# Intervalle de reconnexion envoyé au client (défaut: 3000ms)
|
||||
EXCHANGE_STORE_SSE_RETRY_INTERVAL_MS=3000
|
||||
```
|
||||
|
||||
### 1.3 Diagnostic dans les logs
|
||||
|
||||
Cherchez les entrées `[SSE]` dans les logs applicatifs :
|
||||
|
||||
```bash
|
||||
docker compose logs app | grep '\[SSE\]'
|
||||
```
|
||||
|
||||
Messages attendus :
|
||||
- `[SSE] Connection closed { deviceId: "...", reason: "inactivity" }` — timeout normal
|
||||
- `[SSE] Connection closed { deviceId: "...", reason: "absolute" }` — timeout absolu normal
|
||||
- `[SSE] Connection closed { deviceId: "...", reason: "client_disconnect" }` — client a fermé
|
||||
- `[SSE] Error fetching missed exchanges for reconnection` — problème MongoDB lors du rattrapage
|
||||
|
||||
---
|
||||
|
||||
## 2. Timeouts MongoDB et récupération
|
||||
|
||||
### 2.1 Scénarios de timeout
|
||||
|
||||
| Scénario | Impact | Comportement |
|
||||
|----------|--------|--------------|
|
||||
| **Write timeout (persistance)** | Message potentiellement perdu | Retry (critique) ou discard (non-critique) |
|
||||
| **Read timeout (API >5s)** | Résultats partiels retournés | `isTruncated: true` dans la réponse |
|
||||
| **Connection lost** | Toutes les opérations échouent | Les sessions CWMP continuent sans log |
|
||||
|
||||
### 2.2 Write timeout — Retry automatique
|
||||
|
||||
Le `ExchangeStoreService` gère les échecs d'écriture selon la criticité du message :
|
||||
|
||||
**Messages critiques** (Inform, GetParameterValues, SetParameterValuesResponse, Fault, etc.) :
|
||||
- 3 tentatives avec backoff exponentiel : 1s, 2s, 4s
|
||||
- Si toutes échouent : log error + discard (session CWMP non impactée)
|
||||
|
||||
**Messages non-critiques** (Empty, Unknown) :
|
||||
- Aucun retry — discard immédiat + log warning
|
||||
|
||||
Vérifier les échecs de retry :
|
||||
|
||||
```bash
|
||||
docker compose logs app | grep '\[ExchangeStore\] All retries exhausted'
|
||||
docker compose logs app | grep '\[ExchangeStore\] Critical message persistence failed'
|
||||
```
|
||||
|
||||
Pour ajuster la politique de retry :
|
||||
|
||||
```bash
|
||||
# Nombre max de tentatives (défaut: 3)
|
||||
EXCHANGE_STORE_RETRY_MAX_ATTEMPTS=3
|
||||
|
||||
# Délai de base en ms pour le backoff (défaut: 1000)
|
||||
# Résultat : 1000ms, 2000ms, 4000ms
|
||||
EXCHANGE_STORE_RETRY_BASE_DELAY_MS=1000
|
||||
```
|
||||
|
||||
### 2.3 Read timeout — Résultats partiels
|
||||
|
||||
Si une requête API dépasse 5 secondes, le contrôleur retourne les résultats partiels avec un flag :
|
||||
|
||||
```json
|
||||
{
|
||||
"exchanges": [...],
|
||||
"metadata": {
|
||||
"isTruncated": true,
|
||||
"total": 4500
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Causes courantes :**
|
||||
- Index manquant sur `(deviceId, timestamp)` — voir section 4
|
||||
- Trop de documents pour le device (proche du cap 5000)
|
||||
- Charge MongoDB élevée (vérifier les métriques)
|
||||
|
||||
Pour ajuster le timeout de lecture :
|
||||
|
||||
```bash
|
||||
# Timeout en ms (défaut: 5000)
|
||||
EXCHANGE_STORE_READ_TIMEOUT_MS=5000
|
||||
```
|
||||
|
||||
### 2.4 Vérifier la connectivité MongoDB
|
||||
|
||||
```bash
|
||||
# Se connecter au shell MongoDB
|
||||
docker compose exec mongodb mongosh
|
||||
|
||||
# Vérifier le statut
|
||||
db.adminCommand({ serverStatus: 1 }).connections
|
||||
|
||||
# Vérifier la latence
|
||||
db.adminCommand({ ping: 1 })
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Diagnostic de perte de messages
|
||||
|
||||
### 3.1 Arbre de décision
|
||||
|
||||
```
|
||||
Message manquant dans le viewer ?
|
||||
│
|
||||
├─ Le device est-il en trace ?
|
||||
│ └─ Non → Activer la trace d'abord. Les messages ne sont PAS persistés hors trace.
|
||||
│
|
||||
├─ Le feature flag FEATURE_EXCHANGE_PERSISTENCE est-il à true ?
|
||||
│ └─ Non → Activer la persistance : FEATURE_EXCHANGE_PERSISTENCE=true
|
||||
│
|
||||
├─ Le message apparaît-il dans MongoDB mais pas dans le viewer ?
|
||||
│ └─ Oui → Problème SSE (voir section 1) ou problème frontend (voir section 5)
|
||||
│
|
||||
├─ Le message n'apparaît PAS dans MongoDB ?
|
||||
│ ├─ Vérifier les logs de retry : grep "All retries exhausted"
|
||||
│ ├─ Vérifier le cap (5000 messages) : les anciens messages sont supprimés
|
||||
│ └─ Vérifier le TTL (24h) : les messages >24h sont auto-supprimés
|
||||
│
|
||||
└─ Message tronqué (XML incomplet) ?
|
||||
└─ Le rawXml dépassait 2MB → grep "rawXml exceeds 2MB" dans les logs
|
||||
```
|
||||
|
||||
### 3.2 Vérifier les messages en base
|
||||
|
||||
```bash
|
||||
# Se connecter au shell MongoDB
|
||||
docker compose exec mongodb mongosh
|
||||
|
||||
# Compter les messages pour un device
|
||||
use adam_acs
|
||||
db.cwmp_exchanges.countDocuments({ deviceId: "<DEVICE_ID>" })
|
||||
|
||||
# Voir les derniers messages
|
||||
db.cwmp_exchanges.find({ deviceId: "<DEVICE_ID>" }).sort({ timestamp: -1 }).limit(5).pretty()
|
||||
|
||||
# Voir la séquence max (dernier message)
|
||||
db.cwmp_exchange_counters.findOne({ _id: "<DEVICE_ID>" })
|
||||
```
|
||||
|
||||
### 3.3 Vérifier les logs de persistance
|
||||
|
||||
```bash
|
||||
# Chercher les erreurs de persistance
|
||||
docker compose logs app | grep '\[ExchangeStore\]'
|
||||
|
||||
# Messages spécifiques
|
||||
docker compose logs app | grep 'Failed to persist exchange'
|
||||
docker compose logs app | grep 'Non-critical message persistence failed'
|
||||
docker compose logs app | grep 'All retries exhausted'
|
||||
docker compose logs app | grep 'Message cap enforced'
|
||||
docker compose logs app | grep 'rawXml exceeds 2MB'
|
||||
```
|
||||
|
||||
### 3.4 Message supprimé par le cap
|
||||
|
||||
Le système maintient un maximum de 5000 messages par device. Quand ce cap est dépassé, les messages les plus anciens sont supprimés.
|
||||
|
||||
```bash
|
||||
# Vérifier si le cap a été atteint
|
||||
docker compose logs app | grep 'Message cap enforced'
|
||||
```
|
||||
|
||||
Pour ajuster le cap :
|
||||
|
||||
```bash
|
||||
# Nombre max de messages par device (défaut: 5000)
|
||||
EXCHANGE_STORE_MAX_MESSAGES=5000
|
||||
```
|
||||
|
||||
### 3.5 Message supprimé par le TTL
|
||||
|
||||
MongoDB supprime automatiquement les documents dont `createdAt` dépasse 24 heures (via l'index TTL).
|
||||
|
||||
```bash
|
||||
# Vérifier les documents les plus anciens
|
||||
docker compose exec mongodb mongosh --eval "
|
||||
use adam_acs;
|
||||
db.cwmp_exchanges.find({}, { createdAt: 1, deviceId: 1 }).sort({ createdAt: 1 }).limit(3).pretty()
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Vérification des index et du TTL
|
||||
|
||||
### 4.1 Lister les index existants
|
||||
|
||||
```bash
|
||||
docker compose exec mongodb mongosh --eval "
|
||||
use adam_acs;
|
||||
db.cwmp_exchanges.getIndexes().forEach(idx => printjson(idx))
|
||||
"
|
||||
```
|
||||
|
||||
**Index attendus :**
|
||||
|
||||
| Nom | Champs | Options |
|
||||
|-----|--------|---------|
|
||||
| `_id_` | `{ _id: 1 }` | — |
|
||||
| `idx_deviceId_timestamp` | `{ deviceId: 1, timestamp: 1 }` | — |
|
||||
| `idx_deviceId_sequenceNumber` | `{ deviceId: 1, sequenceNumber: 1 }` | `unique: true` |
|
||||
| `idx_createdAt_ttl` | `{ createdAt: 1 }` | `expireAfterSeconds: 86400` |
|
||||
|
||||
### 4.2 Vérifier l'index TTL
|
||||
|
||||
```bash
|
||||
docker compose exec mongodb mongosh --eval "
|
||||
use adam_acs;
|
||||
const indexes = db.cwmp_exchanges.getIndexes();
|
||||
const ttlIdx = indexes.find(i => i.name === 'idx_createdAt_ttl');
|
||||
if (ttlIdx) {
|
||||
print('TTL index OK — expireAfterSeconds:', ttlIdx.expireAfterSeconds);
|
||||
} else {
|
||||
print('ERREUR: TTL index manquant !');
|
||||
}
|
||||
"
|
||||
```
|
||||
|
||||
### 4.3 Recréer les index manuellement (si manquants)
|
||||
|
||||
Si des index sont manquants après un incident :
|
||||
|
||||
```bash
|
||||
docker compose exec mongodb mongosh --eval "
|
||||
use adam_acs;
|
||||
|
||||
// Index composite (deviceId + timestamp)
|
||||
db.cwmp_exchanges.createIndex(
|
||||
{ deviceId: 1, timestamp: 1 },
|
||||
{ name: 'idx_deviceId_timestamp' }
|
||||
);
|
||||
|
||||
// Index unique (deviceId + sequenceNumber)
|
||||
db.cwmp_exchanges.createIndex(
|
||||
{ deviceId: 1, sequenceNumber: 1 },
|
||||
{ unique: true, name: 'idx_deviceId_sequenceNumber' }
|
||||
);
|
||||
|
||||
// Index TTL (24h)
|
||||
db.cwmp_exchanges.createIndex(
|
||||
{ createdAt: 1 },
|
||||
{ expireAfterSeconds: 86400, name: 'idx_createdAt_ttl' }
|
||||
);
|
||||
|
||||
print('Tous les index recréés.');
|
||||
"
|
||||
```
|
||||
|
||||
### 4.4 Vérifier le statut du TTL monitor
|
||||
|
||||
Le TTL monitor de MongoDB tourne par défaut toutes les 60 secondes. Pour vérifier qu'il fonctionne :
|
||||
|
||||
```bash
|
||||
docker compose exec mongodb mongosh --eval "
|
||||
const status = db.adminCommand({ serverStatus: 1 });
|
||||
print('TTL deletedDocuments:', status.metrics.ttl.deletedDocuments);
|
||||
print('TTL passes:', status.metrics.ttl.passes);
|
||||
"
|
||||
```
|
||||
|
||||
Si `passes` n'augmente pas, le TTL monitor pourrait être bloqué (rare, nécessite un restart MongoDB).
|
||||
|
||||
### 4.5 Modifier le TTL
|
||||
|
||||
Pour changer la durée du TTL sans recréer l'index :
|
||||
|
||||
```bash
|
||||
docker compose exec mongodb mongosh --eval "
|
||||
use adam_acs;
|
||||
db.runCommand({
|
||||
collMod: 'cwmp_exchanges',
|
||||
index: {
|
||||
name: 'idx_createdAt_ttl',
|
||||
expireAfterSeconds: 172800 // 48h
|
||||
}
|
||||
});
|
||||
"
|
||||
```
|
||||
|
||||
N'oubliez pas de mettre à jour la variable d'environnement pour cohérence :
|
||||
|
||||
```bash
|
||||
EXCHANGE_STORE_TTL_SECONDS=172800
|
||||
```
|
||||
|
||||
### 4.6 Vérifier la collection counters
|
||||
|
||||
La collection `cwmp_exchange_counters` stocke les séquences atomiques par device :
|
||||
|
||||
```bash
|
||||
docker compose exec mongodb mongosh --eval "
|
||||
use adam_acs;
|
||||
db.cwmp_exchange_counters.find().forEach(doc => {
|
||||
print(doc._id, '→ seq:', doc.seq);
|
||||
})
|
||||
"
|
||||
```
|
||||
|
||||
Si un compteur est incohérent avec les données réelles :
|
||||
|
||||
```bash
|
||||
# Recalculer le max sequenceNumber pour un device
|
||||
docker compose exec mongodb mongosh --eval "
|
||||
use adam_acs;
|
||||
const deviceId = '<DEVICE_ID>';
|
||||
const maxSeq = db.cwmp_exchanges.find({ deviceId }).sort({ sequenceNumber: -1 }).limit(1).toArray();
|
||||
if (maxSeq.length > 0) {
|
||||
db.cwmp_exchange_counters.updateOne(
|
||||
{ _id: deviceId },
|
||||
{ \$set: { seq: maxSeq[0].sequenceNumber } }
|
||||
);
|
||||
print('Counter reset to', maxSeq[0].sequenceNumber);
|
||||
}
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Problèmes frontend (SequenceViewer)
|
||||
|
||||
### 5.1 Le viewer ne se connecte pas au SSE
|
||||
|
||||
**Causes possibles :**
|
||||
1. Le device n'est pas en trace → activer la trace d'abord
|
||||
2. Le feature flag `FEATURE_SSE_ENABLED` est à `false` → erreur 503
|
||||
3. Problème CORS si accès depuis un domaine différent
|
||||
|
||||
**Diagnostic :** Ouvrir la console navigateur (F12) et chercher les erreurs EventSource.
|
||||
|
||||
### 5.2 Les messages ne s'affichent pas en temps réel
|
||||
|
||||
Si les messages arrivent dans MongoDB mais pas dans le viewer :
|
||||
1. Vérifier l'indicateur de connexion : doit être "connected" (vert)
|
||||
2. Si "disconnected" (rouge) : voir section 1
|
||||
3. Si "paused" (jaune) : l'auto-scroll est désactivé car l'utilisateur a scrollé vers le haut — cliquer "Resume auto-scroll"
|
||||
|
||||
### 5.3 Performance lente avec beaucoup de messages
|
||||
|
||||
Le viewer utilise du virtual scrolling pour >200 messages. Si la performance se dégrade :
|
||||
- Le cap client est fixé à 1000 messages en mémoire (les plus anciens sont purgés côté client)
|
||||
- Si le nombre total dépasse 5000 en base, les plus anciens sont purgés côté serveur
|
||||
- Utiliser "Clear exchanges" pour repartir d'un état vide
|
||||
|
||||
### 5.4 L'inspecteur XML affiche du contenu brut (non formaté)
|
||||
|
||||
Si le XML n'est pas mis en forme :
|
||||
- Le XML est peut-être malformé → l'inspecteur affiche le contenu brut en fallback
|
||||
- Vérifier si le message a un `rawXml` vide (cas "Empty")
|
||||
|
||||
---
|
||||
|
||||
## 6. Feature flags et rollback
|
||||
|
||||
### 6.1 Flags disponibles
|
||||
|
||||
| Variable d'environnement | Défaut | Effet si `false`/`file` |
|
||||
|--------------------------|--------|-------------------------|
|
||||
| `FEATURE_EXCHANGE_PERSISTENCE` | `true` | Les échanges ne sont pas écrits en MongoDB |
|
||||
| `FEATURE_SSE_ENABLED` | `true` | L'endpoint SSE retourne 503, le client utilise le REST polling (5s) |
|
||||
| `FEATURE_EXCHANGE_API_SOURCE` | `mongodb` | Si `file`, l'API lit depuis l'ancien système de fichiers |
|
||||
|
||||
### 6.2 Procédure de rollback rapide
|
||||
|
||||
En cas de problème critique :
|
||||
|
||||
```bash
|
||||
# 1. Désactiver la persistance (les sessions CWMP continuent normalement)
|
||||
FEATURE_EXCHANGE_PERSISTENCE=false
|
||||
|
||||
# 2. Désactiver le SSE (le frontend utilise le polling REST)
|
||||
FEATURE_SSE_ENABLED=false
|
||||
|
||||
# 3. Revenir à la source fichier pour l'API
|
||||
FEATURE_EXCHANGE_API_SOURCE=file
|
||||
```
|
||||
|
||||
Redémarrer l'application après modification :
|
||||
|
||||
```bash
|
||||
docker compose restart app
|
||||
```
|
||||
|
||||
### 6.3 Vérifier l'état actuel des flags
|
||||
|
||||
Les valeurs actives sont loguées au démarrage de l'application. Vous pouvez aussi interroger le health check :
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:8080/api/v1/health/tr069-exchanges | python3 -m json.tool
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Commandes utiles — Résumé rapide
|
||||
|
||||
```bash
|
||||
# Status général
|
||||
docker compose logs app --tail 20 | grep -E '\[ExchangeStore\]|\[SSE\]'
|
||||
|
||||
# Compter les échanges d'un device
|
||||
docker compose exec mongodb mongosh --eval "use adam_acs; db.cwmp_exchanges.countDocuments({deviceId:'<ID>'})"
|
||||
|
||||
# Vérifier les index
|
||||
docker compose exec mongodb mongosh --eval "use adam_acs; db.cwmp_exchanges.getIndexes().forEach(i=>printjson(i))"
|
||||
|
||||
# Vérifier le TTL monitor
|
||||
docker compose exec mongodb mongosh --eval "printjson(db.adminCommand({serverStatus:1}).metrics.ttl)"
|
||||
|
||||
# Tester la connexion SSE manuellement
|
||||
curl -N -H "Accept: text/event-stream" http://localhost:8080/api/v1/tr069-trace/<DEVICE_ID>/exchanges/stream
|
||||
|
||||
# Tester la reconnexion avec Last-Event-ID
|
||||
curl -N -H "Accept: text/event-stream" -H "Last-Event-ID: 42" http://localhost:8080/api/v1/tr069-trace/<DEVICE_ID>/exchanges/stream
|
||||
|
||||
# Purger les échanges d'un device
|
||||
curl -X DELETE http://localhost:8080/api/v1/tr069-trace/<DEVICE_ID>/exchanges
|
||||
```
|
||||
Reference in new issue
Block a user