Skip to content

Architecture

Component topology, data model, deployment patterns, and technology choices for Monoscope.

Component Topology

flowchart TB
    subgraph Clients["Client Applications"]
        Apps["Your Apps\n(OTel SDKs)"]
        Browser["Browser\n(Session Replay SDK)"]
    end

    subgraph Ingestion["Ingestion Layer"]
        OTel["OTel Collector\n(gRPC :4317)"]
        API["Monoscope API\n(Haskell)"]
        Kafka["Kafka Buffer"]
    end

    subgraph Processing["Processing Layer"]
        Worker["Extraction Worker"]
        Agent["AI Agent\nScheduler"]
        LLM["LLM API"]
    end

    subgraph Storage["Storage Layer"]
        TF["TimeFusion\n(Rust + DataFusion)"]
        PG["PostgreSQL\n+ TimescaleDB (pg18)"]
        S3["S3 Bucket\n(Delta Lake / Parquet)"]
        Cache["Foyer Cache\n(512MB mem + 100GB disk)"]
    end

    subgraph UI["Presentation Layer"]
        Web["Web Dashboard\n(HTMX + Tailwind)"]
        Alerts["Alert Channels\n(Slack, Discord, PagerDuty)"]
    end

    Apps -->|"OTLP/gRPC"| OTel
    Browser -->|"Session events"| API
    OTel -->|"Bearer token"| API
    API --> Kafka --> Worker
    Worker --> TF
    Worker --> PG
    TF --> S3
    TF --> Cache
    Agent -->|"Query"| TF
    Agent --> LLM
    Agent --> Alerts
    Web -->|"SQL via pgwire"| TF
    Web --> PG

    style Storage fill:#2e7d32,color:#fff
    style Ingestion fill:#1565c0,color:#fff
    style Processing fill:#e65100,color:#fff

Technology Breakdown

Component Language Framework/Library Purpose
Monoscope Backend Haskell (80.5%) Hasql, Lucid, HTMX, Eff API, ingestion, processing, web UI
TimeFusion Rust DataFusion, pgwire, Delta Lake, Foyer Time-series query engine with S3 storage
Metadata DB PLpgSQL (2.4%) PostgreSQL + TimescaleDB (pg18) Project config, alerts, user management
Frontend TypeScript (11.7%) HTMX, Tailwind v4, DaisyUI v5, ECharts Server-rendered UI with dynamic updates
SDKs Multi-language OTel SDK wrappers Application instrumentation
Migrations PLpgSQL 87KB of SQL migrations Schema evolution

Haskell Backend Internals

  • hasql-interpolate for type-safe PostgreSQL queries (migrated from postgresql-simple in v0.5.0)
  • Lucid for HTML templating (server-rendered)
  • HTMX for dynamic page updates with morphing
  • Eff effect system for IO abstraction
  • Effectful.Time for time operations
  • Fourmolu for code formatting
  • GHC 9.12 compatible

Data Model

Telemetry Storage (TimeFusion / S3)

erDiagram
    PROJECT ||--o{ OTEL_EVENTS : "contains"
    OTEL_EVENTS {
        uuid id PK
        uuid project_id FK
        timestamptz timestamp
        date date_partition
        text name
        bigint duration_ns
        text kind
        text[] hashes
        text attributes
    }
    TRACE {
        uuid trace_id
        uuid span_id
        uuid parent_span_id
        text service_name
        text operation_name
    }
    LOG_ENTRY {
        uuid id PK
        timestamptz timestamp
        text severity
        text body
        text attributes
    }
    METRIC {
        text metric_name
        text metric_type
        float value
        text labels
    }

Metadata Storage (PostgreSQL + TimescaleDB)

  • Projects — tenant isolation, API keys, retention settings
  • Monitors — alerting rules, health checks, renotify intervals
  • Alerting state — active incidents, notification history
  • Users/Teams — authentication, authorization, audit logs
  • AI Agent configs — schedules, LLM prompts, report recipients

Deployment Topologies

Docker Compose (Development / Small Production)

flowchart LR
    subgraph Host["Docker Host"]
        M["monoscope\n:8080"]
        TF["timefusion\n:5432"]
        PG["postgres+timescaledb\n:5433"]
        K["kafka\n:9092"]
        S3["localstack/minio\nS3-compatible"]
    end

    M --> TF --> S3
    M --> PG
    M --> K

Self-Hosted Production

flowchart TB
    subgraph LB["Load Balancer"]
        Nginx["NGINX\n(TLS Termination)"]
    end

    subgraph K8s["Kubernetes Cluster"]
        subgraph Monoscope["Monoscope Pods"]
            M1["monoscope-api-1"]
            M2["monoscope-api-2"]
        end

        subgraph Workers["Background Workers"]
            W1["extraction-worker"]
            W2["ai-agent-scheduler"]
        end

        OTelCol["OTel Collector"]
    end

    subgraph Data["External Data"]
        S3Prod["AWS S3 / MinIO"]
        PGHA["PostgreSQL HA\n(Patroni / RDS)"]
        TFProd["TimeFusion\n(Deployed separately)"]
        KProd["Kafka Cluster"]
    end

    Nginx --> M1
    Nginx --> M2
    M1 --> S3Prod
    M1 --> PGHA
    M1 --> TFProd
    Workers --> S3Prod
    OTelCol --> M1

    style K8s fill:#1565c0,color:#fff
    style Data fill:#2e7d32,color:#fff

Monoscope Cloud (SaaS)

flowchart LR
    subgraph Cloud["Monoscope Cloud"]
        MC["Managed Monoscope\n+ TimeFusion"]
        MCS3["Monoscope S3"]
    end

    subgraph BYOS["Your S3 Bucket\n(optional)"]
        US3["Your S3\n(unlimited retention)"]
    end

    Apps["Your Apps"] -->|"OTLP"| Cloud
    MC --> MCS3
    MC -->|"BYOS mode"| US3

Sources

How It Works

How Monoscope ingests telemetry via OTLP, stores it in S3 through TimeFusion, and provides LLM-powered querying with AI agent scheduling.

Ingestion Pipeline

Monoscope uses OpenTelemetry Protocol (OTLP) as its sole ingestion path:

flowchart LR
    subgraph Apps["Your Applications"]
        SDK1["Go SDK"]
        SDK2["Python SDK"]
        SDK3["Node.js SDK"]
        SDK4["Java Agent"]
    end

    subgraph Collector["OTel Collector"]
        OLTP["OTLP Receiver\n(gRPC :4317)"]
    end

    subgraph Monoscope["Monoscope Backend"]
        API["Ingestion API\n(Haskell)"]
        Kafka["Kafka\n(Buffer)"]
        Worker["Extraction Worker"]
    end

    subgraph Storage["Data Layer"]
        TF["TimeFusion\n(Rust + DataFusion)"]
        PG["PostgreSQL\n+ TimescaleDB"]
        S3["S3 Bucket\n(Delta Lake)"]
    end

    Apps -->|"OTLP"| Collector
    Collector -->|"OTLP/gRPC\nBearer API_KEY"| API
    API --> Kafka --> Worker
    Worker --> TF --> S3
    Worker --> PG

OTLP Ingestion

All telemetry arrives via OTLP over gRPC on port 4317 with Bearer token authentication:

  • Logs — structured and unstructured log events
  • Traces — spans with parent-child relationships, duration, attributes
  • Metrics — Sum, Histogram, ExponentialHistogram, Summary types

The ingestion API normalizes all data into a unified otel_logs_and_spans table schema before passing to TimeFusion.

TimeFusion Storage Engine

TimeFusion is Monoscope's purpose-built time-series database (separate open-source project at monoscope-tech/timefusion):

flowchart TB
    subgraph TF["TimeFusion Engine (Rust)"]
        PGWire["PostgreSQL Wire Protocol\n(pgwire)"]
        DF["Apache DataFusion\n(Query Engine)"]
        Cache["Two-Tier Cache\n(Foyer)"]
        Mem["Memory Cache\n512MB default"]
        Disk["Disk Cache\n100GB default"]
        DL["Delta Lake\n(ACID Transactions)"]
    end

    subgraph S3["S3-Compatible Storage"]
        PQ["Parquet Files\n(Zstd compressed)"]
    end

    PGWire --> DF
    DF --> Cache
    Cache --> Mem
    Cache --> Disk
    DF --> DL --> PQ

    style TF fill:#1565c0,color:#fff
    style S3 fill:#2e7d32,color:#fff

Key Properties

Property Detail
Wire protocol PostgreSQL-compatible via pgwire — any Postgres client can query
Query engine Apache DataFusion with vectorized execution
Storage format Delta Lake with Parquet files on S3
Compression Zstandard (10-20x reduction)
Throughput 500K+ events/sec per instance
ACID Delta Lake transactions for consistency
Caching Foyer adaptive: 512MB memory + 100GB disk, 7-day TTL, 95%+ hit rate
Distributed DynamoDB-based locking for multi-instance deployments

Main Table Schema

The otel_logs_and_spans table stores all telemetry in a unified schema:

Column Type Purpose
name text Span/log name (for example, HTTP endpoint path)
id uuid Unique identifier
project_id uuid Tenant/project isolation
timestamp timestamptz Event timestamp
date date Partition key
hashes text[] Trace lookup hashes
duration bigint Span duration in nanoseconds
attributes___http___response___status_code text Flattened OTel attributes (triple underscore separator)
attributes___user___id text User identity propagation
attributes___error___type text Error classification
kind text Span kind (SERVER, CLIENT, INTERNAL, and more)

Natural Language Query Engine

Monoscope integrates LLMs to translate plain-English queries into SQL executed against TimeFusion:

  1. User input — "Show me all 500 errors from the payments service yesterday"
  2. LLM translation — converts to a parameterized SQL query targeting otel_logs_and_spans
  3. Query execution — TimeFusion executes with vectorized DataFusion engine
  4. Result visualization — charts, log tables, and trace waterfalls rendered in the UI

AI Agent Scheduler

Scheduled agents run LLM-powered analysis on telemetry data:

flowchart LR
    Scheduler["Agent Scheduler\n(Haskell)"]
    LLM["LLM API"]
    Data["TimeFusion\nQuery"]
    Detect["Anomaly Detection"]
    Report["Email Report"]
    Alert["Alert Channels"]

    Scheduler -->|"Query + Analyze"| Data
    Data --> LLM
    LLM --> Detect
    Detect -->|"Anomaly found"| Report
    Detect -->|"Critical"| Alert
  • Configurable intervals: hourly, daily, weekly
  • Anomaly detection: volume spikes, error rate changes, latency degradation
  • Email reports: summary of findings delivered to configured recipients
  • Alerting: critical findings routed to Slack, Discord, PagerDuty, or webhooks

Error Fingerprinting

Monoscope uses a two-tier fingerprinting system:

  1. Jaccard similarity — groups errors with similar stack traces using set-based comparison
  2. Embedding-based merging — semantically similar errors are merged even with different text
  3. Framework-error rollup — known framework errors (for example, Django Http404, Express ECONNREFUSED) are automatically categorized

Session Replay

Browser session recordings synced with backend telemetry:

  1. Browser SDK captures DOM mutations, user interactions, and network requests
  2. Events are batched and sent to Monoscope's ingestion API
  3. Session merging worker combines replay events with backend spans using correlation IDs
  4. Merged sessions are stored in S3 and viewable in the UI alongside traces and logs

Sources

Security

Identity and access control, data protection, network security, and threat model considerations for Monoscope (the open-source observability platform with BYOS storage).

Identity & Access

RBAC (Role-Based Access Control)

Monoscope provides project-level RBAC. Each project acts as a tenant boundary with its own API key, S3 partitioning, and access controls.

Role Permissions
Owner Full project control: manage members, configure alerts, modify retention, delete data
Admin Manage dashboards, monitors, alert channels. Cannot delete the project or manage billing
Member View dashboards, query telemetry, acknowledge alerts. Cannot modify project settings
Viewer Read-only access to dashboards and queries. No alert management

Roles are assigned at the project level. A user can hold different roles across different projects within the same workspace.

Multi-Tenancy

Tenant isolation is enforced at the project level:

  • Each project has a unique API key used for OTLP ingestion authentication
  • TimeFusion partitions telemetry data by project_id in the S3 bucket
  • PostgreSQL metadata tables are scoped by project ID
  • Cross-project data access is not possible through the query engine

Workspace-level multi-tenancy

Multi-tenant workspace support (organizational accounts with centralized user management across projects) is on the roadmap but not yet available in v0.5.0. Currently, each project manages its own member list.

API Authentication

All API and ingestion endpoints require Bearer token authentication:

  • OTLP ingestion: Authorization: Bearer <PROJECT_API_KEY> header on gRPC calls to port 4317
  • Web UI: Session-based authentication (email + password) with CSRF protection via HTMX morphing
  • TimeFusion queries: PostgreSQL wire protocol on port 5432. Access is restricted by network-level controls (no built-in user authentication in TimeFusion itself)

TimeFusion has no built-in auth

TimeFusion exposes a PostgreSQL wire protocol without user authentication. In production deployments, restrict access to TimeFusion's port 5432 using network policies (Kubernetes NetworkPolicy, security groups, or firewall rules). Only the Monoscope backend and authorized operators can get to TimeFusion directly.

Data Protection

Encryption at Rest

Telemetry data is stored as Parquet files in S3 via Delta Lake. Encryption at rest depends on the S3 bucket configuration:

Storage Backend Encryption Method
AWS S3 SSE-S3 (default), SSE-KMS (recommended for audit trails), or SSE-C
MinIO Server-side encryption with KMS integration or MinIO's built-in encryption
Self-hosted S3-compatible Depends on the storage backend. Configure per vendor documentation

PostgreSQL metadata is encrypted at rest if the underlying storage volume supports it (for example, EBS encryption for RDS, LUKS for self-hosted PostgreSQL).

Enable SSE-KMS for compliance

For regulated environments, configure the BYOS S3 bucket with SSE-KMS using a customer-managed key. This provides key audit trails via CloudTrail and the ability to revoke access by disabling the key.

Encryption in Transit

Path Encryption
Client apps to OTel Collector TLS (configure in OTel Collector)
OTel Collector to Monoscope API TLS on gRPC (port 4317 with TLS termination at load balancer or Monoscope)
Monoscope to S3 HTTPS (S3 API calls use TLS by default)
Monoscope to PostgreSQL TLS (configure sslmode=require in DATABASE_URL)
Monoscope to Kafka TLS + SASL (configure in Kafka broker and Monoscope env vars)
Web browser to Monoscope UI HTTPS (TLS termination at reverse proxy / load balancer)

Data Retention Policies

Retention is configurable at the project level:

  • S3 telemetry data: No automatic expiration by default. Configure S3 lifecycle rules on the BYOS bucket to transition old data to Glacier or expire it after a defined period (for example, 90 days for hot, 365 days for archive).
  • PostgreSQL metadata: Retention follows the application lifecycle. Alert history and audit logs grow over time. Implement periodic cleanup or archival.
  • Delta Lake time travel: TimeFusion's Delta Lake format supports versioned data. Use VACUUM operations to reclaim space from old versions and keep the desired retention window.

Network Security

Ingestion Endpoint Security

The OTLP ingestion endpoint (gRPC port 4317) is the primary attack surface because it is Internet-facing in most deployments.

Hardening measures:

  • TLS termination: Terminate TLS at the load balancer or reverse proxy (NGINX, Envoy) in front of Monoscope
  • Bearer token validation: Every OTLP request must include a valid project API key. Requests without a valid token are rejected
  • Rate limiting: Configure rate limiting at the load balancer or reverse proxy to prevent ingestion flooding
  • IP allowlisting: If the set of ingesting services is known, restrict ingress to those IP ranges at the security group / firewall level
  • Network segmentation: Place the ingestion endpoint in a DMZ subnet. Backend components (TimeFusion, PostgreSQL, Kafka) must not be directly accessible from the Internet

Query Authorization

Access to telemetry data through the web UI is mediated by the Monoscope backend, which enforces project-level RBAC. Direct access to TimeFusion bypasses RBAC, so it must be network-restricted.

For Kubernetes deployments, use a NetworkPolicy to restrict access to TimeFusion's pod:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: timefusion-access
  namespace: observability
spec:
  podSelector:
    matchLabels:
      app: timefusion
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: monoscope
      ports:
        - protocol: TCP
          port: 5432

Threat Model

Data Exfiltration Risks

Monoscope stores full-fidelity telemetry including HTTP request/response bodies, user IDs, and potentially PII that flows through OTel attributes. Exfiltration risks include:

  • Compromised API key: An attacker with a valid project API key can query all telemetry for that project via the web UI or API. Mitigate by rotating API keys regularly and monitoring for unusual query patterns.
  • S3 bucket misconfiguration: If the BYOS bucket is publicly accessible or has overly permissive IAM policies, all telemetry data is exposed. Mitigate with S3 Block Public Access, bucket policies restricting to specific IAM roles, and S3 access logging.
  • TimeFusion direct access: If an attacker gains network access to TimeFusion's port 5432, they can query all data without authentication. Mitigate with strict network segmentation.

BYOS Bucket Security

The BYOS model means the customer controls the S3 bucket. Recommended bucket hardening:

  • Enable S3 Block Public Access (all four settings)
  • Enable S3 Versioning for data integrity and recovery
  • Configure bucket policy restricting access to the Monoscope service IAM role and authorized operators only
  • Enable S3 Access Logging or CloudTrail S3 data events for audit
  • Enable SSE-KMS with a customer-managed key for encryption
  • Configure S3 Lifecycle Rules for cost-effective retention tiers
  • Consider S3 Object Lock (compliance mode) for regulatory requirements that mandate immutable storage

LLM Query Injection

Monoscope's natural language query engine translates user input into SQL via an LLM. This introduces a prompt injection risk where a malicious user can craft input to manipulate the generated SQL.

Potential attack vectors:

  • SQL injection via LLM: Crafted natural language input that causes the LLM to generate malicious SQL (for example, DROP TABLE, cross-project queries). Monoscope mitigates this by executing queries through TimeFusion with read-only access and project-scoped table views.
  • Data exfiltration via LLM: Input designed to extract data from other projects or system tables. Mitigated by the project_id partitioning in TimeFusion and restricted query scope.
  • LLM prompt leakage: Input designed to make the LLM reveal its system prompt or internal instructions. This is a low-severity risk because the system prompt contains query templates, not secrets.

LLM query engine is pre-1.0

The LLM query engine is actively evolving. Review Monoscope release notes for security patches related to query injection. Consider restricting LLM query access to trusted roles in sensitive environments until the feature stabilizes.

Recommendations

  1. Rotate API keys on a regular schedule and immediately after team member departures
  2. Audit S3 bucket policies quarterly. Use AWS Config rules like s3-bucket-public-read-prohibited and s3-bucket-ssl-requests-only
  3. Network-segment TimeFusion so only the Monoscope backend can get to it
  4. Enable TLS everywhere: ingestion, database connections, and the web UI
  5. Restrict LLM query access to Admin and Owner roles in projects containing sensitive data
  6. Monitor for PII in telemetry: Use OTel Collector processors (for example, attributes/delete, transform) to strip sensitive attributes before ingestion