SWIRL Docs
/
swirlaiconnect.com
Health Checks, Metrics, and Observability

Monitoring Guide

Overview

SWIRL exposes health check and metrics endpoints for production monitoring. They integrate with standard tools such as Kubernetes, Prometheus, and alerting systems, so operators can track system health, detect issues, and maintain reliability.

For security-focused logging (authentication events, lockouts, SIEM forwarding), see the Security Guide.

Health Check Endpoints

SWIRL exposes health-check endpoints that report the status of critical system components, suitable for Kubernetes readiness and liveness probes.

/swirl/health/celery/ Endpoint

The /swirl/health/celery/ endpoint dispatches a ping task to the Celery workers and reports whether they respond. The same route is also available under the /api/swirl/ prefix.

Usage

To check the health of Celery workers:

curl http://localhost:8000/swirl/health/celery/

Example Response

A healthy response returns HTTP 200:

{
  "status": "ok",
  "message": "Celery is running and responsive"
}

An unresponsive or failing worker returns HTTP 500 with "status": "error" and a message describing the failure (for example, a ping timeout).

Kubernetes Configuration

Use this endpoint as a readiness probe to ensure traffic is only routed to healthy SWIRL instances:

apiVersion: v1
kind: Pod
metadata:
  name: swirl-api
spec:
  containers:
  - name: swirl
    image: swirlai/swirl-search-internal:<version>
    livenessProbe:
      httpGet:
        path: /swirl/health/celery/
        port: 8000
      initialDelaySeconds: 30
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /swirl/health/celery/
        port: 8000
      initialDelaySeconds: 10
      periodSeconds: 5

Metrics Endpoints

SWIRL exposes Prometheus-compatible metrics endpoints for detailed monitoring and alerting integration.

/swirl/metrics/celery/ Endpoint

The /swirl/metrics/celery/ endpoint exposes Celery queue metrics in Prometheus format, including queue depth, active tasks, worker counts, and queue saturation.

Usage

To retrieve Celery metrics:

curl http://localhost:8000/swirl/metrics/celery/

Metrics Provided

The endpoint exposes metrics from the CeleryQueueCollector, including:

  • celery_queue_depth - Number of tasks waiting in each Celery queue
  • celery_queue_active_tasks - Number of currently executing tasks per queue
  • celery_queue_workers - Number of workers serving each queue
  • celery_queue_total_capacity - Total task capacity per queue (workers × concurrency)
  • celery_queue_busy_ratio - Ratio of active tasks to total capacity per queue
  • celery_metrics_collection_error - Set to 1 if metrics collection fails

All gauges are labeled by queue. The queues are health_check, search, page_fetch, interactive, maintenance, and corpus; a totals label aggregates across the queues served by the instance.

Example Metrics Output

## HELP celery_queue_depth Number of tasks waiting in each Celery queue
## TYPE celery_queue_depth gauge
celery_queue_depth{queue="search"} 12
celery_queue_depth{queue="interactive"} 3
celery_queue_depth{queue="totals"} 15

## HELP celery_queue_active_tasks Number of currently executing tasks per queue on this pod
## TYPE celery_queue_active_tasks gauge
celery_queue_active_tasks{queue="search"} 2
celery_queue_active_tasks{queue="interactive"} 1

## HELP celery_queue_busy_ratio Ratio of active tasks to capacity on this pod per queue
## TYPE celery_queue_busy_ratio gauge
celery_queue_busy_ratio{queue="search"} 0.25
celery_queue_busy_ratio{queue="interactive"} 0.13

Prometheus Integration

Integrate SWIRL with Prometheus to collect and store metrics for analysis and alerting.

Configuration

Add the following scrape configuration to your prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'swirl'
    static_configs:
      - targets: ['localhost:8000']
    metrics_path: '/swirl/metrics/celery/'
    scrape_interval: 30s
    scrape_timeout: 10s

Verifying Prometheus Integration

  1. Start Prometheus with the updated configuration:
   prometheus --config.file=prometheus.yml

  1. Open the Prometheus UI at http://localhost:9090

  2. Query SWIRL metrics:

   celery_queue_depth
   celery_queue_active_tasks
   celery_queue_busy_ratio

Log Files

SWIRL maintains separate log files for different components, enabling granular troubleshooting and audit trails.

Log Locations

logs/django.log

Contains Django and API server activity: - HTTP request/response logs - Authentication and authorization events - API startup messages and initialization errors - Database connection issues

Example:

[2026-04-12 10:15:30] INFO [django.request] GET /swirl/search/ 200
[2026-04-12 10:15:45] WARNING [django.db] Slow query detected: 2.3s
[2026-04-12 10:16:00] ERROR [django.security] Failed login attempt from 192.168.1.100

logs/celery-*-worker.log

SWIRL runs one Celery worker per queue, each with its own log file:

  • logs/celery-search-worker.log - search federation execution
  • logs/celery-search-orchestrator-worker.log - search orchestration
  • logs/celery-interactive-worker.log - chat, RAG, and other interactive tasks
  • logs/celery-pagefetch-worker.log - Page Fetcher activity
  • logs/celery-maintenance-worker.log - cleanup and maintenance tasks
  • logs/celery-healthcheck-worker.log - health-check pings
  • logs/celery-corpus-worker.log - ingestion and indexing

Worker logs contain result processing and aggregation, task processing errors, and worker startup and shutdown events.

Example:

[2026-04-12 10:15:32] INFO [celery.tasks] Task swirl.search.federate[id=abc123] started
[2026-04-12 10:15:45] WARNING [celery.search] Provider timeout: GoogleBooks (5.2s)
[2026-04-12 10:15:50] INFO [celery.tasks] Task swirl.search.federate[id=abc123] completed (5.8s)

logs/celery-beats.log

Contains scheduled task execution logs: - Subscription service runs - License expiration checks - Data cleanup and maintenance tasks - Search result expiration

Example:

[2026-04-12 00:05:00] INFO [celery.beat] Running scheduled task: cleanup_expired_results
[2026-04-12 12:00:00] INFO [celery.beat] Running scheduled task: check_subscriptions
[2026-04-12 06:00:00] WARNING [celery.beat] License expires in 7 days

Viewing Live Logs

Use the SWIRL CLI to tail all service logs in real-time:

python swirl.py logs

This command displays a live stream of logs from Django, Celery workers, and Celery Beat, making it easy to monitor system activity as it happens.

To view a specific log file:

tail -f logs/django.log
tail -f logs/celery-search-worker.log
tail -f logs/celery-beats.log

Key Metrics to Monitor

Monitor these critical metrics to ensure SWIRL operates reliably and efficiently.

Search Latency

What to monitor: Time from search creation (NEW_SEARCH) to completion (FULL_RESULTS_READY)

  • Healthy: < 10 seconds for most searches
  • Warning: 10-20 seconds (potential provider delays or network issues)
  • Critical: > 20 seconds (investigate provider response times or queue congestion)

There is no per-task duration metric; measure end-to-end latency from the API or UI, and use queue saturation as the leading indicator:

celery_queue_busy_ratio{queue="search"}
celery_queue_depth{queue="search"}

Celery Queue Depth

What to monitor: Number of pending tasks in each queue

  • Healthy: 0-10 tasks per queue
  • Warning: 10-50 tasks (workers may be falling behind)
  • Critical: > 50 tasks (worker shortage or performance degradation)

Query:

celery_queue_depth

Worker Memory Consumption

Monitor memory usage per worker to detect memory leaks and prevent out-of-memory crashes.

  • Healthy: < 2 GB per worker
  • Warning: 2-4 GB (check for memory leaks)
  • Critical: > 4 GB (restart worker, investigate leaks)

Collection: Use system monitoring tools (Prometheus node_exporter) to scrape worker process metrics.

SearchProvider Response Times

Monitor response times and error rates for each data source:

  • Healthy: < 5 seconds per provider
  • Warning: 5-10 seconds
  • Critical: > 10 seconds or error rate > 5%

How to track: Per-provider response times and errors are reported in the search worker log:

grep -i "timeout" logs/celery-search-worker.log
grep -i "error" logs/celery-search-worker.log

RAG Pipeline Duration

Monitor the time required to generate AI insights:

  • Healthy: < 5 seconds
  • Warning: 5-15 seconds
  • Critical: > 15 seconds (check LLM provider availability)

RAG and chat tasks run on the interactive queue; monitor its saturation:

celery_queue_busy_ratio{queue="interactive"}
celery_queue_depth{queue="interactive"}

Rate Limiting

SWIRL implements API rate limiting to protect the system from overload and ensure fair resource allocation.

Configuration

SWIRL uses the DocsServiceUserIpThrottle throttle class for rate limiting based on user identity and IP address.

Chat Endpoint Rate Limiting

The default chat endpoint throttle rate is 15 requests per minute. It is configured with the DOCS_CHAT_THROTTLE_RATE environment variable in the .env file:

## .env
DOCS_CHAT_THROTTLE_RATE=15/minute

Adjust this setting to match your deployment requirements, then restart SWIRL:

## Increase to 30 requests per minute
DOCS_CHAT_THROTTLE_RATE=30/minute

Monitoring Rate Limit Hits

Monitor API throttling events in logs and metrics:

grep "Throttled" logs/django.log

If throttling is frequent, increase the throttle rate or add capacity.

Alerting Recommendations

Configure alerts to proactively notify operations teams of potential issues.

Disk Usage

alert:
  - name: HighDiskUsage
    condition: disk_usage_percent > 80
    severity: warning
    action: Review logs and clean up old results

  - name: CriticalDiskUsage
    condition: disk_usage_percent > 95
    severity: critical
    action: Immediately clean up results or expand storage

Celery Queue Depth

alert:
  - name: HighQueueDepth
    condition: celery_queue_depth > 50
    severity: warning
    action: Scale up worker count or investigate performance

  - name: QueueBacklog
    condition: celery_queue_depth > 200
    severity: critical
    action: Immediately scale workers or pause new searches

Worker Health

alert:
  - name: WorkerOffline
    condition: celery_queue_workers == 0 for any queue
    severity: critical
    action: Restart worker, check logs for errors

  - name: HighWorkerMemory
    condition: worker_memory_mb > 4096
    severity: warning
    action: Investigate memory leak, restart worker if persistent

License Expiration

alert:
  - name: LicenseExpiringSoon
    condition: days_until_expiration < 30
    severity: warning
    action: Renew license before expiration

  - name: LicenseExpired
    condition: current_date > license_expiration_date
    severity: critical
    action: Renew license immediately to restore functionality

Provider Availability

alert:
  - name: ProviderHighErrorRate
    condition: provider_error_rate > 5%
    severity: warning
    action: Check provider status page, verify credentials

  - name: ProviderTimeout
    condition: provider_response_time > 10 seconds
    severity: warning
    action: Check network connectivity, adjust timeout settings

Search Latency

alert:
  - name: SearchLatencyHigh
    condition: search_duration_p95 > 20 seconds
    severity: warning
    action: Review provider performance, check queue depth

Setting Up Alerts in Prometheus

Add alerting rules to prometheus-rules.yml:

groups:
  - name: swirl_alerts
    interval: 30s
    rules:
      - alert: HighCeleryQueueDepth
        expr: celery_queue_depth > 50
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High Celery queue depth detected"
          description: "Queue {{ $labels.queue }} has {{ $value }} pending tasks"

      - alert: WorkerMemoryHigh
        expr: process_resident_memory_bytes > 4294967296
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Worker memory usage exceeding 4GB"
          description: "Worker {{ $labels.worker }} is using {{ $value }} bytes"

The WorkerMemoryHigh rule uses process_resident_memory_bytes, which requires a process or node exporter alongside SWIRL; it is not exposed by the SWIRL metrics endpoint.

Configure notifications via Alertmanager to email, Slack, or PagerDuty:

## alertmanager.yml
route:
  receiver: 'default'
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h

receivers:
  - name: 'default'
    slack_configs:
      - api_url: 'YOUR_SLACK_WEBHOOK_URL'
        channel: '#alerts'
        title: 'SWIRL Alert'

Need Help?

For additional support and questions about monitoring SWIRL: - Review the User Guide for system operation - Check the Architecture documentation for system design - Visit the SWIRL GitHub Issues for known issues