# AI-Powered Log Aggregation (ClickHouse + Ollama)

# AI-Powered Log Aggregation with ClickHouse & Ollama

This page documents the minimalist log aggregation and AI search pipeline built for the Flora family homelab (terra + opnsense).

## Architecture

- **Aggregator**: ClickHouse (bare-metal on terra)
- **Log Shipper**: Fluent Bit (bare-metal on terra)
- **AI Brain**: Ollama (Docker on terra) running `nomic-embed-text`
- **Deduplication**: `ReplacingMergeTree` engine in ClickHouse using `log_id` (UUID)

## ClickHouse Table Schema

```
CREATE TABLE system_logs (
    timestamp DateTime64(3),
    host String,
    unit String,
    message String,
    priority Int8,
    log_id String,
    embedding Array(Float32),
    INDEX ann_idx embedding TYPE vector_similarity('hnsw', 'cosineDistance', 768) GRANULARITY 1
) ENGINE = ReplacingMergeTree()
ORDER BY (log_id);
```

## Components

### 1. Fluent Bit (Ingestion)

Fluent Bit reads the systemd journal and ships logs to ClickHouse via HTTP. Key configuration includes using the `record_modifier` filter with `Uuid_key log_id` to ensure unique identification.

### 2. Log Brain (Embedding Loop)

A Python background service (`log-brain.service`) runs an embedding loop:

1. Queries logs where `length(embedding) = 0` using `FINAL` to handle merging.
2. Sends the message to Ollama for embedding.
3. Upserts the record back to ClickHouse with the vector.

### 3. Reverse Proxy

Caddy provides internal TLS for both Ollama (`ollama.flora.family`) and ClickHouse (`logs.flora.family`).

## Search Examples

I (Flobot) can now search logs using standard SQL for keywords or `cosineDistance` for semantic similarity.