# Terra (Primary Compute)

Deep dive into the main workhorse of the homelab, covering hardware specs, ZFS storage architecture, and container standards.

# Hardware & OS

# Hardware & OS

Terra is the primary compute node for the Flora Family homelab, designed for high-concurrency tasks like AI processing, media transcoding, and ZFS storage management.

## 🖥️ Physical Specs
*   **Motherboard:** Asus Prime X570-Pro
*   **CPU:** AMD Ryzen 9 5950X (16 Cores, 32 Threads)
    *   **Scaling Driver:** `amd_pstate=active` (EPP)
    *   **Frequency Range:** 550MHz - 5000MHz
*   **RAM:** 128GB (4x32GB) Corsair Vengeance LPX DDR4 3200MHz CL16 (Non-ECC)
    *   *Note: Currently running at 2133MHz; XMP/DOCP troubleshooting in progress.*
*   **GPU:** Nvidia Quadro P2200 (5GB VRAM)
*   **Chassis:** iStar D-400 with Red Drive Doors (4U Rackmount)
*   **Power Supply:** be quiet! Dark Power Pro 11 650W (80 Plus Platinum)
*   **Expansion & Storage Connectivity:
    *   **LSI 9207-8i HBA:** Driving the 4x 12TB HDD stack.
    *   **Motherboard SATA:** Driving the 6x 2TB SSD stack.
    *   **4-port Gigabit NIC:** Realtek chipset; currently unused.

## 🌬️ Cooling (Noctua Standards)
*   **CPU Cooler:** Noctua NH-C14S (with dual 140mm fans).
*   **NVMe Cooling:** Both drives equipped with heatsinks including integrated active fans.
    *   *Note: Active cooling reduced temperatures significantly. Current stats (composite): drive 0400 @ 42C, 0100 @ 37C.*
*   **Case Fans:** All stock fans replaced with Noctua units.
*   **Additional Airflow:** Added 40mm fan on the HBA and 2x 80mm fans at the chassis front.

### Fan Connectivity Map
| Sensor | Description | Connection |
| :--- | :--- | :--- |
| fan1 | CHA_FAN1 | back exhaust fans (2x80mm) |
| fan2 | CPU_FAN | bottom CPU fan (1x120mm) |
| fan3 | CHA_FAN2 | SDD drive cage fan (1x40mm) |
| fan4 | CHA_FAN3 | front intake fans (2x80mm) |
| fan5 | ? | Shows 0 rpm |
| fan6 | W_PUMP | HBA & NVMe fans (1x40mm + 2x20mm) |
| fan7 | ? | Shows 0 rpm |
| ? | CPU_OPT | top CPU fan (1x120mm) |
| ? | AIO_PUMP | empty |
| - | - | HDD drive cage fan (1x80mm) |


## 💾 Storage Media
*   **NVMe:** 2x 500GB WD Black SN7100 (Used for OS mirrors and ZFS Special VDEVs/L2ARC).
*   **SSD:** 6x 2TB Samsung 870 EVO (Main pool).
*   **HDD:** 4x 12TB Seagate Exos X16 (Bulk pool).

## ⚙️ OS & Software
*   **OS:** Ubuntu 24.04 LTS
*   **Kernel:** ZFS-on-Linux (OpenZFS)
*   **Management:** Sanoid/Syncoid for snapshots and replication.

# ZFS Storage Architecture

# ZFS Storage Architecture

Terra utilizes ZFS for primary data storage, leveraging its advanced features for data integrity, snapshots, and caching.

## 🏊 Pools

### 🚀 `main` (SSD Pool)
*   **Hardware:** Raid-Z2 (6x 2TB SSD)
*   **Total Capacity:** ~12TB Raw
*   **Use Case:** High-performance data, VM disks, Docker root, and active AppData.
*   **L2ARC:** 48GB limit (leveraging NVMe partitions).

### 🐘 `bulk` (HDD Pool)
*   **Hardware:** Raid-Z1 (4x 12TB HDD)
*   **Total Capacity:** ~48TB Raw
*   **Use Case:** Large media storage, backups, and long-term archival.
*   **Special VDEV:** Mirrored NVMe partitions used for metadata offloading to speed up file listing on HDDs.

## 📂 Key Datasets & Mounts
*   **/srv**: Docker-compose configuration files (Stored on `main`).
*   **/main/appdata**: Primary persistent storage for containers (Stored on `main`).
*   **/bulk/pics**: Family photo/video archive (Stored on `bulk`).

## 🛡️ Health & Maintenance

### 🔍 Automated Monitoring (`smartd`)
Drive health is monitored via `smartmontools` with automated short and long self-tests configured to email results. Test schedules are staggered to prevent resource contention.

*   **NVMe (2x 500GB):** Short tests daily at 1am & 2am; Long tests quarterly on the 1st-2nd.
*   **HDD (4x 12TB):** Short tests daily at 3am-6am; Long tests on the 4th, 6th, 8th, and 10th of each month.
*   **SSD (6x 2TB):** Short tests daily from 1am-6am; Long tests quarterly on the 3rd, 5th, 7th, 9th, 11th, and 12th.

### 🧹 ZFS Scrubbing
A scrub is performed automatically on the 3rd Sunday of every month to ensure data integrity and prevent bitrot.

**Scrub Schedule (Cron):**
`24 0 15-21 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi`

# Container Architecture

# Container Architecture

The Flora Family homelab relies on a standardized Docker deployment pattern to ensure security, portability, and ease of management.

## 📂 Directory Layout
To maintain a clean separation between configuration and state, Terra uses two primary directories:

*   **/srv/[service-name]/**: Contains `docker-compose.yaml` and `.env` files. 
    *   *Ownership:* `root:root` (Modified only via sudo).
    *   *Permissions:* folders `755`, configs `644`, .env `600`.
*   **/main/appdata/[service-name]/**: Contains persistent application data.
    *   *Ownership:* Mapped to the specific service account (UID/GID).
    *   *Permissions:* `755` (folders) / `644` (files) generally.

## 🛠️ Deployment Standards

### 🔐 Non-Root Execution
Every service is configured to run as a non-privileged user, with the specific exceptions of **Watchtower** and **Beszel Agent** (which require root/host socket access to monitor system health and container status).

Non-root execution is achieved through one of three methods:

1.  **Standard `user:` flag**: For images that support it (e.g., `user: "51100:61100"`).
2.  **Environment Variables**: Many linuxserver.io images use `PUID`/`PGID` variables.
3.  **Custom Dockerfile Builds**: Used for "scratch" images or minimalist images (like OpenWebUI and SearXNG) to manually inject the desired UID/GID and install local CA certificates.

### 🔗 Shared Infrastructure
Common backend services are standardized to simplify inter-container networking and permissions:

*   **Postgres:** Standardized on GID `70` across all instances (Immich, n8n, etc.).
*   **Redis/Valkey:** Standardized on GID `970` for caching and session management.
*   **Shared Identities:** The `node-user` (UID 1000) is shared between several JS-based apps (n8n, Uptime Kuma) where common file access is required.

## 🔄 Lifecycle Management
*   **Updates:** Managed by **Watchtower**, configured to run every Sunday at 04:00 AM.
*   **Image Tagging:**
    *   *Pinned:* Databases and critical infrastructure (e.g., `postgres:18`) to prevent breaking updates.
    *   *Floating:* Application layers (e.g., `sonarr:latest`) to receive automated security patches.

# Performance & Stability Optimizations

# Performance & Stability Optimizations

This page documents the specific kernel and hardware tweaks applied to Terra (Ryzen 5950X / Ubuntu 24.04) to optimize for container performance, storage throughput, and long-term stability.

## 🛡️ Stability & Power Management (The "Golden Config")

### Typical Current Idle (UEFI)
*   **Setting:** Advanced > AMD CBS > CPU Common > Power Supply Idle Control = "Typical Current Idle"
*   **Benefit:** Prevents the Ryzen "Sleep of Death" where the PSU cuts power during deep C-state transitions. Essential for 24/7 server stability on X570 boards.

### Processor C-State Limit (Kernel)
*   **Driver:** `processor.max_cstate=1`
*   **Benefit:** Software-side enforcement of the Idle fix. Limits the CPU to C1 (Halt/Idle) instead of deeper sleep states (C6), preventing interrupt processing delays and hardware hangs.
*   **Configuration:** Added to `GRUB_CMDLINE_LINUX_DEFAULT` in `/etc/default/grub`.

### 65W Eco Mode (Manual PBO)
*   **Settings:** PPT: 88, TDC: 60, EDC: 90
*   **Benefit:** Drastically reduces thermals (peak temps dropped from ~90°C to 58°C) without significant impact on multi-core performance. Ideal for 4U rackmount chassis airflow.
*   **Configuration:** UEFI > Precision Boost Overdrive > Manual.

## ⚡ CPU & Power Performance

### AMD P-State (Active Mode)
*   **Driver:** `amd_pstate=active` (EPP - Energy Performance Preference)
*   **Benefit:** Allows the kernel to communicate directly with the Zen 3 power management controller. Enables a wider frequency range (550MHz - 5000MHz).

### IRQ Balancing
*   **Service:** `irqbalance` (Installed/Enabled)
*   **Benefit:** Distributes hardware interrupts (HBA, NVMe, NIC) across all threads, preventing Core 0 bottlenecks.

## 🧠 Memory Tuning

### Speed & Stability
*   **Configuration:** Currently hard-coded to **2133MHz** at **AUTO** voltage.
*   **Note:** Troubleshooting in Feb 2026 confirmed that 3000MHz at 1.35V caused instantaneous reboots. Stability requires sticking to JEDEC defaults for now.

### Static Hugepages
*   **Setting:** `hugepages=2048` (Reserving 4GB total as 2MB pages)
*   **Benefit:** Reduces TLB misses for memory-heavy apps like Minecraft and Postgres.

## 💾 I/O & Network Tuning (sysctl)

### ZFS Write Buffering
*   **Settings:** `vm.dirty_background_bytes = 268435456`, `vm.dirty_bytes = 1073741824`
*   **Benefit:** Prevents the kernel from buffering too much data in RAM before flushing to ZFS, eliminating system-wide UI/process stutter.

### Fluent-bit Batching (ClickHouse)
*   **Setting:** `Flush 60` in `fluent-bit.conf`
*   **Benefit:** Reduced ClickHouse CPU usage from 80% spikes to <5% by reducing transaction frequency and optimizing ZFS write contiguousness.

## 🎮 GPU Optimization (Quadro P2200)

### Persistence Mode
*   **Command:** `sudo nvidia-smi -pm 1`
*   **Benefit:** Eliminates the 2-second initialization delay for AI inference and transcoding containers.

# Luna (Backup Server)

# Luna (Backup Server)

Luna is the secondary compute and backup node for the Flora Family homelab. It serves as the primary replication target for Terra.

## 🖥️ System Status
*   **Current Status:** **ONLINE** (Recovered 2026-02-25)
*   **Role:** ZFS Replication Target, Thermal Monitoring
*   **Observability:** Fully integrated into the Vector/ClickHouse logging pipeline (Feb 2026).

## 💾 Storage Architecture (ZFS)
*   **Pool Configuration:** Striped Raid-Z1 vdevs
*   **Capacity:** 
    *   4x 4TB Drives
    *   4x 3TB Drives

## 🌡️ Monitoring
*   **Integrated Script:** `/usr/local/bin/cpu_temp.sh` (Runs via CRON every 5 minutes)
*   **Reporting:** Logs to ClickHouse via Vector.
*   **Average Temps:** ~46°C - 48°C (Idle/Light load)

## ⚙️ Software & Management
*   **Management:** Syncoid target for Terra's datasets.
*   **Logging:** Migrated from Fluent-Bit to Vector (Feb 25, 2026). 
*   **Sanoid:** Configured with `--quiet` to reduce log spam from snapshot cleanup.
*   **Timezone:** Correctly set to `America/New_York (EST)`. System clock and Vector timestamps are aligned.

## 🛠️ Recovery Notes (Feb 2026)
*   Luna was brought back online after an outage. 
*   Vector configuration synchronized via Ansible with Terra.
*   Verified successful `syncoid` backup completions from Terra at 07:44 AM EST (Feb 25).