Terra (Primary Compute) Deep dive into the main workhorse of the homelab, covering hardware specs, ZFS storage architecture, and container standards. Hardware & OS Hardware & OS Terra is the primary compute node for the Flora Family homelab, designed for high-concurrency tasks like AI processing, media transcoding, and ZFS storage management. ๐Ÿ–ฅ๏ธ Physical Specs Motherboard: Asus Prime X570-Pro CPU: AMD Ryzen 9 5950X (16 Cores, 32 Threads) Scaling Driver: amd_pstate=active (EPP) Frequency Range: 550MHz - 5000MHz RAM: 128GB (4x32GB) Corsair Vengeance LPX DDR4 3200MHz CL16 (Non-ECC) Note: Currently running at 2133MHz; XMP/DOCP troubleshooting in progress. GPU: Nvidia Quadro P2200 (5GB VRAM) Chassis: iStar D-400 with Red Drive Doors (4U Rackmount) Power Supply: be quiet! Dark Power Pro 11 650W (80 Plus Platinum) **Expansion & Storage Connectivity: LSI 9207-8i HBA: Driving the 4x 12TB HDD stack. Motherboard SATA: Driving the 6x 2TB SSD stack. 4-port Gigabit NIC: Realtek chipset; currently unused. ๐ŸŒฌ๏ธ Cooling (Noctua Standards) CPU Cooler: Noctua NH-C14S (with dual 140mm fans). NVMe Cooling: Both drives equipped with heatsinks including integrated active fans. Note: Active cooling reduced temperatures significantly. Current stats (composite): drive 0400 @ 42C, 0100 @ 37C. Case Fans: All stock fans replaced with Noctua units. Additional Airflow: Added 40mm fan on the HBA and 2x 80mm fans at the chassis front. Fan Connectivity Map Sensor Description Connection fan1 CHA_FAN1 back exhaust fans (2x80mm) fan2 CPU_FAN bottom CPU fan (1x120mm) fan3 CHA_FAN2 SDD drive cage fan (1x40mm) fan4 CHA_FAN3 front intake fans (2x80mm) fan5 ? Shows 0 rpm fan6 W_PUMP HBA & NVMe fans (1x40mm + 2x20mm) fan7 ? Shows 0 rpm ? CPU_OPT top CPU fan (1x120mm) ? AIO_PUMP empty - - HDD drive cage fan (1x80mm) ๐Ÿ’พ Storage Media NVMe: 2x 500GB WD Black SN7100 (Used for OS mirrors and ZFS Special VDEVs/L2ARC). SSD: 6x 2TB Samsung 870 EVO (Main pool). HDD: 4x 12TB Seagate Exos X16 (Bulk pool). โš™๏ธ OS & Software OS: Ubuntu 24.04 LTS Kernel: ZFS-on-Linux (OpenZFS) Management: Sanoid/Syncoid for snapshots and replication. ZFS Storage Architecture ZFS Storage Architecture Terra utilizes ZFS for primary data storage, leveraging its advanced features for data integrity, snapshots, and caching. ๐ŸŠ Pools ๐Ÿš€ main (SSD Pool) Hardware: Raid-Z2 (6x 2TB SSD) Total Capacity: ~12TB Raw Use Case: High-performance data, VM disks, Docker root, and active AppData. L2ARC: 48GB limit (leveraging NVMe partitions). ๐Ÿ˜ bulk (HDD Pool) Hardware: Raid-Z1 (4x 12TB HDD) Total Capacity: ~48TB Raw Use Case: Large media storage, backups, and long-term archival. Special VDEV: Mirrored NVMe partitions used for metadata offloading to speed up file listing on HDDs. ๐Ÿ“‚ Key Datasets & Mounts /srv: Docker-compose configuration files (Stored on main). /main/appdata: Primary persistent storage for containers (Stored on main). /bulk/pics: Family photo/video archive (Stored on bulk). ๐Ÿ›ก๏ธ Health & Maintenance ๐Ÿ” Automated Monitoring ( smartd) Drive health is monitored via smartmontools with automated short and long self-tests configured to email results. Test schedules are staggered to prevent resource contention. NVMe (2x 500GB): Short tests daily at 1am & 2am; Long tests quarterly on the 1st-2nd. HDD (4x 12TB): Short tests daily at 3am-6am; Long tests on the 4th, 6th, 8th, and 10th of each month. SSD (6x 2TB): Short tests daily from 1am-6am; Long tests quarterly on the 3rd, 5th, 7th, 9th, 11th, and 12th. ๐Ÿงน ZFS Scrubbing A scrub is performed automatically on the 3rd Sunday of every month to ensure data integrity and prevent bitrot. Scrub Schedule (Cron): 24 0 15-21 * * root if [ $(date +\%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi Container Architecture Container Architecture The Flora Family homelab relies on a standardized Docker deployment pattern to ensure security, portability, and ease of management. ๐Ÿ“‚ Directory Layout To maintain a clean separation between configuration and state, Terra uses two primary directories: /srv/[service-name]/: Contains docker-compose.yaml and .env files. Ownership: root:root (Modified only via sudo). Permissions: folders 755, configs 644, .env 600. /main/appdata/[service-name]/: Contains persistent application data. Ownership: Mapped to the specific service account (UID/GID). Permissions: 755 (folders) / 644 (files) generally. ๐Ÿ› ๏ธ Deployment Standards ๐Ÿ” Non-Root Execution Every service is configured to run as a non-privileged user, with the specific exceptions of Watchtower and Beszel Agent (which require root/host socket access to monitor system health and container status). Non-root execution is achieved through one of three methods: Standard user: flag: For images that support it (e.g., user: "51100:61100"). Environment Variables: Many linuxserver.io images use PUID/ PGID variables. Custom Dockerfile Builds: Used for "scratch" images or minimalist images (like OpenWebUI and SearXNG) to manually inject the desired UID/GID and install local CA certificates. ๐Ÿ”— Shared Infrastructure Common backend services are standardized to simplify inter-container networking and permissions: Postgres: Standardized on GID 70 across all instances (Immich, n8n, etc.). Redis/Valkey: Standardized on GID 970 for caching and session management. Shared Identities: The node-user (UID 1000) is shared between several JS-based apps (n8n, Uptime Kuma) where common file access is required. ๐Ÿ”„ Lifecycle Management Updates: Managed by Watchtower, configured to run every Sunday at 04:00 AM. Image Tagging: Pinned: Databases and critical infrastructure (e.g., postgres:18) to prevent breaking updates. Floating: Application layers (e.g., sonarr:latest) to receive automated security patches. Performance & Stability Optimizations Performance & Stability Optimizations This page documents the specific kernel and hardware tweaks applied to Terra (Ryzen 5950X / Ubuntu 24.04) to optimize for container performance, storage throughput, and long-term stability. ๐Ÿ›ก๏ธ Stability & Power Management (The "Golden Config") Typical Current Idle (UEFI) Setting: Advanced > AMD CBS > CPU Common > Power Supply Idle Control = "Typical Current Idle" Benefit: Prevents the Ryzen "Sleep of Death" where the PSU cuts power during deep C-state transitions. Essential for 24/7 server stability on X570 boards. Processor C-State Limit (Kernel) Driver: processor.max_cstate=1 Benefit: Software-side enforcement of the Idle fix. Limits the CPU to C1 (Halt/Idle) instead of deeper sleep states (C6), preventing interrupt processing delays and hardware hangs. Configuration: Added to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub. 65W Eco Mode (Manual PBO) Settings: PPT: 88, TDC: 60, EDC: 90 Benefit: Drastically reduces thermals (peak temps dropped from ~90ยฐC to 58ยฐC) without significant impact on multi-core performance. Ideal for 4U rackmount chassis airflow. Configuration: UEFI > Precision Boost Overdrive > Manual. โšก CPU & Power Performance AMD P-State (Active Mode) Driver: amd_pstate=active (EPP - Energy Performance Preference) Benefit: Allows the kernel to communicate directly with the Zen 3 power management controller. Enables a wider frequency range (550MHz - 5000MHz). IRQ Balancing Service: irqbalance (Installed/Enabled) Benefit: Distributes hardware interrupts (HBA, NVMe, NIC) across all threads, preventing Core 0 bottlenecks. ๐Ÿง  Memory Tuning Speed & Stability Configuration: Currently hard-coded to 2133MHz at AUTO voltage. Note: Troubleshooting in Feb 2026 confirmed that 3000MHz at 1.35V caused instantaneous reboots. Stability requires sticking to JEDEC defaults for now. Static Hugepages Setting: hugepages=2048 (Reserving 4GB total as 2MB pages) Benefit: Reduces TLB misses for memory-heavy apps like Minecraft and Postgres. ๐Ÿ’พ I/O & Network Tuning (sysctl) ZFS Write Buffering Settings: vm.dirty_background_bytes = 268435456, vm.dirty_bytes = 1073741824 Benefit: Prevents the kernel from buffering too much data in RAM before flushing to ZFS, eliminating system-wide UI/process stutter. Fluent-bit Batching (ClickHouse) Setting: Flush 60 in fluent-bit.conf Benefit: Reduced ClickHouse CPU usage from 80% spikes to <5% by reducing transaction frequency and optimizing ZFS write contiguousness. ๐ŸŽฎ GPU Optimization (Quadro P2200) Persistence Mode Command: sudo nvidia-smi -pm 1 Benefit: Eliminates the 2-second initialization delay for AI inference and transcoding containers. Luna (Backup Server) Luna (Backup Server) Luna is the secondary compute and backup node for the Flora Family homelab. It serves as the primary replication target for Terra. ๐Ÿ–ฅ๏ธ System Status Current Status: ONLINE (Recovered 2026-02-25) Role: ZFS Replication Target, Thermal Monitoring Observability: Fully integrated into the Vector/ClickHouse logging pipeline (Feb 2026). ๐Ÿ’พ Storage Architecture (ZFS) Pool Configuration: Striped Raid-Z1 vdevs Capacity: 4x 4TB Drives 4x 3TB Drives ๐ŸŒก๏ธ Monitoring Integrated Script: /usr/local/bin/cpu_temp.sh (Runs via CRON every 5 minutes) Reporting: Logs to ClickHouse via Vector. Average Temps: ~46ยฐC - 48ยฐC (Idle/Light load) โš™๏ธ Software & Management Management: Syncoid target for Terra's datasets. Logging: Migrated from Fluent-Bit to Vector (Feb 25, 2026). Sanoid: Configured with --quiet to reduce log spam from snapshot cleanup. Timezone: Correctly set to America/New_York (EST). System clock and Vector timestamps are aligned. ๐Ÿ› ๏ธ Recovery Notes (Feb 2026) Luna was brought back online after an outage. Vector configuration synchronized via Ansible with Terra. Verified successful syncoid backup completions from Terra at 07:44 AM EST (Feb 25).