Homelab/AGENTS.md

# AGENTS - Docker Infrastructure Documentation

## Overview
This document provides a comprehensive overview of the Docker infrastructure managed in `/docker/`. This self-hosted ecosystem contains 50+ services organized into 5 compose stacks, all integrated through a reverse proxy and secured with proper network segmentation.

## Compose Stack Organization

The infrastructure is split into 5 independent compose files sharing a common `.env`:

| Stack | File | Services |
|-------|------|----------|
| **Infrastructure** | `docker-compose.infra.yml` | Core: portainer, npm, dockerproxy, homepage, wud, ntopng, newt |
| **Media** | `docker-compose.media.yml` | Media: arr stack, jellyfin, qbittorrent, slskd, metube, maloja, scrobbler |
| **Documents** | `docker-compose.documents.yml` | Paperless-ngx + AI, onlyoffice, stirling-pdf, open-webui |
| **Photo & ROM Library** | `docker-compose.photo-roms.yml` | Immich, syncthing, retrom |
| **Utilities** | `docker-compose.utils.yml` | Gitea, tandoor, speedtest-tracker, rustdesk, redbot, linkwarden, neolink, iperf3 |

## Core Infrastructure Agents (infra.yml)

### Management & Monitoring
- **Portainer** - Container management UI (port 9443)
- **Homepage** - Service dashboard (port 7575)
- **WUD (WhatsUpDocker)** - Docker image management utility
- **ntopng** - Network traffic monitoring (port 3939)

### Reverse Proxy & Security
- **Nginx Proxy Manager** - SSL termination and reverse proxy (ports 80, 443, 81)
- **Docker Socket Proxy** - Secure Docker API access (port 2375)
- **Newt** - Pangolin tunnel client

## Media & Entertainment Stack (media.yml)

### Media Server
- **Jellyfin** - Media server with hardware transcoding (port 8096)
  - Intel GPU acceleration via render device
  - NAS storage integration

### Download & Index Management
- **qBittorrent** - Torrent client (ports 56881, 7070)
- **Prowlarr** - Indexer manager (port 9696)
- **FlareSolverr** - Cloudflare bypass service (port 8191)
- **Metube** - YouTube video downloader (port 8081)

### Content Management
- **Radarr** - Movie management (port 7878)
- **Sonarr** - TV show management (port 8989)
- **Lidarr** - Music management (port 8686)
- **Bazarr** - Subtitle management (port 6767)
- **Jellyseerr** - Media request system (port 5055)

### File Sharing
- **Slskd** - Soulseek file sharing client (ports 5030, 5031, 50300)

### Music & Scrobbling
- **Maloja** - Music scrobbling service (port 42010)
- **Multi-Scrobbler** - Cross-platform scrobbling (port 9078)
  - Integrates with Jellyfin and Last.fm

## Document & AI Suite (documents.yml)

### Document Processing
- **Paperless-ngx** - Document management (port 8100)
  - MariaDB database, Redis broker, Tika text extraction, Gotenberg PDF processing
- **Paperless-AI** - AI-powered document processing for Paperless-ngx
- **Stirling-PDF** - PDF manipulation tools (port 8090)
- **OnlyOffice** - Document collaboration (port 8091)

### AI Interface
- **Open WebUI** - LLM interface (port 3000)

## Photo Management & Library (photo-roms.yml)

### Photo Management
- **Immich** - AI-powered photo management (port 2283)
  - PostgreSQL with vector search, Valkey/Redis cache
  - Local SSD storage for config and thumbs

### File Synchronization
- **Syncthing** - File synchronization (ports 21027, 22000, 8384)
  - Obsidian vault synchronization
  - Multiple shared folders

### ROM Management
- **Retrom** - ROM library management service
  - **retrom**: Main ROM service container (port 5111)
  - **retrom-db**: PostgreSQL database for Retrom
  - **retrom-adminer**: Adminer interface for database management (port 8080)
  - **retrom-jaeger**: Distributed tracing for performance monitoring

## Utilities Stack (utils.yml)

### Development Tools
- **Gitea** - Git service (ports 222, 8418)
  - MySQL database

### Personal Tools
- **Tandoor Recipes** - Self-hosted recipe management (port 8450)
  - PostgreSQL database

### Finance & Tracking
- **Speedtest Tracker** - Network monitoring (port 8180)
  - MariaDB database, automated testing

### Bookmark Management
- **Linkwarden** - Collaborative bookmark manager (port 3400)
  - PostgreSQL database
  - Meilisearch for full-text search

### Remote Access
- **RustDesk** - Remote desktop (host mode)

### Home Automation
- **Neolink** - Reolink camera bridge for Frigate/Home Assistant integration

### Discord Bots
- **RedBot** - Discord bot with custom cogs (internal_net)

### Network Tools
- **iperf3-server** - Network performance testing (port 5201)

## Independent Stacks (separate compose files)

These services are deployed independently and are not part of the main 5-stack split:

| Service | Directory | Role |
|---------|-----------|------|
| **Discord Agent Bot** | `discord-agent/` | AI-powered Discord bot using Ollama LLM |
| **Foundry Watcher** | `foundry-watcher/` | Foundry VTT player monitoring |
| **GrampsWeb Jamie** | `gramps-jamie/` | Genealogy UI (port 5511) |
| **GrampsWeb Helen** | `gramps-helen/` | Genealogy UI (port 5512) |
| **Kasm** | `kasm/` | Browser isolation workspace |
| **LiteLLM** | `litellm/` | AI gateway (port 4000) |
| **MBI Poller** | `mbi-poller/` | MBI data polling service |

## Network Architecture

### Docker Networks
- **media_net** - Media services and *arr stack
- **db_net** - Database services (internal only)
- **web_net** - Web-accessible services
- **internal_net** - Internal service communication

Networks are defined in `docker-compose.infra.yml` and referenced as `external: true` by all other stacks.

### Storage Structure
```
/docker/
├── Arrs/ (Media stack: Prowlarr, Radarr, Sonarr, Lidarr, Bazarr, Jellyfin, Jellyseerr)
├── immich/ (Photo management)
├── paperless/ (Document management)
├── discord-agent/ (Independent — Discord bot with Ollama)
├── gitea/ (Git service)
├── gramps-jamie/ (Independent — Genealogy)
├── gramps-helen/ (Independent — Genealogy)
├── npm/ (Nginx Proxy Manager)
├── qBittorrent/ (Download client)
├── slskd/ (Soulseek client)
├── speedtest-tracker/ (Network monitoring)
├── maloja/ (Music scrobbling)
├── scrobble/ (Multi-scrobbler)
├── syncthing/ (File sync)
├── Homepage/ (Dashboard)
├── retrom/ (ROM library management)
├── wud/ (Docker image management)
├── ntopng/ (Network traffic monitoring)
├── kasm/ (Independent — Browser workspace)
├── litellm/ (Independent — AI gateway)
├── neolink/ (Reolink camera bridge)
├── linkwarden/ (Bookmark manager)
├── surmai/ (Flight tracking)
├── tandoor/ (Recipe management)
├── foundry-watcher/ (Independent — Foundry VTT)
├── rustdesk/ (Remote desktop)
├── redbot/ (Discord bot)
├── stirling/ (PDF tools)
└── Various other service configs
```

## Deployment

### Deploy all main stacks
```bash
for f in docker-compose.infra.yml docker-compose.media.yml docker-compose.documents.yml docker-compose.photo-roms.yml docker-compose.utils.yml; do
  docker compose -f "$f" up -d
done
```

### Deploy a specific stack
```bash
docker compose -f docker-compose.media.yml up -d
```

### Stop a specific stack
```bash
docker compose -f docker-compose.media.yml down
```

## Key Features & Configuration

### Security
- Container security with `no-new-privileges:true`
- Network segmentation (databases internal-only via `db_net`)
- Reverse proxy with SSL termination

### Performance
- Intel GPU hardware acceleration for Jellyfin
- Log rotation (10MB max, 3 files)
- Resource limits (Portainer: 512MB RAM)

### Data Management
- Comprehensive backup script included (`backup.sh`)
- Separate volumes for critical data
- NAS storage integration for media files

### Environment
- System configured for Pacific/Auckland timezone
- PUID/PGID for proper file permissions
- Extensive environment variable configuration
- Shared `.env` file across all stacks

## Service Dependencies

### Database Services
- MariaDB: npm, paperless, speedtest-tracker
- PostgreSQL: immich, retrom, tandoor, linkwarden
- MySQL: gitea
- Redis/Valkey: paperless-broker, immich-redis
- Meilisearch: linkwarden

### Network Dependencies
- `infra.yml` must be deployed first (creates shared networks)
- All web services route through Nginx Proxy Manager
- Internal services communicate via `internal_net`
- Media services isolated on `media_net`
- Database services on internal-only `db_net`

## Maintenance

### Automated Tasks
- **Backup Script** (`backup.sh`) - Regular data backups
- **Speedtest Tracker** - Automated network testing
- **WUD** - Automatic Docker image update monitoring

### Manual Tasks
- Service monitoring via Portainer
- Container management via Portainer
- Service configuration via Nginx Proxy Manager

## Development & Debugging

### Service Access
- **Web Services**: Access through reverse proxy (80/443)
- **Admin Interfaces**: Direct ports for management (e.g., 9443, 7575)
- **Internal Services**: Host-only access for security

### Troubleshooting
- Container logs accessible via Portainer
- Network diagnostics via iperf3-server
- Per-stack logs: `docker compose -f docker-compose.<stack>.yml logs -f <service>`

This Docker infrastructure represents a comprehensive self-hosted ecosystem covering media management, document processing, AI services, development tools, and personal productivity applications, all integrated through a reverse proxy and organized into independent compose stacks with proper network segmentation.