Jamie/Homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Split monolithic compose into 5 independent stacks

Browse Source 
- docker-compose.infra.yml: core infrastructure (portainer, npm, homepage, wud, etc.)
- docker-compose.media.yml: media stack (arrs, jellyfin, qbittorrent, scrobbling)
- docker-compose.documents.yml: paperless-ngx, onlyoffice, stirling, open-webui
- docker-compose.photo-roms.yml: immich, syncthing, retrom
- docker-compose.utils.yml: gitea, tandoor, speedtest, linkwarden, rustdesk, etc.

Each stack has its own project name (docker-infra, docker-media, etc.) to prevent
orphan warnings. Networks defined in infra.yml, referenced as external by others.
Original preserved as docker-compose.full.yaml.bak.
Updated .gitignore, README, AGENTS.md, and RESTORE.md to reflect new structure.
This commit is contained in:
KansaiGaijin
2026-05-14 22:58:11 +12:00
parent 76e99f9df9
commit b2f4d37f19
10 changed files with 1675 additions and 601 deletions
Download Patch File Download Diff File Expand all files Collapse all files

209
AGENTS.md
View File

@@ -1,20 +1,34 @@
# AGENTS - Docker Infrastructure Documentation
## Overview
This document provides a comprehensive overview of the Docker infrastructure managed in `/docker/`. This self-hosted ecosystem contains 30+ services organized into functional categories, all integrated through a reverse proxy and secured with proper network segmentation.
This document provides a comprehensive overview of the Docker infrastructure managed in `/docker/`. This self-hosted ecosystem contains 50+ services organized into 5 compose stacks, all integrated through a reverse proxy and secured with proper network segmentation.
## Core Infrastructure Agents
## Compose Stack Organization
The infrastructure is split into 5 independent compose files sharing a common `.env`:
| Stack | File | Services |
|-------|------|----------|
| **Infrastructure** | `docker-compose.infra.yml` | Core: portainer, npm, dockerproxy, homepage, wud, ntopng, newt |
| **Media** | `docker-compose.media.yml` | Media: arr stack, jellyfin, qbittorrent, slskd, metube, maloja, scrobbler |
| **Documents** | `docker-compose.documents.yml` | Paperless-ngx + AI, onlyoffice, stirling-pdf, open-webui |
| **Photo & ROM Library** | `docker-compose.photo-roms.yml` | Immich, syncthing, retrom |
| **Utilities** | `docker-compose.utils.yml` | Gitea, tandoor, speedtest-tracker, rustdesk, redbot, linkwarden, neolink, iperf3 |
## Core Infrastructure Agents (infra.yml)
### Management & Monitoring
- **Portainer** - Container management UI (port 9443)
- **Homepage** - Service dashboard (port 7575)
- **WUD (WhatsUpDocker)** - Docker image management utility
- **ntopng** - Network traffic monitoring (port 3939)
### Reverse Proxy & Security
- **Nginx Proxy Manager** - SSL termination and reverse proxy (ports 80, 443, 81)
- **Docker Socket Proxy** - Secure Docker API access (port 2375)
- **Newt** - Pangolin tunnel client
## Media & Entertainment Stack
## Media & Entertainment Stack (media.yml)
### Media Server
- **Jellyfin** - Media server with hardware transcoding (port 8096)
@@ -25,6 +39,7 @@ This document provides a comprehensive overview of the Docker infrastructure man
- **qBittorrent** - Torrent client (ports 56881, 7070)
- **Prowlarr** - Indexer manager (port 9696)
- **FlareSolverr** - Cloudflare bypass service (port 8191)
- **Metube** - YouTube video downloader (port 8081)
### Content Management
- **Radarr** - Movie management (port 7878)
@@ -36,7 +51,12 @@ This document provides a comprehensive overview of the Docker infrastructure man
### File Sharing
- **Slskd** - Soulseek file sharing client (ports 5030, 5031, 50300)
## Document & Data Management
### Music & Scrobbling
- **Maloja** - Music scrobbling service (port 42010)
- **Multi-Scrobbler** - Cross-platform scrobbling (port 9078)
- Integrates with Jellyfin and Last.fm
## Document & AI Suite (documents.yml)
### Document Processing
- **Paperless-ngx** - Document management (port 8100)
@@ -45,9 +65,14 @@ This document provides a comprehensive overview of the Docker infrastructure man
- **Stirling-PDF** - PDF manipulation tools (port 8090)
- **OnlyOffice** - Document collaboration (port 8091)
### AI Interface
- **Open WebUI** - LLM interface (port 3000)
## Photo Management & Library (photo-roms.yml)
### Photo Management
- **Immich** - AI-powered photo management (port 2283)
- PostgreSQL with vector search, Redis cache
- PostgreSQL with vector search, Valkey/Redis cache
- Local SSD storage for config and thumbs
### File Synchronization
@@ -55,58 +80,6 @@ This document provides a comprehensive overview of the Docker infrastructure man
- Obsidian vault synchronization
- Multiple shared folders
## AI & Development Services
### AI Infrastructure
- **Open WebUI** - LLM interface (port 3000)
- **LiteLLM** - AI gateway (port 4000)
- PostgreSQL database for configuration
- Supports multiple AI providers
### Development Tools
- **Gitea** - Git service (ports 222, 8418)
- MySQL database
- **Newt** - AI service integration
## Genealogy & Personal Tools
### Family History
- **GrampsWeb Jamie** - Genealogy UI (port 5511)
- **GrampsWeb Helen** - Genealogy UI (port 5512)
- Shared Redis broker
- Separate trees for different family branches
### Finance & Tracking
- **Speedtest Tracker** - Network monitoring (port 8180)
- MariaDB database, automated testing
### Personal Tools
- **Tandoor Recipes** - Self-hosted recipe management
- **Surmai** - Personal flight tracking tool
### Music & Scrobbling
- **Maloja** - Music scrobbling service (port 42010)
- **Multi-Scrobbler** - Cross-platform scrobbling (port 9078)
- Integrates with Jellyfin and Last.fm
## Utilities & External Services
### Remote Access
- **RustDesk** - Remote desktop (host mode)
### Network Tools
- **iperf3-server** - Network performance testing (port 5201)
- **ntopng** - Network traffic monitoring tool (port 3939)
### Browser Workspace
- **Kasm** - Browser isolation workspace (containerized browsing)
### Game Servers
- **Foundry Watcher** - Foundry VTT player monitoring service
- SSH log tailing from Foundry server
- REST API for player status (port 30001)
- MQTT integration for connection events
### ROM Management
- **Retrom** - ROM library management service
- **retrom**: Main ROM service container (port 5111)
@@ -114,29 +87,50 @@ This document provides a comprehensive overview of the Docker infrastructure man
- **retrom-adminer**: Adminer interface for database management (port 8080)
- **retrom-jaeger**: Distributed tracing for performance monitoring
## Utilities Stack (utils.yml)
### Development Tools
- **Gitea** - Git service (ports 222, 8418)
- MySQL database
### Personal Tools
- **Tandoor Recipes** - Self-hosted recipe management (port 8450)
- PostgreSQL database
### Finance & Tracking
- **Speedtest Tracker** - Network monitoring (port 8180)
- MariaDB database, automated testing
### Bookmark Management
- **Linkwarden** - Collaborative bookmark manager (port 3400)
- PostgreSQL database
- Meilisearch for full-text search
### Remote Access
- **RustDesk** - Remote desktop (host mode)
### Home Automation
- **Neolink** - Reolink camera bridge for Frigate/Home Assistant integration
### Discord Bot (OpenCode)
- **Service**: Discord Agent Bot
- **Role**: AI-powered Discord bot using Ollama LLM
- **Container**: `discord-agent`
- **Network**: `internal_net`, `db_net`
- **Configuration**: `/docker/discord-agent/config/agent-config.yaml`
- **Data**: `/docker/discord-agent/data/`
- **LLM**: Ollama (ministral-3:8b) at `http://192.168.0.31:11434`
- **Features**: Discord commands, service integrations, AI chat capabilities
- **Database**: MySQL for conversation persistence
- **Cogs**: Modular architecture with base_cog and integration_cog
### Discord Bots
- **RedBot** - Discord bot with custom cogs (internal_net)
### RedBot
- **Service**: RedBot Discord Bot
- **Role**: Alternative Discord bot with custom cogs
- **Container**: `redbot`
- **Network**: `web_net`
### Network Tools
- **iperf3-server** - Network performance testing (port 5201)
### Other Tools
- **Newt** - AI service integration
## Independent Stacks (separate compose files)
These services are deployed independently and are not part of the main 5-stack split:
| Service | Directory | Role |
|---------|-----------|------|
| **Discord Agent Bot** | `discord-agent/` | AI-powered Discord bot using Ollama LLM |
| **Foundry Watcher** | `foundry-watcher/` | Foundry VTT player monitoring |
| **GrampsWeb Jamie** | `gramps-jamie/` | Genealogy UI (port 5511) |
| **GrampsWeb Helen** | `gramps-helen/` | Genealogy UI (port 5512) |
| **Kasm** | `kasm/` | Browser isolation workspace |
| **LiteLLM** | `litellm/` | AI gateway (port 4000) |
| **MBI Poller** | `mbi-poller/` | MBI data polling service |
## Network Architecture
@@ -146,17 +140,18 @@ This document provides a comprehensive overview of the Docker infrastructure man
- **web_net** - Web-accessible services
- **internal_net** - Internal service communication
Networks are defined in `docker-compose.infra.yml` and referenced as `external: true` by all other stacks.
### Storage Structure
```
/docker/
├── Arrs/ (Media stack: Prowlarr, Radarr, Sonarr, Lidarr, Bazarr, Jellyfin, Jellyseerr)
├── immich/ (Photo management)
├── paperless/ (Document management)
├── litellm/ (AI gateway)
├── discord-agent/ (Discord bot with Ollama LLM integration)
├── discord-agent/ (Independent — Discord bot with Ollama)
├── gitea/ (Git service)
├── gramps-jamie/ (Genealogy)
├── gramps-helen/ (Genealogy)
├── gramps-jamie/ (Independent — Genealogy)
├── gramps-helen/ (Independent — Genealogy)
├── npm/ (Nginx Proxy Manager)
├── qBittorrent/ (Download client)
├── slskd/ (Soulseek client)
@@ -168,19 +163,43 @@ This document provides a comprehensive overview of the Docker infrastructure man
├── retrom/ (ROM library management)
├── wud/ (Docker image management)
├── ntopng/ (Network traffic monitoring)
├── kasm/ (Browser workspace)
├── kasm/ (Independent — Browser workspace)
├── litellm/ (Independent — AI gateway)
├── neolink/ (Reolink camera bridge)
├── linkwarden/ (Bookmark manager)
├── surmai/ (Flight tracking)
├── tandoor/ (Recipe management)
├── foundry-watcher/ (Foundry VTT monitoring)
├── foundry-watcher/ (Independent — Foundry VTT)
├── rustdesk/ (Remote desktop)
├── redbot/ (Discord bot)
├── stirling/ (PDF tools)
└── Various other service configs
```
## Deployment
### Deploy all main stacks
```bash
for f in docker-compose.infra.yml docker-compose.media.yml docker-compose.documents.yml docker-compose.photo-roms.yml docker-compose.utils.yml; do
docker compose -f "$f" up -d
done
```
### Deploy a specific stack
```bash
docker compose -f docker-compose.media.yml up -d
```
### Stop a specific stack
```bash
docker compose -f docker-compose.media.yml down
```
## Key Features & Configuration
### Security
- Container security with `no-new-privileges:true`
- Network segmentation (databases internal-only)
- Network segmentation (databases internal-only via `db_net`)
- Reverse proxy with SSL termination
### Performance
@@ -189,7 +208,7 @@ This document provides a comprehensive overview of the Docker infrastructure man
- Resource limits (Portainer: 512MB RAM)
### Data Management
- Comprehensive backup script included
- Comprehensive backup script included (`backup.sh`)
- Separate volumes for critical data
- NAS storage integration for media files
@@ -197,25 +216,30 @@ This document provides a comprehensive overview of the Docker infrastructure man
- System configured for Pacific/Auckland timezone
- PUID/PGID for proper file permissions
- Extensive environment variable configuration
- Shared `.env` file across all stacks
## Service Dependencies
### Database Services
- MariaDB instances for media stack, tracking, and document services
- PostgreSQL for AI services, photo management, and finance
- Redis for caching and message brokering
- MariaDB: npm, paperless, speedtest-tracker
- PostgreSQL: immich, retrom, tandoor, linkwarden
- MySQL: gitea
- Redis/Valkey: paperless-broker, immich-redis
- Meilisearch: linkwarden
### Network Dependencies
- `infra.yml` must be deployed first (creates shared networks)
- All web services route through Nginx Proxy Manager
- Internal services communicate via internal_net
- Media services isolated on media_net
- Database services on internal-only db_net
- Internal services communicate via `internal_net`
- Media services isolated on `media_net`
- Database services on internal-only `db_net`
## Maintenance Agents
## Maintenance
### Automated Tasks
- **Backup Script** - Regular data backups
- **Backup Script** (`backup.sh`) - Regular data backups
- **Speedtest Tracker** - Automated network testing
- **WUD** - Automatic Docker image update monitoring
### Manual Tasks
- Service monitoring via Portainer
@@ -232,5 +256,6 @@ This document provides a comprehensive overview of the Docker infrastructure man
### Troubleshooting
- Container logs accessible via Portainer
- Network diagnostics via iperf3-server
- Per-stack logs: `docker compose -f docker-compose.<stack>.yml logs -f <service>`
This Docker infrastructure represents a comprehensive self-hosted ecosystem covering media management, document processing, AI services, development tools, and personal productivity applications, all integrated through a reverse proxy and organized with proper network segmentation.
This Docker infrastructure represents a comprehensive self-hosted ecosystem covering media management, document processing, AI services, development tools, and personal productivity applications, all integrated through a reverse proxy and organized into independent compose stacks with proper network segmentation.