Jamie/Homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Split monolithic compose into 5 independent stacks

Browse Source 
- docker-compose.infra.yml: core infrastructure (portainer, npm, homepage, wud, etc.)
- docker-compose.media.yml: media stack (arrs, jellyfin, qbittorrent, scrobbling)
- docker-compose.documents.yml: paperless-ngx, onlyoffice, stirling, open-webui
- docker-compose.photo-roms.yml: immich, syncthing, retrom
- docker-compose.utils.yml: gitea, tandoor, speedtest, linkwarden, rustdesk, etc.

Each stack has its own project name (docker-infra, docker-media, etc.) to prevent
orphan warnings. Networks defined in infra.yml, referenced as external by others.
Original preserved as docker-compose.full.yaml.bak.
Updated .gitignore, README, AGENTS.md, and RESTORE.md to reflect new structure.
This commit is contained in:
KansaiGaijin
2026-05-14 22:58:11 +12:00
parent 76e99f9df9
commit b2f4d37f19
10 changed files with 1675 additions and 601 deletions
Download Patch File Download Diff File Expand all files Collapse all files

371
README.md
View File

@@ -2,101 +2,62 @@
## Overview
This infrastructure manages a comprehensive self-hosted environment including media management (Jellyfin, *arr stack), document management (Paperless-ngx), AI services (Open WebUI, LiteLLM), photo management (Immich), genealogy tools (GrampsWeb), and various utilities.
This infrastructure manages a comprehensive self-hosted environment including media management (Jellyfin, *arr stack), document management (Paperless-ngx), AI services (Open WebUI), photo management (Immich), and various utilities.
The deployment is split into **5 independent compose stacks** sharing a common `.env` file and Docker networks.
## Compose Stacks
| Stack | File | Services |
|-------|------|----------|
| **Infrastructure** | `docker-compose.infra.yml` | portainer, ntopng, wud, homepage, dockerproxy, newt, npm, npm-db |
| **Media** | `docker-compose.media.yml` | prowlarr, flaresolverr, qbittorrent, radarr, sonarr, lidarr, bazarr, seerr, jellyfin, slskd, metube, maloja, multi-scrobbler |
| **Documents** | `docker-compose.documents.yml` | onlyoffice, paperless-ngx (db, broker, tika, gotenberg, webserver, ai), stirling-pdf, open-webui |
| **Photo & ROM Library** | `docker-compose.photo-roms.yml` | immich (server, postgres, ml, redis), syncthing, retrom (db, adminer, service, jaeger) |
| **Utilities** | `docker-compose.utils.yml` | gitea (server + db), tandoor (db + web), speedtest-tracker (app + db), rustdesk (hbbs + hbbr), redbot, iperf3-server, neolink, linkwarden (app + db + meilisearch) |
## Architecture
### Internal Docker Services
| Service Categories | Service | Role / Component | Web UI Port |
|---|---|---|---|
| Media & Arr Stack | Jellyfin | Media server with hardware transcoding | 8096 |
| Media & Arr Stack | Jellyseerr | Media request management | 5055 |
| Media & Arr Stack | Radarr | Movie management | 7878 |
| Media & Arr Stack | Sonarr | TV show management | 8989 |
| Media & Arr Stack | Lidarr | Music management | 8686 |
| Media & Arr Stack | Bazarr | Subtitle management | 6767 |
| Media & Arr Stack | Prowlarr | Indexer manager | 9696 |
| Media & Arr Stack | FlareSolverr | Cloudflare bypass service | 8191 |
| Media & Arr Stack | qBittorrent | Download client | 7070 |
| Data Management | Paperless | Document management interface | 8100 |
| Data Management | Paperless-AI | AI-powered document processing | - |
| Data Management | Stirling-PDF | PDF manipulation tools | 8090 |
| Data Management | OnlyOffice | Document collaboration server | 8091 |
| Local AI | Open WebUI | LLM interface | 3000 |
| Local AI | LiteLLM | AI gateway | 4000 |
| Data Management | Immich | Photo management | 2283 |
| Data Management | Syncthing | File synchronization | 8384 |
| Management & Infrastructure | Gitea | Git service | 8418 |
| Data Management | GrampsWeb Jamie | Genealogy UI | 5511 |
| Data Management | GrampsWeb Helen | Genealogy UI | 5512 |
| Data Management | Speedtest Tracker | Network performance | 8180 |
| Management & Infrastructure | NPM Admin | NPM admin UI | 81 |
| Data Management | Maloja | Music scrobbling | 42010 |
| Data Management | Multi-Scrobbler | Cross-platform scrobbler | 9078 |
| Management & Infrastructure | Portainer | Container management UI | 9443 |
| Management & Infrastructure | Homepage | Service dashboard | 7575 |
| Management & Infrastructure | WUD | Docker image management | 3000 |
| Management & Infrastructure | ntopng | Network traffic monitoring | 3939 |
| Management & Infrastructure | Retrom | ROM library management | 5111 |
| Management & Infrastructure | iperf3-server | Network performance testing | 5201 |
| Management & Infrastructure | Foundry Watcher | Foundry VTT player monitoring | 30001 |
| Discord Bots | Discord Agent Bot | AI-powered Discord bot with Ollama | - |
| Discord Bots | RedBot | Alternative Discord bot | - |
### Discord Agent Integration
### Discord Bot Service
- **Service**: Discord Agent Bot
- **Role**: AI-powered Discord bot using Ollama LLM
- **Container**: `discord-agent`
- **Network**: `internal_net`, `db_net`
- **Web UI**: Discord (no direct web interface)
- **Configuration**: `/docker/discord-agent/config/agent-config.yaml`
- **Data**: `/docker/discord-agent/data/`
### Features
- **AI Integration**: Uses Ollama LLM (ministral-3:8b) at `http://192.168.0.31:11434`
- **Modular Architecture**: Cogs for service integrations
- **Database Support**: MySQL for conversation persistence
- **Caching**: Redis for improved performance (configurable)
- **Security**: Discord role-based permissions
- **Logging**: Configurable logging levels (default: INFO)
- **Rate Limiting**: Built-in rate limiting for API calls
### Available Commands
- `!agent` - Chat with the AI assistant
- `!status` - Check bot status and latency
- `!help` - Show available commands
- Service integration commands (configurable)
### Configuration
- **Main Config**: `config/agent-config.yaml` (YAML-based with env var substitution)
- **Discord Token**: Sourced from `DISCORD_BOT_TOKEN` environment variable
- **Ollama Config**: Model, endpoint, temperature, top_p, top_k parameters
- **Integrations**: Configurable service integrations in YAML
### Environment Variables
```
DISCORD_BOT_TOKEN=your_discord_bot_token_here
OLLAMA_ENDPOINT=http://192.168.0.31:11434
OLLAMA_MODEL=ministral-3:8b
TZ=Pacific/Auckland
```
### Bot Architecture
- **Main Bot**: `discord_agent.py` - Core Discord bot implementation
- **Base Cog**: `cogs/base_cog.py` - Base functionality and commands
- **Integration Cog**: `cogs/integration_cog.py` - Service integrations
- **Config**: YAML-based configuration with environment variable substitution
- **Database**: MySQL with asyncpg driver for conversation history
### Security
- Runs on internal networks only
- Discord role-based access control
- Configurable rate limiting to prevent abuse
- No direct database access from external networks
| Stack | Service | Role | Port |
|-------|---------|------|------|
| Infrastructure | Nginx Proxy Manager | SSL termination & reverse proxy | 80, 443, 81 |
| Infrastructure | Portainer | Container management UI | 9443 |
| Infrastructure | Homepage | Service dashboard | 7575 |
| Infrastructure | WUD | Docker image update monitoring | 3666 |
| Infrastructure | ntopng | Network traffic monitoring | 3939 |
| Infrastructure | Docker Socket Proxy | Secure Docker API access | 2375 |
| Infrastructure | Newt | Pangolin tunnel client | - |
| Media | Jellyfin | Media server (HW transcoding) | 8096 |
| Media | Jellyseerr (seerr) | Media request management | 5055 |
| Media | Radarr | Movie management | 7878 |
| Media | Sonarr | TV show management | 8989 |
| Media | Lidarr | Music management | 8686 |
| Media | Bazarr | Subtitle management | 6767 |
| Media | Prowlarr | Indexer manager | 9696 |
| Media | FlareSolverr | Cloudflare bypass | 8191 |
| Media | qBittorrent | Download client | 7070 |
| Media | Slskd | Soulseek file sharing | 5030, 5031 |
| Media | Metube | YouTube downloader | 8081 |
| Media | Maloja | Music scrobbling | 42010 |
| Media | Multi-Scrobbler | Cross-platform scrobbler | 9078 |
| Documents | Paperless-ngx | Document management | 8100 |
| Documents | Paperless-AI | AI document processing | 3040 |
| Documents | Stirling-PDF | PDF manipulation | 8090 |
| Documents | OnlyOffice | Document collaboration | 8091 |
| Documents | Open WebUI | LLM interface | 3000 |
| Photo & ROM | Immich | Photo management | 2283 |
| Photo & ROM | Syncthing | File synchronization | 8384 |
| Photo & ROM | Retrom | ROM library management | 5111 |
| Utilities | Gitea | Git service | 8418 |
| Utilities | Tandoor Recipes | Recipe management | 8450 |
| Utilities | Speedtest Tracker | Network monitoring | 8180 |
| Utilities | RustDesk | Remote desktop (host mode) | - |
| Utilities | RedBot | Discord bot | - |
| Utilities | iperf3-server | Network performance test | 5201 |
| Utilities | Neolink | Reolink camera bridge | 8554 |
| Utilities | Linkwarden | Bookmark manager | 3400 |
### Network Segmentation
@@ -105,47 +66,95 @@ TZ=Pacific/Auckland
- **web_net**: Web-accessible services
- **internal_net**: Internal service communication
Networks are created by `infra.yml` and referenced as external by all other stacks.
## Storage Structure
### Local Docker Configs (`/docker/`)
```
/docker/
├── Homepage/
├── Arrs/ (Prowlarr, Radarr, Sonarr, Lidarr, Bazarr, Jellyfin, Jellyseerr)
├── qBittorrent/
├── discord-agent/ (Discord bot with Ollama LLM)
├── paperless/
├── stirling/
├── Homepage/
├── immich/
├── syncthing/
├── paperless/
├── discord-agent/ (Separate stack)
├── gitea/
├── gramps-jamie/
├── gramps-helen/
├── speedtest-tracker/
├── rustdesk/
├── redbot/
├── gramps-jamie/ (Separate stack)
├── gramps-helen/ (Separate stack)
├── kasm/ (Separate stack)
├── foundry-watcher/ (Separate stack)
├── litellm/ (Separate stack)
├── linkwarden/
├── maloja/
├── scrobble/
├── litellm/
├── retrom/
├── wud/
├── neolink/
├── npm/
├── ntopng/
├── qBittorrent/
├── redbot/
├── retrom/
├── rustdesk/
├── scrobble/ (Multi-scrobbler)
├── slskd/
└── npm/
├── speedtest-tracker/
├── stirling/
├── surmai/
├── syncthing/
├── tandoor/
├── wud/
└── .env (shared environment variables)
```
### NAS Storage (`/mnt/Nas-Storage/data/`)
### NAS Storage (`/mnt/nas-storage/data/`)
```
/mnt/Nas-Storage/data/
/mnt/nas-storage/data/
├── media/
│ ├── movies/
│ ├── tv/
│ ├── music/
│ └── images/
│ └── romms/
└── torrents/
├── metube/
└── soulsync/
```
## Deployment
### First-time deploy (or after network cleanup)
```bash
# Deploy infrastructure first (creates shared networks)
docker compose -f docker-compose.infra.yml up -d
# Then deploy remaining stacks in any order
docker compose -f docker-compose.media.yml up -d
docker compose -f docker-compose.documents.yml up -d
docker compose -f docker-compose.photo-roms.yml up -d
docker compose -f docker-compose.utils.yml up -d
```
### Deploy all stacks
```bash
for f in docker-compose.*.yml; do
if [ "$f" != "docker-compose.full.yaml.bak" ]; then
docker compose -f "$f" up -d
fi
done
```
### Stop/start individual stacks
```bash
docker compose -f docker-compose.media.yml down
docker compose -f docker-compose.media.yml up -d
```
### View logs
```bash
docker compose -f docker-compose.media.yml logs -f jellyfin
```
## Prerequisites
### Required Environment Variables (.env file)
@@ -159,7 +168,6 @@ TZ=Pacific/Auckland
# URLs
JELLYFIN_URL=https://your-jellyfin-domain.com
PAPERLESS_URL=https://your-paperless-domain.com
WYGIWYH_URL=https://your-wygiwyh-domain.com
HOMEPAGE_ALLOWED_HOSTS=your-homepage-domain.com
# Nginx Proxy Manager
@@ -175,10 +183,6 @@ PAPERLESS_SECRET_KEY=your_secret_key
# Immich
IMMICH_POSTGRES_PASSWORD=your_secure_password
# LiteLLM
GROQ_API_KEY=your_groq_api_key
LITELLM_MASTER_KEY=your_master_key
# Slskd
SLSKD_USERNAME=your_username
SLSKD_PASSWORD=your_secure_password
@@ -189,15 +193,35 @@ SPEEDTEST_DB_USER=speedtest
SPEEDTEST_DB_PASSWORD=your_secure_password
SPEEDTEST_APP_KEY=base64:your_generated_key
# WYGIWYH
WYGIWYH_DB_DATABASE=wygiwyh
WYGIWYH_DB_USER=wygiwyh
WYGIWYH_DB_PASSWORD=your_secure_password
WYGIWYH_SECRET_KEY=your_secret_key
WYGIWYH_ALLOWED_HOSTS=localhost,127.0.0.1,your-domain.com
# RedBot
REDBOT_TOKEN=your_discord_bot_token
# Tandoor
TANDOOR_SECRET_KEY=your_secret_key
TANDOOR_POSTGRES_DB=tandoor
TANDOOR_POSTGRES_USER=tandoor
TANDOOR_POSTGRES_PASSWORD=your_secure_password
# Linkwarden
LINKWARDEN_DB_PASSWORD=your_secure_password
NEXTAUTH_SECRET=your_secret
NEXTAUTH_URL=https://your-linkwarden-domain.com
# WUD
WUD_REGISTRY_HUB_0_LOGIN=your_dockerhub_username
WUD_REGISTRY_HUB_0_TOKEN=your_dockerhub_token
# OnlyOffice
OO_JWT_SECRET=your_secret
# Maloja
MALOJA_FORCE_PASSWORD=your_password
MALOJA_URL=http://maloja:42010
MALOJA_API_KEY=your_key
# Jellyfin
JELLYFIN_APIKEY_MS=your_api_key_for_multiscrobbler
JELLYFIN_USER=your_username
```
### System Requirements
@@ -209,27 +233,15 @@ REDBOT_TOKEN=your_discord_bot_token
## Installation
1. Clone/Download this repository
2. Create required directories
1. Clone this repository
2. Create `.env` from `.env.example` and fill in values
3. Run infrastructure:
```bash
# See directory_structure.sh for automated setup
chmod +x directory_structure.sh
./directory_structure.sh
docker compose -f docker-compose.infra.yml up -d
```
3. Create .env file
4. Deploy desired stacks:
```bash
cp .env.example .env
# Edit .env with your values
nano .env
```
4. Set correct permissions
```bash
sudo chown -R $PUID:$PGID /docker
sudo chown -R $PUID:$PGID /mnt/Nas-Storage/data
```
5. Start services
```bash
docker compose up -d
docker compose -f docker-compose.media.yml up -d
```
## Maintenance
@@ -237,34 +249,26 @@ REDBOT_TOKEN=your_discord_bot_token
### Backup Strategy
**Critical Data to Backup:**
- `/docker/` - All service configurations
- Database volumes (see docker-compose.yaml)
- `.env` file (store securely, contains secrets)
- `/docker/` — all service configurations
- Named volumes: `portainer_data`, `docker_dbdata`, `open-webui`
- `.env` file (contains secrets)
**Optional (can be regenerated):**
- Media files on NAS
### Updates
To manually update containers:
```
docker compose pull
docker compose up -d
```
### Logs
View logs for any service:
```
docker compose logs -f [service_name]
```bash
for f in docker-compose.infra.yml docker-compose.media.yml docker-compose.documents.yml docker-compose.photo-roms.yml docker-compose.utils.yml; do
docker compose -f "$f" pull
docker compose -f "$f" up -d
done
```
### Resource Limits
- Portainer: 512MB RAM limit
- Log rotation: 10MB max, 3 files
- Log rotation: 10MB max, 3 files (on services with logging config)
## Hardware Acceleration
@@ -273,71 +277,68 @@ Jellyfin is configured for Intel GPU transcoding:
- Group: `104` (render group)
Verify GPU access:
```
```bash
ls -l /dev/dri/renderD128
```
## Security Considerations
1. Secrets Management: Store `.env` securely, never commit to version control
2. Network Segmentation: Database network is internal-only
2. Network Segmentation: Database network is internal-only (`db_net`)
3. Container Security: `no-new-privileges:true` on supported services
4. Reverse Proxy: Use NPM for SSL termination and authentication
5. Updates: Regularly update containers with `docker compose pull && docker compose up -d`
5. Updates: Regularly update containers as shown above
## Troubleshooting
### Service won't start
```
docker compose logs [service_name]
docker compose restart [service_name]
```bash
docker compose -f docker-compose.media.yml logs service_name
docker compose -f docker-compose.media.yml restart service_name
```
### Database connection issues
```
# Check database is healthy
docker compose ps
# Verify network connectivity
docker compose exec [service] ping [db_service]
```bash
docker compose -f docker-compose.documents.yml ps
docker compose -f docker-compose.documents.yml exec paperless-webserver ping paperless-db
```
### Permission errors
```
# Verify ownership
ls -la /docker/[service]/
# Fix if needed
sudo chown -R $PUID:$PGID /docker/[service]/
```bash
ls -la /docker/service_name/
sudo chown -R $PUID:$PGID /docker/service_name/
```
### Storage full
```
# Check Docker disk usage
```bash
docker system df
# Clean up unused resources
docker system prune -a
```
## Contributing Services
To add new services:
1. Add service definition to appropriate section
1. Add service definition to the appropriate compose file
2. Assign to correct network(s)
3. Add volume mounts under `/docker/[service]/`
4. Update this README with service description
5. Add required environment variables to `.env.example`
## Independent Stacks
The following services have their own compose files (not part of the main split):
- **discord-agent/** — Discord Agent Bot
- **foundry-watcher/** — Foundry VTT monitoring
- **gramps-jamie/** — Genealogy (Jamie)
- **gramps-helen/** — Genealogy (Helen)
- **kasm/** — Browser workspace
- **litellm/** — AI gateway
- **mbi-poller/** — MBI poller service
## License
This is a personal infrastructure setup. Adapt as needed for your use case.
## Support
For issues with specific services, consult their official documentation:
- Jellyfin: https://jellyfin.org/docs/
- Paperless-ngx: https://docs.paperless-ngx.com/
- Immich: https://immich.app/docs/
- And so on...
---
**Last Updated**: January 2026
**Last Updated**: May 2026