Jamie/Homelab
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Split monolithic compose into 5 independent stacks

Browse Source 
- docker-compose.infra.yml: core infrastructure (portainer, npm, homepage, wud, etc.)
- docker-compose.media.yml: media stack (arrs, jellyfin, qbittorrent, scrobbling)
- docker-compose.documents.yml: paperless-ngx, onlyoffice, stirling, open-webui
- docker-compose.photo-roms.yml: immich, syncthing, retrom
- docker-compose.utils.yml: gitea, tandoor, speedtest, linkwarden, rustdesk, etc.

Each stack has its own project name (docker-infra, docker-media, etc.) to prevent
orphan warnings. Networks defined in infra.yml, referenced as external by others.
Original preserved as docker-compose.full.yaml.bak.
Updated .gitignore, README, AGENTS.md, and RESTORE.md to reflect new structure.
This commit is contained in:
KansaiGaijin
2026-05-14 22:58:11 +12:00
parent 76e99f9df9
commit b2f4d37f19
10 changed files with 1675 additions and 601 deletions
Download Patch File Download Diff File Expand all files Collapse all files

325
RESTORE.md
View File

@@ -4,274 +4,187 @@ This guide walks you through restoring your Docker infrastructure from a backup.
## Architecture Alignment
The architecture inventory is documented in README.md as two tables under Architecture:
- Internal Docker Services (Service | Category | Web UI Port)
- External Non-Docker Services (Service | Category | IP | Web UI Port)
The infrastructure is split into 5 compose files (stacks) sharing a common `.env`:
If you are restoring, follow the standard restoration steps below, and refer to README.md for the exact service inventory and ports as implemented in your environment.
| Stack | File | Purpose |
|-------|------|---------|
| Infrastructure | `docker-compose.infra.yml` | Portainer, NPM, homepage, WUD, ntopng, newt, dockerproxy |
| Media | `docker-compose.media.yml` | arr stack, Jellyfin, qBittorrent, slskd, metube, scrobbling |
| Documents | `docker-compose.documents.yml` | Paperless-ngx, AI, OnlyOffice, Stirling-PDF, Open WebUI |
| Photo & ROM | `docker-compose.photo-roms.yml` | Immich, Syncthing, Retrom |
| Utilities | `docker-compose.utils.yml` | Gitea, Tandoor, Speedtest, RustDesk, RedBot, Linkwarden, Neolink, iperf3 |
Refer to README.md for the exact service inventory and ports.
## Prerequisites
- Fresh system with Docker and Docker Compose installed
- Backup archive (`docker-backup-YYYYMMDD_HHMMSS.tar.gz`)
- Fresh system with Docker and Docker Compose V2 installed
- Backup archive containing `/docker/` configs
- Access to NAS storage (if applicable)
- Root or sudo access
## Discord Agent Service
### Step 1: Create Discord Agent Directory
```bash
mkdir -p /docker/discord-agent/config /docker/discord-agent/data/logs /docker/discord-agent/data/database /docker/discord-agent/data/cache /docker/discord-agent/cogs /docker/discord-agent/scripts
```
### Step 2: Copy Configuration Files
```bash
# Copy agent configuration
cp agent-config.yaml /docker/discord-agent/config/
cp permissions.json /docker/discord-agent/config/
# Copy scripts
cp startup.sh /docker/discord-agent/scripts/
cp health_check.sh /docker/discord-agent/scripts/
```
### Step 3: Copy Python Files
```bash
cp discord_agent.py /docker/discord-agent/
cp base_cog.py /docker/discord-agent/cogs/
cp integration_cog.py /docker/discord-agent/cogs/
```
### Step 4: Copy Requirements
```bash
cp requirements.txt /docker/discord-agent/
cp Dockerfile /docker/discord-agent/
```
### Step 5: Update Environment Variables
Add these to your `.env` file:
```bash
# Discord Agent
DISCORD_BOT_TOKEN=your_discord_bot_token_here
OLLAMA_ENDPOINT=http://192.168.0.31:11434
OLLAMA_MODEL=ministral-3:8b
TZ=Pacific/Auckland
```
### Step 6: Update Configuration
Edit `/docker/discord-agent/config/agent-config.yaml` with your specific settings:
```yaml
discord:
token: ${DISCORD_BOT_TOKEN}
prefix: "!"
status: "AI Assistant | !help"
ollama:
endpoint: "${OLLAMA_ENDPOINT:http://192.168.0.31:11434}"
model: "${OLLAMA_MODEL:ministral-3:8b}"
parameters:
temperature: 0.7
top_p: 0.9
top_k: 40
timeout: 60
```
### Step 7: Build and Start Service
```bash
# Build the Discord agent image
docker compose build discord-agent
# Start the service
docker compose up -d discord-agent
# Verify the service is running
docker compose logs discord-agent
```
### Step 8: Verify Integration
```bash
# Check Discord bot connection
docker exec discord-agent python3 -c "import discord; print('Discord library available')"
# Check Ollama connection
curl http://192.168.0.31:11434/api/tags
# Check MySQL database
docker compose exec agent-db mysql -u agent -pagent -e "SHOW DATABASES;"
```
### Step 9: Test Bot Functionality
- The bot should appear in your Discord server
- Test with `!help` command to verify functionality
- Test with `!agent <message>` to verify Ollama integration
- Test with `!status` to check bot status
### Troubleshooting
- If bot doesn't start, check logs: `docker compose logs discord-agent`
- Verify DISCORD_BOT_TOKEN is set correctly in `.env` file
- Ensure Ollama is running and accessible at the configured endpoint
- Verify internal_net and db_net networks are available
- Check that MySQL database agent-db is running and healthy
## Step 2: Restore Directory Structure
## Step 1: Restore Directory Structure
```bash
# Copy the directory structure script
cp directory_structure.sh /opt/docker-compose/
cd /opt/docker-compose/
# Create the root docker directory
mkdir -p /docker
cd /docker
# Make it executable and run
chmod +x directory_structure.sh
./directory_structure.sh
```
## Step 3: Restore Docker Compose Configuration
```bash
# Copy main configuration files
cp /path/to/backup/docker-compose.yaml .
# Copy configuration files from backup
cp /path/to/backup/.env .
cp /path/to/backup/.gitignore .
cp /path/to/backup/README.md .
cp /path/to/backup/AGENTS.md .
cp /path/to/backup/RESTORE.md .
cp /path/to/backup/backup.sh .
# Copy all compose files
cp /path/to/backup/docker-compose.infra.yml .
cp /path/to/backup/docker-compose.media.yml .
cp /path/to/backup/docker-compose.documents.yml .
cp /path/to/backup/docker-compose.photo-roms.yml .
cp /path/to/backup/docker-compose.utils.yml .
# Copy the original full backup (for reference)
cp /path/to/backup/docker-compose.full.yaml.bak .
# IMPORTANT: Edit .env file with new system-specific values
nano .env
# Verify the configuration
docker compose config
```
## Step 4: Restore Service Configurations
## Step 2: Restore Service Configuration Directories
```bash
# Copy service configurations back to /docker
rsync -av /path/to/backup/docker/ /docker/
# Create required subdirectories that may be missing
mkdir -p /docker/Arrs/{Prowlarr,Radarr,Sonarr,Lidarr,Bazarr,Jellyfin,Seerr}/config
mkdir -p /docker/{immich,qBittorrent,paperless,stirling,syncthing,gitea}
mkdir -p /docker/{speedtest-tracker,rustdesk,redbot,maloja,scrobble}
mkdir -p /docker/{retrom/config,retrom/data}
mkdir -p /docker/{wud,ntopng,slskd,npm/data,npm/letsencrypt,npm/mysql}
mkdir -p /docker/{Homepage/config,neolink,tandoor,linkwarden}
# Set correct permissions
PUID=$(id -u)
PGID=$(id -g)
sudo chown -R $PUID:$PGID /docker
```
## Step 5: Restore Docker Volumes
## Step 3: Restore Docker Volumes
For each volume backup in `/path/to/backup/volumes/`:
For each volume backup:
```bash
# Create the volume if it doesn't exist
docker volume create VOLUME_NAME
# Restore the volume data
docker run --rm \
-v VOLUME_NAME:/volume \
-v /path/to/backup/volumes:/backup \
alpine \
tar xzf /backup/VOLUME_NAME.tar.gz -C /volume
```
Example for specific volumes:
```bash
# Restore Portainer data
docker volume create portainer_data
docker run --rm \
-v portainer_data:/volume \
-v /path/to/backup/volumes:/backup \
alpine \
tar xzf /backup/portainer_data.tar.gz -C /volume
# Restore Open WebUI data
docker volume create open-webui
docker run --rm \
-v open-webui:/volume \
-v /path/to/backup/volumes:/backup \
alpine \
tar xzf /backup/open-webui.tar.gz -C /volume
docker volume create docker_dbdata
docker volume create docker_aidata
docker volume create docker_onlyoffice
docker volume create docker_redisdata
# Repeat for other volumes...
# Restore volume data
for volume in portainer_data open-webui docker_dbdata docker_aidata docker_onlyoffice docker_redisdata; do
if [ -f "/path/to/backup/volumes/${volume}.tar.gz" ]; then
docker run --rm \
-v ${volume}:/volume \
-v /path/to/backup/volumes:/backup \
alpine \
tar xzf /backup/${volume}.tar.gz -C /volume
fi
done
```
## Step 6: Start Database Services First
## Step 4: Start Infrastructure First
```bash
# Start only database services
docker compose up -d paperless-db immich-postgres litellm-postgres gitea-db speedtest-db npm-db
# Start infrastructure (creates shared networks: media_net, db_net, web_net, internal_net)
docker compose -f docker-compose.infra.yml up -d
# Verify networks were created
docker network ls | grep -E "media_net|db_net|web_net|internal_net"
```
## Step 5: Start Database Services
```bash
# Start database-dependent stacks
docker compose -f docker-compose.documents.yml up -d paperless-db paperless-broker
docker compose -f docker-compose.photo-roms.yml up -d immich-postgres immich-redis retrom-db
docker compose -f docker-compose.utils.yml up -d gitea-db speedtest-db tandoor_db linkwarden-db meilisearch
# Wait for databases to be healthy
docker compose ps
# Check logs for any errors
docker compose logs -f paperless-db
# Press Ctrl+C to exit logs
docker compose -f docker-compose.documents.yml ps
docker compose -f docker-compose.utils.yml ps
```
## Step 7: Restore Database Dumps
## Step 6: Restore Database Dumps (if available)
### Paperless MariaDB
```
# Copy SQL file into container
docker cp /path/to/backup/database-dumps/paperless.sql paperless-db:/tmp/
# Import the database
```bash
docker exec -i paperless-db mysql -u root -p"${PAPERLESS_DB_ROOT_PASSWORD}" paperless < /path/to/backup/database-dumps/paperless.sql
```
### Immich PostgreSQL
```
```bash
docker exec -i immich_postgres psql -U postgres immich < /path/to/backup/database-dumps/immich.sql
```
### LiteLLM PostgreSQL
```
docker exec -i litellm-postgres psql -U litellm litellm_db < /path/to/backup/database-dumps/litellm.sql
```
### Gitea MySQL
```
```bash
docker exec -i gitea-db mysql -u root -pgitea gitea < /path/to/backup/database-dumps/gitea.sql
```
### Speedtest Tracker MariaDB
```
```bash
docker exec -i speedtest-db mysql -u root -p"${SPEEDTEST_DB_PASSWORD}" speedtest < /path/to/backup/database-dumps/speedtest.sql
```
## Step 8: Mount NAS Storage (if applicable)
### Tandoor PostgreSQL
```bash
docker exec -i tandoor_DB psql -U "${TANDOOR_POSTGRES_USER}" "${TANDOOR_POSTGRES_DB}" < /path/to/backup/database-dumps/tandoor.sql
```
## Step 7: Mount NAS Storage (if applicable)
```bash
# Create mount point
sudo mkdir -p /mnt/Nas-Storage
sudo mkdir -p /mnt/nas-storage
# Add to /etc/fstab for permanent mounting
# Example for NFS:
# nas-server:/volume1/data /mnt/Nas-Storage nfs defaults 0 0
# Example for CIFS/SMB:
# //nas-server/data /mnt/Nas-Storage cifs credentials=/root/.smbcredentials,uid=1000,gid=1000 0 0
# nas-server:/volume1/data /mnt/nas-storage nfs defaults 0 0
# Mount immediately
sudo mount -a
# Verify mount
df -h /mnt/Nas-Storage
df -h /mnt/nas-storage
```
## Step 9: Start All Services
## Step 8: Start All Services
```bash
# Start all services
docker compose up -d
# Deploy all stacks
for f in docker-compose.infra.yml docker-compose.media.yml docker-compose.documents.yml docker-compose.photo-roms.yml docker-compose.utils.yml; do
docker compose -f "$f" up -d
done
# Watch the startup process
docker compose logs -f
# Watch the startup process for a specific stack
docker compose -f docker-compose.media.yml logs -f
# Check service health
docker compose ps
# Check service health across all stacks
for f in docker-compose.*.yml; do
if [ "$f" != "docker-compose.full.yaml.bak" ]; then
echo "=== $f ==="
docker compose -f "$f" ps
fi
done
```
## Step 10: Verify Services
Go through each service and verify it's working correctly:
## Step 9: Verify Services
### Check Web Interfaces
- Homepage: http://your-server:7575
@@ -279,9 +192,11 @@ Go through each service and verify it's working correctly:
- Jellyfin: http://your-server:8096
- Paperless: http://your-server:8100
- Immich: http://your-server:2283
- Gitea: http://your-server:8418
- Speedtest: http://your-server:8180
### Verify Databases
```
```bash
# Paperless
docker exec paperless-db mysql -u root -p"${PAPERLESS_DB_ROOT_PASSWORD}" -e "SELECT COUNT(*) FROM paperless.documents_document;"
@@ -289,9 +204,35 @@ docker exec paperless-db mysql -u root -p"${PAPERLESS_DB_ROOT_PASSWORD}" -e "SEL
docker exec immich_postgres psql -U postgres -d immich -c "SELECT COUNT(*) FROM assets;"
```
... (rest of file continues)
## Step 10: Restore Independent Stacks (if applicable)
These services have their own compose files and are restored separately:
```bash
# Discord Agent Bot
cd /docker/discord-agent
docker compose up -d
# LiteLLM
cd /docker/litellm
docker compose up -d
# Gramps genealogy
cd /docker/gramps-jamie
docker compose up -d
cd /docker/gramps-helen
docker compose up -d
# Other independent stacks
cd /docker/kasm && docker compose up -d
cd /docker/foundry-watcher && docker compose up -d
```
## Architecture Alignment
- This section connects to the Architecture two-table layout described in README.md.
- This section connects to the Architecture table described in README.md.
- Each compose file corresponds to a functional domain in the architecture.
- Networks are created by `infra.yml` and shared across all other stacks.
**Last Updated**: December 2025
---
**Last Updated**: May 2026