diff --git a/.gitignore b/.gitignore index d99af5d..5b6de06 100644 --- a/.gitignore +++ b/.gitignore @@ -7,6 +7,7 @@ !.gitignore !README.md !RESTORE.md +!AGENTS.md # Never track actual secrets .env diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..004069f --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,211 @@ +# AGENTS - Docker Infrastructure Documentation + +## Overview +This document provides a comprehensive overview of the Docker infrastructure managed in `/docker/`. This self-hosted ecosystem contains 30+ services organized into functional categories, all integrated through a reverse proxy and secured with proper network segmentation. + +## Core Infrastructure Agents + +### Management & Monitoring +- **Glances** - System monitoring dashboard with hardware access +- **Portainer** - Container management UI (port 9443) +- **Watchtower** - Automated container updates (daily polling) +- **Homepage** - Service dashboard (port 7575) + +### Reverse Proxy & Security +- **Nginx Proxy Manager** - SSL termination and reverse proxy (ports 80, 443, 81) +- **Docker Socket Proxy** - Secure Docker API access (port 2375) + +## Media & Entertainment Stack + +### Media Server +- **Jellyfin** - Media server with hardware transcoding (port 8096) + - Intel GPU acceleration via render device + - NAS storage integration + +### Download & Index Management +- **qBittorrent** - Torrent client (ports 56881, 7070) +- **Prowlarr** - Indexer manager (port 9696) +- **FlareSolverr** - Cloudflare bypass service (port 8191) + +### Content Management +- **Radarr** - Movie management (port 7878) +- **Sonarr** - TV show management (port 8989) +- **Lidarr** - Music management (port 8686) +- **Bazarr** - Subtitle management (port 6767) +- **Jellyseerr** - Media request system (port 5055) + +### File Sharing +- **Slskd** - Soulseek file sharing client (ports 5030, 5031, 50300) + +## Document & Data Management + +### Document Processing +- **Paperless-ngx** - Document management (port 8100) + - MariaDB database, Redis broker, Tika text extraction, Gotenberg PDF processing +- **Stirling-PDF** - PDF manipulation tools (port 8090) +- **OnlyOffice** - Document collaboration (port 8091) + +### Photo Management +- **Immich** - AI-powered photo management (port 2283) + - PostgreSQL with vector search, Redis cache + - Local SSD storage for config and thumbs + +### File Synchronization +- **Syncthing** - File synchronization (ports 21027, 22000, 8384) + - Obsidian vault synchronization + - Multiple shared folders + +## AI & Development Services + +### AI Infrastructure +- **Open WebUI** - LLM interface (port 3000) +- **LiteLLM** - AI gateway (port 4000) + - PostgreSQL database for configuration + - Supports multiple AI providers + +### Development Tools +- **Gitea** - Git service (ports 222, 8418) + - MySQL database +- **Newt** - AI service integration + +## Genealogy & Personal Tools + +### Family History +- **GrampsWeb Jamie** - Genealogy UI (port 5511) +- **GrampsWeb Helen** - Genealogy UI (port 5512) + - Shared Redis broker + - Separate trees for different family branches + +### Finance & Tracking +- **WYGIWYH** - Financial tracking (port 9008) + - PostgreSQL database +- **Speedtest Tracker** - Network monitoring (port 8180) + - MariaDB database, automated testing + +### Music & Scrobbling +- **Maloja** - Music scrobbling service (port 42010) +- **Multi-Scrobbler** - Cross-platform scrobbling (port 9078) + - Integrates with Jellyfin and Last.fm + +## Utilities & External Services + +### Remote Access +- **RustDesk** - Remote desktop (host mode) + +### Network Tools +- **iperf3-server** - Network performance testing (port 5201) + +### Discord Bot (OpenCode) +- **Service**: Discord Agent Bot +- **Role**: AI-powered Discord bot using Ollama LLM +- **Container**: `discord-agent` +- **Network**: `internal_net`, `db_net` +- **Configuration**: `/docker/discord-agent/config/agent-config.yaml` +- **Data**: `/docker/discord-agent/data/` +- **LLM**: Ollama (mistral-3:8b) at `http://192.168.0.31:11434` +- **Features**: Discord commands, service integrations, AI chat capabilities +- **Database**: MySQL for conversation persistence +- **Cogs**: Modular architecture with base_cog and integration_cog + +### RedBot +- **Service**: RedBot Discord Bot +- **Role**: Alternative Discord bot with custom cogs +- **Container**: `redbot` +- **Network**: `web_net` + +### Other Tools +- **ByteStash** - File sharing service +- **KitchenOwl** - Recipe management +- **PinchFlat** - Cycling tracking + +## Network Architecture + +### Docker Networks +- **media_net** - Media services and *arr stack +- **db_net** - Database services (internal only) +- **web_net** - Web-accessible services +- **internal_net** - Internal service communication + +### Storage Structure +``` +/docker/ +├── Arrs/ (Media stack: Prowlarr, Radarr, Sonarr, Lidarr, Bazarr, Jellyfin, Jellyseerr) +├── immich/ (Photo management) +├── paperless/ (Document management) +├── litellm/ (AI gateway) +├── discord-agent/ (Discord bot with Ollama LLM integration) +├── gitea/ (Git service) +├── gramps-jamie/ (Genealogy) +├── gramps-helen/ (Genealogy) +├── npm/ (Nginx Proxy Manager) +├── qBittorrent/ (Download client) +├── slskd/ (Soulseek client) +├── speedtest-tracker/ (Network monitoring) +├── maloja/ (Music scrobbling) +├── scrobble/ (Multi-scrobbler) +├── syncthing/ (File sync) +├── Homepage/ (Dashboard) +└── Various other service configs +``` + +## Key Features & Configuration + +### Security +- Container security with `no-new-privileges:true` +- Network segmentation (databases internal-only) +- Automated updates via Watchtower +- Reverse proxy with SSL termination + +### Performance +- Intel GPU hardware acceleration for Jellyfin +- Log rotation (10MB max, 3 files) +- Resource limits (Portainer: 512MB RAM) + +### Data Management +- Comprehensive backup script included +- Separate volumes for critical data +- NAS storage integration for media files + +### Environment +- System configured for Pacific/Auckland timezone +- PUID/PGID for proper file permissions +- Extensive environment variable configuration + +## Service Dependencies + +### Database Services +- MariaDB instances for media stack, tracking, and document services +- PostgreSQL for AI services, photo management, and finance +- Redis for caching and message brokering + +### Network Dependencies +- All web services route through Nginx Proxy Manager +- Internal services communicate via internal_net +- Media services isolated on media_net +- Database services on internal-only db_net + +## Maintenance Agents + +### Automated Tasks +- **Watchtower** - Daily container updates +- **Backup Script** - Regular data backups +- **Speedtest Tracker** - Automated network testing + +### Manual Tasks +- Service monitoring via Glances +- Container management via Portainer +- Service configuration via Nginx Proxy Manager + +## Development & Debugging + +### Service Access +- **Web Services**: Access through reverse proxy (80/443) +- **Admin Interfaces**: Direct ports for management (e.g., 9443, 7575) +- **Internal Services**: Host-only access for security + +### Troubleshooting +- Container logs accessible via Portainer +- System metrics via Glances +- Network diagnostics via iperf3-server + +This Docker infrastructure represents a comprehensive self-hosted ecosystem covering media management, document processing, AI services, development tools, and personal productivity applications, all integrated through a reverse proxy and organized with proper network segmentation. \ No newline at end of file diff --git a/README.md b/README.md index 07b6f22..519ac2a 100644 --- a/README.md +++ b/README.md @@ -35,19 +35,58 @@ This infrastructure manages a comprehensive self-hosted environment including me | Management & Infrastructure | Portainer | Container management UI | 9443 | | Management & Infrastructure | Homepage | Service dashboard | 7575 | -### External Non-Docker Services +### Discord Agent Integration -| Service Categories| Service | Role / Description | Web UI Port | -|---|---|---|---| -| Management & Infrastructure | Adguard | DNS/ad-blocking | 3000 | -| Data Management | Vaultwarden | Password manager | 80/8080 | -| Entertainment | Crafty (Crafty Controller) | Minecraft server management | 3000 | -| Data Management | Nextcloud | File sync and collaboration | 443 | -| Automation | Homeassistant | Home automation | 8123 | -| Entertainment | Foundryvtt | Foundry VTT | 30000 | -| Data Management | Openmediavault | NAS management | 80/443 | -| Website Hosting | Wordpress-irodori | WordPress site | 80 | -| Website Hosting | Wordpress-dustin | WordPress site | 80 | +### Discord Bot Service +- **Service**: Discord Agent Bot +- **Role**: AI-powered Discord bot using Ollama LLM +- **Container**: `discord-agent` +- **Network**: `internal_net`, `db_net` +- **Web UI**: Discord (no direct web interface) +- **Configuration**: `/docker/discord-agent/config/agent-config.yaml` +- **Data**: `/docker/discord-agent/data/` + +### Features +- **AI Integration**: Uses Ollama LLM (mistral-3:8b) at `http://192.168.0.31:11434` +- **Modular Architecture**: Cogs for service integrations +- **Database Support**: MySQL for conversation persistence +- **Caching**: Redis for improved performance (configurable) +- **Security**: Discord role-based permissions +- **Logging**: Configurable logging levels (default: INFO) +- **Rate Limiting**: Built-in rate limiting for API calls + +### Available Commands +- `!agent` - Chat with the AI assistant +- `!status` - Check bot status and latency +- `!help` - Show available commands +- Service integration commands (configurable) + +### Configuration +- **Main Config**: `config/agent-config.yaml` (YAML-based with env var substitution) +- **Discord Token**: Sourced from `DISCORD_BOT_TOKEN` environment variable +- **Ollama Config**: Model, endpoint, temperature, top_p, top_k parameters +- **Integrations**: Configurable service integrations in YAML + +### Environment Variables +``` +DISCORD_BOT_TOKEN=your_discord_bot_token_here +OLLAMA_ENDPOINT=http://192.168.0.31:11434 +OLLAMA_MODEL=mistral-3:8b +TZ=Pacific/Auckland +``` + +### Bot Architecture +- **Main Bot**: `discord_agent.py` - Core Discord bot implementation +- **Base Cog**: `cogs/base_cog.py` - Base functionality and commands +- **Integration Cog**: `cogs/integration_cog.py` - Service integrations +- **Config**: YAML-based configuration with environment variable substitution +- **Database**: MySQL with asyncpg driver for conversation history + +### Security +- Runs on internal networks only +- Discord role-based access control +- Configurable rate limiting to prevent abuse +- No direct database access from external networks ### Network Segmentation @@ -65,12 +104,10 @@ This infrastructure manages a comprehensive self-hosted environment including me ├── Homepage/ ├── Arrs/ (Prowlarr, Radarr, Sonarr, Lidarr, Bazarr, Jellyfin, Jellyseerr) ├── qBittorrent/ -├── slskd/ -├── soulsync/ +├── discord-agent/ (Discord bot with Ollama LLM) ├── paperless/ ├── stirling/ ├── immich/ -├── obsidian/ ├── syncthing/ ├── gitea/ ├── gramps-jamie/ diff --git a/RESTORE.md b/RESTORE.md index 71338ce..659ba8d 100644 --- a/RESTORE.md +++ b/RESTORE.md @@ -17,19 +17,102 @@ If you are restoring, follow the standard restoration steps below, and refer to - Access to NAS storage (if applicable) - Root or sudo access -## Step 1: Extract Backup +## Discord Agent Service +### Step 1: Create Discord Agent Directory ```bash -# Navigate to your backup location -cd /path/to/backups - -# Extract the backup archive -tar xzf docker-backup-YYYYMMDD_HHMMSS.tar.gz - -# Navigate into the extracted backup -cd YYYYMMDD_HHMMSS +mkdir -p /docker/discord-agent/config /docker/discord-agent/data/logs /docker/discord-agent/data/database /docker/discord-agent/data/cache /docker/discord-agent/cogs /docker/discord-agent/scripts ``` +### Step 2: Copy Configuration Files +```bash +# Copy agent configuration +cp agent-config.yaml /docker/discord-agent/config/ +cp permissions.json /docker/discord-agent/config/ + +# Copy scripts +cp startup.sh /docker/discord-agent/scripts/ +cp health_check.sh /docker/discord-agent/scripts/ +``` + +### Step 3: Copy Python Files +```bash +cp discord_agent.py /docker/discord-agent/ +cp base_cog.py /docker/discord-agent/cogs/ +cp integration_cog.py /docker/discord-agent/cogs/ +``` + +### Step 4: Copy Requirements +```bash +cp requirements.txt /docker/discord-agent/ +cp Dockerfile /docker/discord-agent/ +``` + +### Step 5: Update Environment Variables +Add these to your `.env` file: +```bash +# Discord Agent +DISCORD_BOT_TOKEN=your_discord_bot_token_here +OLLAMA_ENDPOINT=http://192.168.0.31:11434 +OLLAMA_MODEL=mistral-3:8b +TZ=Pacific/Auckland +``` + +### Step 6: Update Configuration +Edit `/docker/discord-agent/config/agent-config.yaml` with your specific settings: +```yaml +discord: + token: ${DISCORD_BOT_TOKEN} + prefix: "!" + status: "AI Assistant | !help" + +ollama: + endpoint: "${OLLAMA_ENDPOINT:http://192.168.0.31:11434}" + model: "${OLLAMA_MODEL:mistral-3:8b}" + parameters: + temperature: 0.7 + top_p: 0.9 + top_k: 40 + timeout: 60 +``` + +### Step 7: Build and Start Service +```bash +# Build the Discord agent image +docker compose build discord-agent + +# Start the service +docker compose up -d discord-agent + +# Verify the service is running +docker compose logs discord-agent +``` + +### Step 8: Verify Integration +```bash +# Check Discord bot connection +docker exec discord-agent python3 -c "import discord; print('Discord library available')" + +# Check Ollama connection +curl http://192.168.0.31:11434/api/tags + +# Check MySQL database +docker compose exec agent-db mysql -u agent -pagent -e "SHOW DATABASES;" +``` + +### Step 9: Test Bot Functionality +- The bot should appear in your Discord server +- Test with `!help` command to verify functionality +- Test with `!agent ` to verify Ollama integration +- Test with `!status` to check bot status + +### Troubleshooting +- If bot doesn't start, check logs: `docker compose logs discord-agent` +- Verify DISCORD_BOT_TOKEN is set correctly in `.env` file +- Ensure Ollama is running and accessible at the configured endpoint +- Verify internal_net and db_net networks are available +- Check that MySQL database agent-db is running and healthy + ## Step 2: Restore Directory Structure ```bash diff --git a/docker-compose.yaml b/docker-compose.yaml index 9d6b269..b97b08d 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -189,7 +189,7 @@ services: - PUID=${PUID} - PGID=${PGID} volumes: - - /mnt/Nas-Storage/data:/data + - /mnt/nas-storage/data:/data - /docker/qBittorrent/config:/config logging: *default-logging @@ -206,7 +206,7 @@ services: - PGID=${PGID} - TZ=${TZ} volumes: - - /mnt/Nas-Storage/data/:/data + - /mnt/nas-storage/data/:/data - /docker/Arrs/Radarr/config:/config logging: *default-logging @@ -223,7 +223,7 @@ services: - PGID=${PGID} - TZ=${TZ} volumes: - - /mnt/Nas-Storage/data:/data + - /mnt/nas-storage/data:/data - /docker/Arrs/Sonarr/config:/config logging: *default-logging @@ -240,7 +240,7 @@ services: - PUID=${PUID} - PGID=${PGID} volumes: - - /mnt/Nas-Storage/data:/data + - /mnt/nas-storage/data:/data - /docker/Arrs/Lidarr/config:/config logging: *default-logging @@ -257,7 +257,7 @@ services: - PGID=${PGID} - TZ=${TZ} volumes: - - /mnt/Nas-Storage/data:/data + - /mnt/nas-storage/data:/data - /docker/Arrs/Bazarr/config:/config logging: *default-logging @@ -295,13 +295,13 @@ services: devices: - /dev/dri/renderD128:/dev/dri/renderD128 volumes: - - /mnt/Nas-Storage/data:/data + - /mnt/nas-storage/data:/data - /docker/Arrs/Jellyfin/cache:/cache - /docker/Arrs/Jellyfin/config:/config logging: *default-logging slskd: - image: slskd/slskd + image: slskd/slskd:latest container_name: slskd restart: unless-stopped networks: @@ -317,33 +317,11 @@ services: - SLSKD_USERNAME=${SLSKD_USERNAME} - SLSKD_PASSWORD=${SLSKD_PASSWORD} volumes: - - /mnt/Nas-Storage/data/torrents/soulsync/complete:/downloads - - /mnt/Nas-Storage/data/torrents/soulsync/incomplete:/incomplete + - /mnt/nas-storage/data/torrents/soulsync/complete:/data/torrents/soulsync/complete + - /mnt/nas-storage/data/torrents/soulsync/incomplete:/data/torrents/soulsync/incomplete - /docker/slskd:/app logging: *default-logging - soulsync-webui: - image: boulderbadgedad/soulsync:latest - container_name: soulsync-webui - restart: unless-stopped - networks: - media_net: - ports: - - "8887:8008" - extra_hosts: - - "host.docker.internal:host-gateway" - environment: - - TZ=${TZ} - - PUID=${PUID} - - PGID=${PGID} - - FLASK_ENV=production - volumes: - - /mnt/Nas-Storage/data/media/music:/music:ro - - /mnt/Nas-Storage/data/torrents/soulsync:/app/downloads - - /docker/soulsync/logs:/app/logs - - docker_soulsync:/app/database - logging: *default-logging - # --- DOCUMENT & AI SUITE --- onlyoffice: image: onlyoffice/documentserver:latest @@ -473,7 +451,8 @@ services: - "8090:8080" environment: - UI_APPNAME=Stirling-PDF - - SHOW_SURVEY=true + - SHOW_SURVEY=false + - SECURITY_ENABLELOGIN=false - SYSTEM_MAXFILESIZE=100 - PUID=${PUID} - PGID=${PGID} @@ -689,6 +668,7 @@ services: ports: - "5512:5000" environment: + - GRAMPSWEB_TREE=Helen Tree - GRAMPSWEB_CELERY_CONFIG__broker_url=redis://grampsweb_redis:6379/0 volumes: - /docker/gramps-helen/cache:/app/cache @@ -773,6 +753,24 @@ services: volumes: - /docker/redbot:/data logging: *default-logging + + discord-agent: + build: ./discord-agent + container_name: discord-agent + restart: unless-stopped + networks: + - internal_net + volumes: + - ./discord-agent/data:/app/data + env_file: + - .env + environment: + - DISCORD_BOT_TOKEN=${DISCORD_BOT_TOKEN} + - OLLAMA_ENDPOINT=${OLLAMA_ENDPOINT:-http://ollama:11434} + - OLLAMA_MODEL=${OLLAMA_MODEL:-llama3.2} + - OPENCODE_ENDPOINT=${OPENCODE_ENDPOINT:-http://192.168.0.10:4096} + - OPENCODE_API_KEY=${OPENCODE_API_KEY} + logging: *default-logging iperf3-server: image: networkstatic/iperf3 @@ -828,6 +826,8 @@ services: image: krateng/maloja:latest container_name: maloja restart: unless-stopped + environment: + - MALOJA_FORCE_PASSWORD=${MALOJA_FORCE_PASSWORD} networks: - internal_net - web_net @@ -847,7 +847,14 @@ services: ports: - "9078:9078" environment: - - MALOJA_URL=http://maloja:42010 + - MALOJA_URL=${MALOJA_URL} + - MALOJA_API_KEY=${MALOJA_API_KEY} + - JELLYFIN_URL=${JELLYFIN_URL} + - JELLYFIN_APIKEY=${JELLYFIN_APIKEY_MS} + - JELLYFIN_USER=${JELLYFIN_USER} + - JELLYFIN_TRANSFORMS=musicbrainz + - MB_CONTACT=${MB_CONTACT} + - MB_PRESETS=default,sensible,native volumes: - /docker/scrobble/config:/config logging: *default-logging @@ -865,7 +872,7 @@ networks: internal_net: name: internal_net driver: bridge - + volumes: docker_aidata: docker_dbdata: @@ -875,3 +882,4 @@ volumes: docker_soulsync: open-webui: portainer_data: + discord_agent_data: