# ErezAgent -- Deployment Guide **Version: v4.94.5** This guide covers deploying ErezAgent in production environments: single-server Docker, reverse proxy with SSL, cloud VM, and multi-tenant setup. > **Version note**: This guide's deployment mechanics (Docker Compose, reverse proxy, upgrade flow) apply unchanged across releases and are not tied to any single version. For the full version-by-version feature history, see `CHANGELOG.md`. The Web Installer Wizard is the recommended installation method for all platforms -- download it from the [ErezAgent-Deploy Releases page](https://github.com/erezrozenbaum/ErezAgent-Deploy/releases). --- ## Quickest Path: Web Installer Wizard For a new deployment on any platform, the easiest method is the **Web Installer Wizard** -- a real, CI-built binary (Windows/macOS/Linux), not a script you have to configure by hand. 1. Download the installer for your OS from the [ErezAgent-Deploy Releases page](https://github.com/erezrozenbaum/ErezAgent-Deploy/releases/latest): - `ErezAgent-Setup-.exe` -- Windows 10/11, Server 2019/2022/2025 - `ErezAgent-Setup--macos` -- macOS 13+ (Apple Silicon or Intel) - `ErezAgent-Setup--linux` -- Ubuntu / Debian / RHEL 2. Run it. A browser wizard opens automatically at `http://localhost:7373` (it also opens if you navigate there yourself). 3. Follow the 9 guided steps: **Docker** check, **Obsidian** setup, **License** (a starting plan preference; the 30-day trial starts at first login), **AI & Owner** (identity, domain, AI provider and API keys, voice, backup destination), **Secrets** (generated in-browser, never sent to any server), **.env** review, **Launch** (pulls images, starts the stack, repairs data-volume ownership where needed), **Health** check, and **Foundation** (Company Foundation setup). 4. On first login, click **Start Trial** to activate your 30-day evaluation period (set `TRIAL_TYPE=trial_14` in `.env` for a 14-day trial). If you already have a purchased license, you can paste it in Settings -> Platform License instead. While the trial runs, a banner and reminders by email and Telegram start 14 days before it ends (at 14, 7, 3 and 1 days; a 14-day trial shows the banner throughout and is not sent the 14-day reminder). Email needs a provider: choose SendGrid (API key and a verified sender address) in the installer's optional email step, or set `EMAIL_PROVIDER`, `SENDGRID_API_KEY` and `EMAIL_FROM_ADDRESS` in `.env` later; without one, reminders appear in the app and on Telegram only. When it ends the app is locked until a licence is entered; data is kept untouched. To renew, send the machine fingerprint (shown in Settings -> Platform License) to `VENDOR_CONTACT_EMAIL`, or visit `VENDOR_RENEWAL_URL`; both have defaults and rarely need changing. Once complete, ErezAgent is running at `http://localhost:3000` (if you left the domain blank) or `https://your-domain` (with automatic HTTPS via Caddy, if you set one in step 3). The wizard automates every step below. Continue reading only if you prefer a manual/scripted installation, need a headless server flow, or want to understand what it's doing under the hood. --- ## Before You Start -- Pre-Deployment Checklist Complete every item in this section **before** running any installer. Missing a step here causes failures mid-installation that are harder to debug than getting it right up front. ### Step 1 -- Choose your deployment type | Type | Where Docker runs | Obsidian integration | Recommended for | |---|---|---|---| | **Local** | Your own Mac or Windows machine (Docker Desktop) | Yes | Personal use, daily operator | | **Cloud server** | Remote VPS (Hetzner, DigitalOcean, EC2, Azure) | No -- Obsidian requires a GUI, not available on headless servers | Production, selling to customers | > **Important:** Obsidian integration (`OBSIDIAN_API_URL`, `OBSIDIAN_API_KEY`) only works in local deployments where Docker and Obsidian run on the same machine. For cloud deployments, skip all Obsidian steps. ### Step 2 -- Gather your API keys Have these ready before running the installer. You cannot complete setup without the required ones. | Key | Required | Where to get it | |---|---|---| | `ANTHROPIC_API_KEY` | **Required** | console.anthropic.com -> API Keys | | `ADMIN_PASSWORD` | **Required** | Choose a strong password now (or let the installer generate one) | | `OPENAI_API_KEY` | Recommended | platform.openai.com -> API Keys (used for memory embeddings) | | `TELEGRAM_BOT_TOKEN` | Optional | @BotFather on Telegram -> /newbot | | `TELEGRAM_OWNER_CHAT_ID` | Optional (with Telegram) | Message @userinfobot to get your chat ID | | `PERPLEXITY_API_KEY` | Optional | perplexity.ai -> API (used for morning briefing and research features) | | `OPENWEATHERMAP_API_KEY` | Optional | openweathermap.org -> free tier API key | | `GITHUB_TOKEN` | Optional | GitHub -> Settings -> Developer settings -> Personal access tokens | | `ANTHROPIC_ADMIN_API_KEY` | Optional (v4.77.0) | console.anthropic.com -> Settings -> Admin API Keys -- separate from `ANTHROPIC_API_KEY` above; enables real 30-day spend visibility on the AI Resources page | | `OPENAI_ADMIN_API_KEY` | Optional (v4.77.0) | platform.openai.com -> Settings -> Admin keys -- separate from `OPENAI_API_KEY` above; enables real 30-day spend visibility | | `XAI_MANAGEMENT_API_KEY` / `XAI_TEAM_ID` | Optional (v4.77.0) | xAI Console -> Settings -> Management Keys / Team -- enables a real remaining prepaid balance, not just spend | ### Step 3 -- Prepare Obsidian (local deployments only) Skip this step entirely if you are deploying on a cloud server. **3a. Install Obsidian** on the same machine that will run Docker Desktop. Download from obsidian.md. **3b. Open your vault** (or create a dedicated vault for ErezAgent). Recommendation: create a separate vault named `ErezAgent` to keep agent-generated notes separate from your personal notes. **3c. Install the Local REST API plugin:** 1. Obsidian -> Settings -> Community plugins -> turn off Safe mode 2. Click Browse -> search "Local REST API" -> Install -> Enable 3. Go to Settings -> Local REST API -> the API key is shown and auto-generated -> **copy it** 4. Note the port (default: 27123) **3d. Keep Obsidian running** whenever ErezAgent is running. The Local REST API is active only while Obsidian is open. If you close Obsidian, agents cannot read or write notes until you reopen it. **3e. Note your vault path.** Both `install.ps1` and the Web Installer Wizard can detect vaults you already have open in Obsidian, or let you type a path directly, and will copy its contents into ErezAgent's storage once at first launch -- so if you already have notes you want ErezAgent to start with, having this path ready saves a manual import later. This is a one-time copy, not an ongoing sync -- ErezAgent's vault and your original Obsidian vault are independent after that point. > **Linux Docker note:** On Linux, `host.docker.internal` does not resolve automatically. Add this to the `api` service in your `docker-compose.yml`: > ```yaml > extra_hosts: > - "host.docker.internal:host-gateway" > ``` ### Step 4 -- Provision your server (cloud deployments only) For local Docker Desktop deployments, skip to Step 5. **Server spec:** Sized around the real per-container CPU/memory limits `docker-compose.yml` now ships with (`API_CPU_LIMIT`/`API_MEM_LIMIT_MB` and the matching `WORKER_*`/`POSTGRES_*`/`REDIS_*` pairs in `env.example`), not a guess -- those are hard ceilings each container can reach individually, and this table leaves comfortable headroom for the rest of the stack (connector/security services, sandbox-executor, frontend, and the observability containers) running alongside them. | Component | Minimum | Recommended | Large | |---|---|---|---| | CPU | 4 cores | 8 cores | 16 cores | | RAM | 8 GB | 16 GB | 32 GB | | Disk | 40 GB SSD | 100 GB SSD | 250 GB SSD | | OS | Ubuntu 22.04 / Debian 12 | Ubuntu 24.04 LTS | Ubuntu 24.04 LTS | | Docker | 24.x | 26.x | 26.x | | Docker Compose | v2.17 or newer | v2.17 or newer | v2.17 or newer | The install and upgrade scripts (`install.sh`, `install.ps1`, `upgrade.sh`, `upgrade.ps1`), the web installer and `setup.py` check the Docker Compose version (`docker compose version --short`) before they change anything and stop with a short message when it is older than 2.17.0. Update Docker Desktop, or on Linux the `docker-compose-plugin` package. A version that cannot be read only produces a warning. - **Minimum** -- personal use, evaluating ErezAgent, or a light workload with few missions running at once. The trial/mock-mode paths (Option 5 below) are comfortable here. - **Recommended** -- a real single-operator company with regular mission activity and several connectors active. This is what the Web Installer Wizard's own defaults assume. - **Large** -- multiple active ventures/customers, frequent concurrent missions, or the add-ons below. **If you also want...** (add to the Recommended tier, not the Minimum one): | Add-on | Add this much | |---|---| | Local voice (Voicebox -- Settings > Voice Profiles, `TTS_PROVIDER=voicebox`/`STT_PROVIDER=voicebox`) | +4 GB RAM, +2 CPU cores. Voicebox is a separate app you run yourself (desktop app or its own deployment, see `32_VOICE_AND_CHAT.md`) -- ErezAgent only connects to it over HTTP, so this is capacity on top of the table above, not inside it. A GPU speeds it up substantially but isn't required. | | Android Customer Delivery (building native Android apps for customers) | +11 GB disk for the build image (downloaded when first needed); see the note right below this table for the full detail. | | Running several missions with many concurrent agent steps | +2-4 CPU cores, +4-8 GB RAM per tier above what's shown -- the container limits above are hard ceilings per service, so more true concurrency benefits from more host capacity to actually use them. | **Open ports:** | Port | Purpose | |---|---| | 22 | SSH (your IP only) | | 80 | HTTP (SSL certificate verification) | | 443 | HTTPS (production) | Only ports 80 and 443 (Caddy) are public. The API (8000) and the frontend (3000) are bound to `127.0.0.1`; use an SSH tunnel if you need plain-HTTP access before SSL is set up. **Recommended providers, sized to the Recommended tier above (8 cores / 16 GB):** Hetzner CCX33 or CPX51, a DigitalOcean 16 GB Droplet, or AWS/Azure's 8-vCPU/16+ GB general-purpose tier (e.g. `m5.2xlarge`/`Standard_D8s_v5`). For the Minimum tier (4 cores / 8 GB), step down one size on any of the same providers (e.g. Hetzner CX32, a DigitalOcean 8 GB Droplet, `t3.xlarge`) -- comfortable for evaluating ErezAgent or light personal use, not for the Large tier's concurrent-mission load. > **Android Customer Delivery:** add ~11 GB disk if you plan to build native Android apps for customers. > The Android build image (`erezagent-sandbox-android`, about 4 GB to download and 11 GB on disk) is no longer > downloaded on every install or upgrade: it is downloaded on request (`--with-android` / `-WithAndroid`, or the > install prompt), when it is already in use, or automatically the first time an Android app is built, and it is > not downloaded again while its toolchain is unchanged. If the host cannot reach the registry, it is built > locally instead the first time it is needed (10-20 minutes). If the disk has room for everything except this image, > the installer continues without it and says so. > Since v4.91.1 the image targets Expo SDK 57 (Android API 36); a local image from an earlier release is > detected as out of date and rebuilt the same way. > Real hardware-accelerated emulator verification needs `/dev/kvm` on the host, which most bare-metal > Linux servers have and most non-bare-metal cloud VMs (including the providers above, on their standard > tiers) do not expose. Without it, Android builds still work -- emulator verification runs a slower, > real software-rendering fallback instead. Both installers print an honest status line either way; > check it yourself anytime with: > `docker run --rm --device /dev/kvm alpine:3.20 true && echo available || echo unavailable`. ### Step 5 -- Point your domain (if using SSL) Before starting the installer: 1. Buy or use an existing domain. 2. Create an A record pointing `yourdomain.com` (or `admin.yourdomain.com`) -> your server IP. 3. Wait for DNS to propagate (check with `dig yourdomain.com`). ErezAgent uses Caddy for automatic Let's Encrypt SSL. Caddy fails to get a certificate if the domain is not already pointing at the server when Docker starts. --- ## Deployment -- Step by Step ### 1. Install Docker **Linux (cloud server):** ```bash curl -fsSL https://get.docker.com | sh sudo usermod -aG docker $USER newgrp docker ``` Verify: ```bash docker --version docker compose version ``` **Windows / Mac:** Install Docker Desktop from docker.com. Enable Docker Compose v2 in settings. ### 2. Get the installer ```bash git clone https://github.com/erezrozenbaum/ErezAgent-Deploy.git cd ErezAgent-Deploy ``` ### 3. Run the interactive installer ```bash # Linux / Mac chmod +x install.sh ./install.sh # Windows (PowerShell as Administrator) Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass .\install.ps1 ``` The installer: - Checks Docker, Docker Compose, and port availability - Asks for your API keys (have Step 2 keys ready) - Generates secure random secrets (DB/Redis passwords, JWT signing key, API key, encryption key, sandbox and doc-renderer internal tokens) - Writes a complete `.env` file and hardens its file permissions (on a re-run that keeps your existing `.env`, any newly introduced settings are appended with safe defaults and existing values are never changed, except the few the API refuses to start with: `LICENSE_MODE=dev`, `DISABLE_AUTH=true`, signing secrets under 32 characters or placeholders) - Pulls Docker images from GHCR (including the Android sandbox image -- see the Step 4 note above) - Starts all containers - Runs database migrations automatically - Validates health of all services; if the API does not start, it prints the API's own reason ### 4. Add Obsidian to `.env` (local deployments only) Both `install.ps1` and the Web Installer Wizard prompt for your Obsidian API key and vault path during install, so this step is normally already done. If you skipped it, or used `install.sh` (which does not prompt for Obsidian), open `.env` and add: ```env OBSIDIAN_API_URL=https://host.docker.internal:27124 OBSIDIAN_API_KEY= OBSIDIAN_VAULT_PATH=/obsidian-vault ``` `OBSIDIAN_VAULT_PATH` is the path *inside the container* -- leave it as `/obsidian-vault`, do not put your host machine's vault folder path there. The vault itself lives in a Docker-managed volume (`vaultdata`), not a folder on your host; to load your existing notes into it, use the installer's vault-import step, or restart and re-run the installer against the same `.env` if you already have Obsidian open. Then restart the API: ```bash docker compose restart api ``` Verify the connection: - Open ErezAgent -> Platform -> Connectors -> Obsidian -> CONNECT -> paste the same API key -> Connect - Test: Executive Office -> Command Center -> type "Read my recent Obsidian notes" ### 5. Configure SSL with Caddy (cloud deployments with a domain) Make sure your domain already points to the server (Step 5 above). Then add to `.env`: ```env DOMAIN=yourdomain.com ACME_EMAIL=you@yourdomain.com ``` Start Caddy: ```bash docker compose up -d caddy ``` Caddy requests a Let's Encrypt certificate automatically. Your app is available at `https://yourdomain.com` within seconds. See "Production Setup with SSL" below for the full Caddy configuration, including self-service venture custom domains. ### 6. Verify the deployment ```bash # All containers healthy? docker compose ps # API responding? curl http://localhost:8000/api/health ``` On a cold start the `api` container can take about two minutes to report healthy while it prepares the database (its health-check grace period is 180 seconds), and the web wizard, `setup.py` and the rollback script wait for it. Run `docker compose ps` again before assuming something is wrong. Open `http://localhost:3000` (local) or `https://yourdomain.com` (cloud) and log in with `admin` / your `ADMIN_PASSWORD`. ### 7. Run first-time setup in the UI 1. Log in -> go to **Platform -> Connectors**. 2. Connect Telegram (for mobile approval notifications) -- paste `TELEGRAM_BOT_TOKEN` and `TELEGRAM_OWNER_CHAT_ID`. 3. Connect Obsidian (local only) -- if you added it to `.env` in Step 4. 4. Set up your **Company Foundation** -- see the dedicated section below. Every AI Executive Board deliberation reads it first. 5. Go to **Executive Office -> Command Center** and type your first goal. --- ## Production Setup with SSL ### Option A: Caddy (Automatic Let's Encrypt -- Recommended) ErezAgent includes a Caddy reverse proxy service that handles TLS automatically. **1. Set env vars in `.env`:** ```env DOMAIN=your-domain.com ACME_EMAIL=you@example.com ``` **2. Start Caddy:** ```bash docker compose up -d caddy ``` Caddy requests a Let's Encrypt certificate via TLS-ALPN-01 (port 443 only -- no port 80 required). Your app is available at `https://your-domain.com` within seconds. > **Mobile app:** set `EXPO_PUBLIC_API_URL=https://your-domain.com` in `mobile/.env`. #### Self-service venture custom domains The `Caddyfile` also configures Caddy's built-in **on-demand TLS** for any Host other than your own `DOMAIN` above: a global `on_demand_tls { ask http://api:8000/api/domains/tls-ask }` block plus one catch-all `:443 { tls { on_demand } }` site block. This is what makes per-venture self-service custom domains work (Ventures screen -> a venture's Support workspace -> "Self-Service Domains" -- see `ADMIN_GUIDE.md`): the first real request for a venture's claimed hostname makes Caddy call the `tls-ask` endpoint, which only approves a certificate request when that hostname (a) has a real claim in the `venture_domains` table, (b) genuinely resolves (via DNS) to this server's own `PUBLIC_SERVER_IP`, and (c) since v4.91.0 has a proven owner: the claim's DNS TXT record `_erezagent-challenge.` = `erezagent-verify=` was seen (the venture's Self-Service Domains card shows the exact record). The `tls-ask` endpoint decides from stored state and the pointing check only -- it never looks up the TXT record, so an unproven claim is refused at once and Caddy never waits on DNS; the record is checked by the owner's Verify now and by the worker's check about every 30 minutes, after which the first visit to the domain gets its certificate. Several organisations can hold an unproven claim on the same name; the first to prove it owns it. A claim that was already active before v4.91.0 is accepted without the record and can be proven from the same card. An unclaimed, non-resolving or unproven hostname is always refused -- this endpoint is the real security boundary between an anonymous Host header and a real Let's Encrypt request, so if you ever re-point `on_demand_tls`'s `ask` URL (e.g. a non-default internal network layout), make sure it still reaches the real `api` container and nothing else. Your own single `DOMAIN` site block is matched first, exactly as before this feature existed. This requires `PUBLIC_SERVER_IP` in `.env` (see `21_ENVIRONMENT_AND_CONFIG.md`) -- without it, every domain claim is refused up front rather than silently accepted with nothing to verify against. ### Option B: Nginx Reverse Proxy ```bash sudo apt install nginx certbot python3-certbot-nginx -y ``` Create `/etc/nginx/sites-available/erezagent`: ```nginx server { listen 80; server_name your-domain.com; return 301 https://$server_name$request_uri; } server { listen 443 ssl http2; server_name your-domain.com; ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem; ssl_protocols TLSv1.2 TLSv1.3; ssl_prefer_server_ciphers on; # Frontend (React) location / { proxy_pass http://localhost:3000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } # API location /api/ { proxy_pass http://localhost:8000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } # SSE (real-time events) -- long-lived connections location /api/events { proxy_pass http://localhost:8000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_http_version 1.1; proxy_set_header Connection ''; proxy_buffering off; proxy_cache off; proxy_read_timeout 86400; } } ``` Enable and get a certificate: ```bash sudo ln -s /etc/nginx/sites-available/erezagent /etc/nginx/sites-enabled/ sudo nginx -t sudo certbot --nginx -d your-domain.com sudo systemctl reload nginx ``` --- ## Cloud VM Deployment ### AWS EC2 **Instance type:** `t3.xlarge` (4 vCPU, 16 GB RAM) for the Minimum tier; `m5.2xlarge` (8 vCPU, 32 GB RAM) for Recommended. See the server-spec table above for what each tier is actually sized for. **Security Groups:** - Inbound 22 (SSH) from your IP only - Inbound 80 (HTTP) from anywhere - Inbound 443 (HTTPS) from anywhere - All outbound allowed ```bash ssh -i your-key.pem ubuntu@YOUR_EC2_IP # Install Docker curl -fsSL https://get.docker.com | sh sudo usermod -aG docker ubuntu # A swap file is still good practice as a safety margin against a burst # above the sized-for tier, even though it's no longer load-bearing at # these instance sizes. sudo fallocate -l 4G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab # Then follow "Deployment -- Step by Step" above ``` ### Azure VM **Size:** `Standard_D4s_v5` (4 vCPU, 16 GB RAM) for the Minimum tier; `Standard_D8s_v5` (8 vCPU, 32 GB RAM) for Recommended. **Network Security Group rules:** allow 22 (SSH), 80 (HTTP), 443 (HTTPS) inbound. Steps are identical to EC2 -- Docker install, then the standard deployment steps above. ### DigitalOcean / Hetzner An 8 GB Droplet or Hetzner CX32 covers the Minimum tier; a 16 GB Droplet or Hetzner CCX33 covers Recommended. Install Docker, then follow "Deployment -- Step by Step" above. --- ## Post-Deploy Setup Once ErezAgent is running and healthy, complete these two setup tasks before putting it in front of real use. ### Company Foundation Setup Company Foundation is the identity and direction layer of ErezAgent. It consists of two records that every AI Executive Board deliberation reads **before** reasoning: - **Company Constitution** -- your company's invariant identity: who you are, what you stand for, what the board must never violate. - **Active Mission** -- your highest-priority goal right now, with a measurable target, deadline, and key results. These records are **per-tenant** (per installation). They ship empty. You set them up after deployment. #### Why this matters Every time you run a Goal Command or the CEO convenes the board, the system prepends the full Company Foundation block to the deliberation prompt. The board sees your company's identity statement first, then the constitution, then the mission -- before any analysis begins. Every recommendation is grounded in your company's stated values, non-negotiables, and current strategic direction. Without a constitution, the board operates on generic principles. Without an active mission, it cannot calibrate recommendations toward your current goal. #### Option A: Set during the Web Installer Wizard (Step 9) The installer wizard includes a dedicated **Company Foundation** step (Step 9) after health check. 1. Complete Steps 1-8 (Docker, Obsidian, License, AI & Owner, Secrets, .env, Launch, Health). 2. The wizard opens the Company Foundation step. 3. Fill in two tabs: **Constitution tab** -- required fields: Company Name; Company Identity (one sentence -- e.g. "ErezAgent is an AI Company Operating System that enables organizations to think, decide, execute, learn, and grow through coordinated artificial intelligence."); Tagline; Long-Term Vision; Value Proposition; Target Market; Business Model; Competitive Moat; Core Values (3-7); Non-Negotiables (5-10 ethical boundaries the board must never violate); Additional Context (optional raw text that appears verbatim in every deliberation). **Mission tab** -- required fields: Mission Headline (e.g. "Reach $60k ARR by end of Q3 2026"); Context; Primary Metric; Target Value; Deadline; Key Results (3-5); Constraints (e.g. "Solo founder -- no hiring before $30k ARR"). 4. Click **Seed Foundation**. The wizard `POST`s to `/api/company-foundation/seed` using your API key. This is idempotent -- running it again on an already-seeded system is a no-op. 5. To skip setup during install, click **Skip** -- you can set up Company Foundation from the app at any time. #### Option B: Set from the ErezAgent UI (after install) Navigate to **Company Foundation** in the left sidebar (under Intelligence). Three tabs: - **Constitution tab** -- click **Add Constitution** if none exists. To update an existing one, click **Amend Constitution**, which creates a new version and supersedes the old one (preserving history). Only one constitution can be `active` at a time per tenant; use the **History** button to view previous versions. - **Active Mission tab** -- click **Set Mission** if none is active. Click **Achieved** to archive a completed mission and open a new one. Click **Pivot** to record the reason, archive the current mission, and start a new one immediately. - **Board Preview tab** -- shows exactly what the Executive Board reads at the start of every deliberation; use it to verify formatting before running Goal Commands. If no active constitution exists, amber warning banners appear throughout the Company Foundation page. #### Option C: Set via API (programmatic / CI deployment) ```bash curl -X POST https://your-domain.com/api/company-foundation/seed \ -H "Content-Type: application/json" \ -H "X-Api-Key: your-erezagent-api-key" \ -d '{ "constitution": { "company_name": "Acme Corp", "company_identity": "Acme Corp is a platform for...", "tagline": "Build faster with AI", "long_term_vision": "...", "core_values": ["Quality", "Customer First"], "non_negotiables": ["Never deceive customers"], "raw_text": "Decision Principles:\n1. Evidence over opinion..." }, "mission": { "headline": "Reach $100k ARR by Q4 2026", "primary_metric": "ARR", "target_value": "$100,000", "deadline": "2026-12-31", "key_results": ["20 paying customers", "NPS >= 40"] } }' ``` Response if seeded: `{"seeded": true}`. Response if already seeded: `{"seeded": false, "reason": "already_seeded"}`. #### Multi-business / multi-tenant deployments ErezAgent supports multiple tenants on a single installation (for MSP and enterprise use cases). Each tenant has its **own** completely independent Company Constitution and Active Mission -- RLS enforces isolation at the database level, so one tenant never sees another's foundation data. Each tenant sets up their own constitution and mission via the UI or API; the Executive Board always reads only the active records for the current tenant's session. The installer seed endpoint is intended for the primary (first) tenant -- additional tenants configure their foundation from the ErezAgent UI after their accounts are created. #### Mission lifecycle | Status | Meaning | |---|---| | `draft` | Created but not yet active | | `active` | Current highest-priority goal; injected into every board deliberation | | `achieved` | Mission was accomplished; recorded as a win | | `pivoted` | Strategy changed; reason recorded; new mission created | | `abandoned` | Discontinued without completion | When you mark a mission **Achieved**, the board stops receiving it in deliberations. Create a new mission immediately to keep the board directionally grounded. ### Department / Cost Center Setup Departments let you group agents under a real cost center with its own monthly budget, alert threshold, and chargeback reporting. This is optional -- skip it if you don't need per-department budget attribution. ```bash # Create a department (use your Bearer token from /api/auth/login) curl -X POST https://your-domain/api/departments \ -H "Authorization: Bearer YOUR_JWT" \ -H "Content-Type: application/json" \ -d '{ "name": "AI Engineering", "cost_center_code": "ENG-AI", "budget_monthly_usd": 500, "alert_threshold_pct": 80 }' # Assign an agent to the department curl -X POST https://your-domain/api/departments/{DEPT_ID}/assign-agent \ -H "Authorization: Bearer YOUR_JWT" \ -d '{"agent_id": "AGENT_UUID"}' ``` **Viewing budget utilization:** ```bash GET https://your-domain/api/departments/budget-status # Returns: all departments with spend_usd, budget_monthly_usd, budget_utilization_pct, budget_status ``` **Generating chargeback reports:** ```bash GET https://your-domain/api/value-intelligence/chargeback?period=30d # Returns: per-department gross_cost_usd, total_savings_usd, net_cost_usd, cache_hit_rate_pct # Plus summary: total_gross_cost_usd, total_savings_usd, total_net_cost_usd, department_count ``` The chargeback endpoint is also visible in the **Value Intelligence** page (scroll to the Chargeback section at the bottom). --- ## Multi-Tenant (Row-Level Security) Multi-tenant isolation is real, automatic, and requires no license tier or configuration flag -- a default tenant is seeded from the first owner user on migration, and PostgreSQL Row-Level Security enforces isolation at the database row level from that point on. There is no `MULTI_TENANT_MODE` env var and no per-tenant schema prefix -- isolation is row-level, not schema-level. ### How it works `TenantContextMiddleware` extracts `tenant_id` from the JWT on every request and sets `app.current_tenant_id` as a session-scoped PostgreSQL config value (`deps.py::get_tenant_session`). RLS policies now cover 300+ tables across the schema (this section originally listed a fixed 12-table set from the platform's early build -- long since superseded release over release as new features shipped, each with its own tenant-scoped tables and RLS policy from creation) -- every one filters queries against that value automatically, so a request for one tenant's data structurally cannot see another's. Workers and migrations run with a null setting and get full access via each policy's null-context clause. A schema-wide invariant test (`test_rls_schema_invariant.py`) enforces that every table with a `tenant_id` column has RLS forced, not just enabled, so this can't silently regress table-by-table. ### Verify RLS is active ```bash curl -H "Authorization: Bearer $JWT" https://your-domain/api/tenants/rls-status ``` Owner-only diagnostic: lists every table with its `rowsecurity` status from `pg_tables`, plus any table that has a `tenant_id` column but no RLS policy. Tenant metadata itself is managed via `GET`/`PATCH /api/tenants/current` (name, settings) -- see `ADMIN_GUIDE.md`'s "Tenants & Multi-Tenancy RLS" section for the full endpoint reference. --- ## License Installation The recommended path for every deployment is upgrading a running trial into a licensed install -- no file to copy, no restart required: 1. Log in and go to **Settings -> Platform License**. 2. Click **Enter License Key**, then paste the signed license JSON you received. Only an owner can do this. 3. Click **Activate**. This calls `POST /api/trial/upgrade`: the signature is checked first, and a forged, edited, unsigned or expired file is refused with a message and nothing is saved. A valid license takes effect immediately, without a restart, and is recorded in the audit log. After that the trial screen no longer appears, and every feature area and the limits in the license apply. 4. Verify: ```bash curl http://localhost:8000/api/system/license ``` This same flow also clears an expired-trial lockout (`HTTP 423`) -- the same "paste your license JSON" screen appears automatically when a trial expires. When an installed license itself lapses (after its offline grace period) the screen says the license has expired, and pasting a renewed license clears it. ### Advanced: file-based licensing (`LICENSE_MODE=local`) For deployments that should never run through the trial gate at all (for example, a pre-licensed customer rollout), set: ```env LICENSE_MODE=local LICENSE_FILE_PATH=/app/license/local.json LICENSE_PUBLIC_KEY= ``` The default install does not bind-mount a host directory to `/app/license` inside the `api` container, so you must add one yourself -- create a `docker-compose.override.yml` alongside your `docker-compose.yml`: ```yaml services: api: volumes: - ./licenses:/app/license:ro ``` `LICENSE_PUBLIC_KEY` already holds the vendor public key (the same key is built into the images), and a blank value behaves the same, so leave it alone unless you trust a different issuer; the install and upgrade scripts fill a blank value and never overwrite one you set. Place your signed `.license.json` file at `./licenses/local.json` on the host, then: ```bash docker compose up -d api curl http://localhost:8000/api/system/license ``` `LICENSE_MODE=evaluation` is the built-in default. `LICENSE_MODE=dev` skips license/trial enforcement entirely -- intended for a development machine only, and never written by an installer or an upgrade. ### Plan & Billing (self-service) **Settings -> Plan & Billing** shows your usage against your limits and the source of your entitlement (license, subscription, trial, or the platform-owner instance). On a subscription install it also lets you upgrade or downgrade, switch between monthly/annual/lifetime billing, add add-ons (pending until payment is recorded) and redeem promo codes; on a licensed install, capacity and add-ons come from your license and are changed by your vendor through an updated license. How you bill your own customers (metering or a payment gateway, usage statements and invoice drafts) is covered in the Admin Guide, section 11. The installer's "Initial Plan" prompt only sets a starting preference -- it never limits your trial and can be changed at any time from this screen. --- ## Backup Strategy Backups are managed by ErezAgent itself, not an external cron job -- open **Backup Center** in the sidebar for full management: - **Overview** -- last backup status, total jobs, storage used, failed count. - **Jobs tab** -- every backup job with status; a completed/verified job shows a **RESTORE** button with a dry-run option (validates the archive without writing data). - **Repositories tab** -- add backup repositories: a local path or an S3-compatible bucket. - **Schedule tab** -- create backup schedules with preset cron expressions (daily / 6h / weekly / monthly) or a custom cron, backed by the real `backup_policies` table. A background job checks every 5 minutes for policies that are due and runs them -- no host-level `crontab` entry is needed. **Settings -> Backup & Restore** gives quick access to the 10 most recent jobs, an inline **Restore** button, and a **Backup Now** button for an immediate full backup. A full backup archive bundles your database, Obsidian vault, config, repos, license files, and every generated-asset directory (creative assets, generated documents, persona avatars) into one artifact -- there is nothing to configure separately to cover those. ### Data retention A daily maintenance job (04:00 UTC) removes aged operational data in batches: events after 90 days (`EVENT_RETENTION_DAYS`), audit logs after 730 days (`AUDIT_LOG_RETENTION_DAYS`), dismissed or rejected attention signals after 30 days, and email events after 1 year; contact emails on closed deals are anonymised after 6 months (closed lost) or 1 year (closed won). Rows are deleted in batches of `RETENTION_DELETE_BATCH_SIZE` (default 5000), and **only after a backup completed within the last `RETENTION_REQUIRE_BACKUP_WITHIN_HOURS` (default 48)**. If there is no recent backup, the run is skipped and an attention signal asks you to run one. If you take backups outside ErezAgent, mark backups as managed externally in Backup Center so the job is not skipped. Conversations, tasks, model-call logs and approvals are not purged by this job. A separate daily job (03:40 UTC) deletes uploaded Business Twin source files and their extracted text `TWIN_SOURCE_RETENTION_DAYS` days (default 90; `0` keeps them) after a bundle finishes; the evidence quotes kept with the model are not removed. ### Manual backup commands ```bash # Trigger a backup via API curl -X POST http://localhost:8000/api/backups/trigger \ -H "Authorization: Bearer $EREZAGENT_API_KEY" \ -H "Content-Type: application/json" \ -d '{"backup_type": "full"}' # Restore from a specific job (dry-run first) curl -X POST http://localhost:8000/api/backups//restore \ -H "Authorization: Bearer $EREZAGENT_API_KEY" \ -H "Content-Type: application/json" \ -d '{"dry_run": true}' # Then without dry_run to actually restore curl -X POST http://localhost:8000/api/backups//restore \ -H "Authorization: Bearer $EREZAGENT_API_KEY" \ -H "Content-Type: application/json" \ -d '{"dry_run": false}' ``` ### S3 backup repository Set in `.env` (or add a repository from the Backup Center UI): ```env BACKUP_S3_ENABLED=true BACKUP_S3_BUCKET=my-erezagent-backups BACKUP_S3_REGION=us-east-1 BACKUP_S3_ACCESS_KEY=... BACKUP_S3_SECRET_KEY=... ``` ### NFS backup repository ```env BACKUP_NFS_ENABLED=true BACKUP_NFS_HOST=... BACKUP_NFS_EXPORT_PATH=/exports/backups ``` ### Disaster-recovery secrets escrow `LICENSE_PRIVATE_KEY` and `CREDENTIAL_ENCRYPTION_KEY` cannot be regenerated and are deliberately never bundled into the backup archive (an unencrypted archive is the wrong place for live secrets). Retrieve them once and store them in a password manager, outside this server: ```bash curl -H "Authorization: Bearer $EREZAGENT_API_KEY" \ http://localhost:8000/api/backups/dr-secrets-escrow ``` Returns `license_private_key`, `credential_encryption_key`, and `jwt_secret_key`. Losing the server without these saved elsewhere means every stored connector credential becomes permanently undecryptable and no new customer license can ever be issued again -- do this before you need it, not after. --- ## Updating ErezAgent **The upgrade scripts keep themselves current (v4.94.3 and later).** `upgrade.sh` and `upgrade.ps1` compare their own version with the version you ask for. When they are older, they download that release of `ErezAgent-Deploy` (a tagged version that never changes) from GitHub, refresh the `installer/` and `lib/` folders, `docker-compose.yml`, `install.sh` and `install.ps1`, and continue with the new script. What they replace is kept in `installer.backup-/`. `.env`, `docker-compose.override.yml`, backups, licences and data are never touched, so put your own compose settings in `docker-compose.override.yml`. If the release cannot be fetched, the script stops before changing anything and says what to do. `--skip-self-update` / `-SkipSelfUpdate` continues with the folder as it is. Without internet access, download the release archive yourself (`https://github.com/erezrozenbaum/ErezAgent-Deploy/archive/refs/tags/v.zip`, or `.tar.gz` on Linux and macOS) and set `EREZAGENT_DEPLOY_ARCHIVE` to its path before running the script. If the `ErezAgent-Deploy` repository is private for you, the automatic download needs a read-only GitHub token: set `EREZAGENT_DEPLOY_TOKEN` (the release is then fetched through the GitHub API; the token goes only to `api.github.com` and is never printed). Or download the release in your browser while signed in to GitHub (Releases, or Code > Download ZIP of the tag `v`) and set `EREZAGENT_DEPLOY_ARCHIVE` to the file. `EREZAGENT_DEPLOY_ARCHIVE_URL` points the download at an address you host. **The running version is read from the running installation.** The `VERSION` file only records what the last upgrade wrote, so the scripts ask the running API which version is installed and check, after the services are healthy, that the API and every core service run the version that was downloaded. An upgrade that did not start the new version is rolled back instead of reported as successful, and an installation whose `VERSION` file is ahead of what runs is completed instead of answering "already running". **An installer folder older than v4.94.3 cannot refresh itself.** Refresh it once by hand, from the install folder (the one that holds `.env` and `docker-compose.yml`), then run the upgrade. First download the release zip of the version you are installing (`ErezAgent-Deploy`, tag `v`: the Releases page, or Code > Download ZIP of the tag, while signed in to GitHub) and copy it to the server. Windows (PowerShell): ```powershell $v = "" $zip = "C:\path\to\ErezAgent-Deploy-$v.zip" Expand-Archive $zip "$env:TEMP\erezagent-$v" -Force $src = (Get-ChildItem "$env:TEMP\erezagent-$v" -Directory | Select-Object -First 1).FullName Copy-Item docker-compose.yml "docker-compose.yml.before-$v" -Force Copy-Item "$src\installer" . -Recurse -Force Copy-Item "$src\lib" . -Recurse -Force Copy-Item "$src\docker-compose.yml" . -Force .\installer\upgrade.ps1 $v ``` Linux and macOS: ```bash v="" tar -xzf "/path/to/ErezAgent-Deploy-${v}.tar.gz" -C /tmp # or unzip the .zip src="$(ls -d /tmp/*ErezAgent-Deploy*${v}* | head -n 1)" cp docker-compose.yml "docker-compose.yml.before-${v}" cp -R "${src}/installer" "${src}/lib" . cp "${src}/docker-compose.yml" . bash installer/upgrade.sh "$v" ``` (A `git pull` in a clone of `ErezAgent-Deploy` does the same. Keep your `.env` and `docker-compose.override.yml`.) **Installed with the Web Installer?** Start the Web Installer again: when the install folder (default `C:\erezagent` or `~/erezagent`, or the one you type) holds an installation, its first screen shows the version that really runs there next to the version the installer carries and offers **Upgrade** (or **Repair and upgrade** when the settings and the running version disagree). It fetches the release's upgrade script when the folder has none (a web-installed folder holds `.env`, `docker-compose.yml` and `infrastructure/`, but no `installer/` folder), runs it and shows its output: the same backup, migrations, rollback and version check as the scripts. Options: continue without the pre-upgrade backup, or although the free-disk check fails. A folder with no installation goes straight to the normal steps. Without internet access, set `EREZAGENT_DEPLOY_ARCHIVE` to a downloaded release archive before starting the installer. Do not use the installation steps over a working installation to upgrade it: they keep your `.env` and `docker-compose.yml` as they are and so start the version already installed. **Recommended: use the upgrade script** -- it validates the upgrade path, takes a pre-upgrade backup through the Backup Center and stops if it cannot (`--skip-backup` / `-SkipBackup` continues without one; a fresh install with no running API continues with a note), checks that Docker has enough free disk space, downloads the new service images (and the Android build image only when you use Android builds), verifies each pulled image's cosign signature when the release channel publishes its signing key (since v4.49.0 -- a verification failure then blocks the upgrade outright, before anything is restarted; until the key is published the script prints an informational line that signature verification is not enabled for this release and continues), hardens `.env` permissions, health-checks the result, and automatically rolls back if anything fails: ```bash # Linux / macOS -- run from the ErezAgent install root bash installer/upgrade.sh ``` ```powershell # Windows (PowerShell) -- run from the ErezAgent install root .\installer\upgrade.ps1 ``` The upgrade script also appends any settings introduced since your last release to `.env` with safe defaults (it never changes a value you already set; the list covers every optional setting, including the data-retention keys), backfills any internal token that was left blank, corrects the values the API refuses to start with (`LICENSE_MODE=dev`, `DISABLE_AUTH=true`, weak signing secrets), and repairs the ownership of the application data volumes (`/obsidian-vault`, `/repos`, `/creative-assets`, `/generated-documents`, `/persona-assets`) so the non-root `api`/`worker` containers can write uploaded documents and vault notes. The repair only touches a volume that still has files owned by another user, never blocks the upgrade, and is also run by `rollback.sh` and the Web Installer Wizard. A `.env` that does not end with a newline is handled correctly. **Infrastructure files.** `docker-compose.yml` mounts configuration files from `infrastructure/` (Caddy, Prometheus, Loki, Promtail, Tempo, Grafana provisioning, the docker-socket-proxy template). The install and upgrade scripts compare them with the copy the release carries in `installer/web_installer/bundled/infrastructure/` and bring them in step: a missing file is created; a file that is a copy an earlier release shipped (recognised by its sha256 in `installer/lib/infrastructure-known-hashes.txt`) is replaced after its previous copy is kept under `infrastructure.backup-/` (same relative path); a file with any other content is yours, so it is kept as it is, the release's copy is written next to it as `.new`, one warning lists such files and its service is not restarted (merge what you need, then run `docker compose restart `); and files that are not part of the release (your own dashboards) are left alone. On an upgrade this happens right before the services start, and the services that read a replaced file are restarted unless they were started after the files were written. Nothing is replaced when the upgrade stops before changing anything, and a rollback keeps `docker-compose.yml` in place, so it keeps the matching files as well; copy a change back from the backup folder if you need it. The Web Installer Wizard refreshes the files, by the same rule, only when it writes a new `docker-compose.yml`; with an existing one it keeps both. **Provider Fleet (platform-owner installation only).** The Fleet ingress answers at `/api/provider-control/v1/` on the installation's public https address (the same domain as the app); no extra port, proxy rule or environment variable is needed. On any other installation those routes answer 404. **Upgrading to v4.94.5:** migration 566 runs with the API on start. Run the upgrade scripts from the install folder (the one that holds `.env`, `docker-compose.yml` and `installer/`); a mirror address in `EREZAGENT_DEPLOY_ARCHIVE_URL` must be `https://`. The web installer's upgrade continues when its page is closed, and reopening the screen shows the run in progress. **Upgrading to v4.94.4:** no migration. The upgrade scripts and the web installer fetch the release with `EREZAGENT_DEPLOY_TOKEN` when the Deploy repository is private for you. **Upgrading to v4.94.3:** no migration. The upgrade scripts refresh the installer folder from the release they install, read the running version from the API and check that every core service runs it; an installer folder older than v4.94.3 needs the one-time refresh under "Updating ErezAgent". **Upgrading to v4.94.2:** migration 565 runs with the API on start. No installer change. **Upgrading to v4.94.1:** no migration and no installer change. **Upgrading to v4.94.0:** migrations 555-564 run with the API on start. The compose files add health checks, memory limits and image digests for the monitoring services, a logging `tag` option on every service (all containers are recreated once), and an allowlist template for the `docker-socket-proxy` (mounted from `infrastructure/docker-socket-proxy/`; promtail now reads container log files). The installer compose file requires `EREZAGENT_VERSION`. An optional provider connection needs outbound HTTPS from the API and worker to the provider's address; the pinned client ignores proxy environment variables. On an installation with several workspaces, only the home workspace (or a single workspace) keeps using the provider accounts set in `.env`; the other workspaces connect their own HubSpot, Stripe, Rapyd, Pelecard, PayPal, Pipedrive, Salesforce, Hetzner, Vercel, Cloudflare, PostHog and LEON accounts. **Upgrading to v4.93.0:** migrations 553 and 554 run with the API on start. The compose file adds a `docker-socket-proxy` service (the only service besides the sandbox executor that mounts the Docker socket) and promtail's configuration points at it. `docker compose pull` and `docker compose up -d` (and the upgrade scripts) fetch and start the proxy, and the install and upgrade scripts bring the `infrastructure/` files in step with the release (see Infrastructure files). If you replace `docker-compose.yml` by hand, replace the `infrastructure/` folder with it. Docker Compose 2.17 or newer is required. **Upgrading to v4.92.3:** no migration and no installer change. **Upgrading to v4.92.2:** no migration and no installer change. **Upgrading to v4.92.1:** three migrations (agent policies per workspace; Gemini 4 Argon in the built-in catalog; strict workspace isolation for the Chief of Staff tables and a protected audit log) run automatically. **Upgrading to v4.92.0:** no migration. The upgrade scripts take the pre-upgrade backup through the Backup Center and stop if it cannot be created (`--skip-backup` / `-SkipBackup` continues without one) and accept only a plain `MAJOR.MINOR.PATCH` version. The installers keep a timestamped copy of an existing `.env` before replacing it. **Open** on an objective opens its page, approvals show who asked and why, and agent tasks run their safe steps for real: a step that sends or changes something becomes an approval (when a connector is authorised for it) or is shown as blocked with the reason. **Upgrading to v4.91.6:** no migration and no installer change. Executive Office chat and agent replies stay reliable in long conversations that use tools, and answers an agent saves to memory are saved under the right workspace. **Upgrading to v4.91.5:** no migration. Re-download the installer folder first. The upgrade and rollback scripts now add a missing `EREZAGENT_VERSION` line to `.env` and, before stopping anything, check that Docker Compose will start the new version; if a shell variable or a tag written into `docker-compose.yml` or an override file would still start another version, they stop and say so. **Upgrading to v4.91.4:** migration 549 runs automatically and removes the example business records early releases created (deals, a capital plan, a revenue goal, assets, contracts, competitor items, predictions, calibration scores); a row you edited or linked to your own data is kept, and nothing is removed from an installation that already deleted them. No installer change; re-download the installer folder only to keep the scripts current. **Upgrading to v4.91.3:** no migration, the application is unchanged. Re-download the installer folder first. The install and upgrade scripts and the web installer now correct an `.env` the API refuses to start with (`LICENSE_MODE=dev`, `DISABLE_AUTH=true`, signing secrets under 32 characters or placeholders; a replaced `JWT_SECRET_KEY` signs everyone out) and print the API's reason when it still cannot start. **Upgrading to v4.91.2:** no migration. Re-download the installer folder first (`lib/` has new files). See "Disk usage" below for the disk check and the cleanup of older images; completed manual backups beyond the newest 10 that are older than 30 days are deleted by the daily backup job (`BACKUP_MANUAL_KEEP_MIN`, `BACKUP_MANUAL_KEEP_DAYS`). **Upgrading to v4.91.1:** no migration. The upgrade script adds the two optional vendor settings (blank, so the built-in defaults apply). To have trial and license reminders emailed, set a real `EMAIL_PROVIDER` (see ADMIN_GUIDE section 15); the installers now offer SendGrid during setup. **Upgrading to v4.91.0:** migration 548 runs automatically (existing custom domains keep working). Re-download the installer folder (`lib/compose-health.sh` and `lib/compose-health.ps1` are new) before running the upgrade script. The script now waits for every core service to be healthy, reads the API port from `API_PORT`, works when `.env` lists the API key twice and on macOS, and rolls back to the previous version if a core service, the API, the diagnostics check or an image-signature check fails (monitoring containers are reported but never trigger a rollback). In production the API and worker refuse to start with `JWT_SECRET_KEY` or `EREZAGENT_API_KEY` shorter than 32 characters, a placeholder, or `LICENSE_MODE=dev`; check `.env` first. Installs with more than one workspace: move Twilio, Meta Lead Ads and LinkedIn lead webhooks to `/api/webhooks//`. Kubernetes: see KUBERNETES.md for the changed chart defaults. The browser must be current for the web interface (Chrome and Edge 111+, Safari 16.4+, Firefox 128+). **Upgrading to v4.90.0:** nothing manual. Migration 547 runs automatically and moves the fast tier to Claude Haiku 5.5 (rows you changed are left as they are). The OpenAI voice option uses `gpt-transcribe` and `gpt-realtime-2.1-mini` with a one-time fallback to `whisper-1` / `tts-1`; the upgrade script adds four optional `OPENAI_*_MODEL` settings to `.env`. **Upgrading to v4.89.3:** nothing manual. Migration 546 runs automatically. Kubernetes installs set `appDatabase.host` (or keep `APP_DATABASE_URL` in their own Secret) before `helm upgrade`; see KUBERNETES.md. **Upgrading to v4.89.1 / v4.89.2:** the installers generate the credential encryption key in the required format and the upgrade scripts repair a malformed key in `.env` (a malformed key cannot have encrypted anything). The API shows a fix hint if the key is not valid; set a valid key (the Fernet key command is in the CHANGELOG) and run `docker compose up -d --force-recreate api worker`. **Upgrading to v4.89.0 (usage and billing foundation):** nothing manual. A license pasted into the UI of v4.88 or earlier is copied to the new license volume by the upgrade script; if the license screen shows expired after upgrading, paste your license again. The upgrade runs migrations 529-545 and fills a blank `LICENSE_PUBLIC_KEY` with the vendor key (a value you set is kept). Expect these behavior changes: billing changes are owner only (operators read, observers have no billing access), marking an invoice paid needs an amount, a reference and a date, a receiver built for the earlier billing webhook push must be updated (new payload and signature), and the `api` container may need up to two minutes to become healthy after the upgrade. **Upgrading to v4.88.0 (Business Twin Builder):** nothing manual. The upgrade runs migrations 508-528, appends four new optional keys to `.env` with their defaults (`TWIN_MAX_FILES_PER_BUNDLE=25`, `TWIN_MAX_BUNDLE_MB=200`, `TWIN_ANALYSIS_COST_CONFIRM_USD=5.00`, `TWIN_SOURCE_RETENTION_DAYS=90`) and pulls images that already include the new document readers (`pypdfium2` and the Office/PDF libraries in the `worker`). Uploaded Build My Business files are stored under `twin-sources/` in the existing vault volume, so they are covered by your normal backups. Two behaviors to expect: AI analysis of documents is off until the owner grants consent in Settings > Data & Privacy, and the default mode masks identifiers before text is sent to an AI provider. For fully local analysis, run an Ollama server (`OLLAMA_BASE_URL`) and, for scanned pages and images, `ollama pull qwen2.5vl` on that host. **Rolling back:** the upgrade script rolls back automatically if a step, the API health wait, the check that every core service is running and healthy, or the final diagnostics fail (a license problem is reported as a warning and does not trigger a rollback; monitoring containers are reported but not required). It reaches the API on the `API_PORT` set in `.env` (default 8000). It reverts the database schema only to the revision recorded just before migrations ran, and only when migrations were actually applied. To roll back by hand, run `bash installer/rollback.sh [backup_job_id]` on Linux/macOS or `.\installer\rollback.ps1 [backup_job_id]` on Windows from the install root; both accept only a plain `MAJOR.MINOR.PATCH` version. A rollback to an older version also reverts the database schema to that version's migration head, which removes the tables and columns that newer migrations added and the data in them (leaving v4.89.0 removes the usage ledger, statements, invoices, payments and rendered billing documents). The script lists what will be removed, asks you to type `confirm` (`--yes` / `-Yes` skips the prompt; `--dry-run` / `-DryRun` only prints the plan), and pulls and verifies the target images while the running version keeps serving, then stops the application services, writes a database dump into your backups folder and reverts the schema. If the schema revert fails, the script stops before any image or version is changed, starts the current version again and prints how to restore the dump. With a backup job ID they also restore the database and vault from that checkpoint (the upgrade script saves the pre-upgrade checkpoint's ID in `.upgrade_backup_id`); without one the data is left as the schema revert leaves it. **Manual alternative** (no compatibility check, no backup checkpoint, no auto-rollback): ```bash cd /path/to/erezagent # Pull latest images docker compose pull # Restart with new images docker compose up -d # Run any new migrations docker compose exec api alembic upgrade head ``` --- ## Auto-start on Reboot (systemd) Create `/etc/systemd/system/erezagent.service`: ```ini [Unit] Description=ErezAgent AI Company OS After=docker.service Requires=docker.service [Service] Type=oneshot RemainAfterExit=yes WorkingDirectory=/opt/erezagent ExecStart=/usr/bin/docker compose up -d ExecStop=/usr/bin/docker compose down TimeoutStartSec=300 [Install] WantedBy=multi-user.target ``` Enable: ```bash sudo systemctl daemon-reload sudo systemctl enable erezagent sudo systemctl start erezagent ``` --- ## Security Hardening ### Firewall (ufw) ```bash sudo ufw default deny incoming sudo ufw default allow outgoing sudo ufw allow ssh sudo ufw allow 'Nginx Full' # or: sudo ufw allow 80 && sudo ufw allow 443 sudo ufw enable ``` ### Fail2Ban ```bash sudo apt install fail2ban -y sudo systemctl enable fail2ban ``` ### Docker daemon hardening Add to `/etc/docker/daemon.json`: ```json { "no-new-privileges": true, "live-restore": true, "userland-proxy": false, "log-opts": { "max-size": "10m", "max-file": "3" } } ``` Restart Docker after editing: `sudo systemctl restart docker`. ### Port binding (since v4.49.0) `api` (`8000`) and `frontend` (`3000`) now bind to `127.0.0.1` on the host, the same convention already used for the observability stack (Prometheus, Grafana, cAdvisor, node_exporter, Loki, Tempo). Caddy remains the sole public ingress for every service -- local access via `http://127.0.0.1:8000` / `http://127.0.0.1:3000` on the server itself is unaffected; nothing changes for a deployment already reaching the app exclusively through Caddy/your domain. ### Docker socket access (since v4.93.0) cAdvisor reads container information through a filtered socket proxy (`docker-socket-proxy`) on an internal Docker network that only it shares; the proxy publishes no port. It answers four read requests (a container's details, the daemon's information and version, and the ping) and refuses every other request, including container logs, file archives, exports, process lists, exec and every write. A client of the proxy can still read each container's configuration, including its environment, so nothing else is on that network. Besides the proxy, only the sandbox executor mounts the Docker socket. Promtail does not use the Docker API: it reads the log files Docker writes under `/var/lib/docker/containers` (the Docker data root must be that folder), and every service's `tag` log option gives each line its container name. A container started before that option existed appears in Loki once it is recreated. cAdvisor starts after the proxy without waiting for it to be healthy, so an install or upgrade completes even where the proxy cannot use the socket (strict SELinux, rootless Docker, a non-standard socket path); `docker compose ps` then shows the proxy as unhealthy and cAdvisor reports containers by ID (`docker compose logs docker-socket-proxy`, then `docker compose restart docker-socket-proxy`). Container names in cAdvisor need the classic `overlay2` storage driver; on Docker 29 and Docker Desktop with the containerd image store they may be missing, and the Container Crash Loop alert then counts restarts by container ID. `docker compose pull` and `docker compose up -d` (and the upgrade scripts) fetch and start the proxy together with the rest of the stack. The `infrastructure/` folder must be the one of the same release as `docker-compose.yml` (the proxy mounts `infrastructure/docker-socket-proxy/haproxy.cfg.template` and promtail's configuration reads the tagged log files); the install and upgrade scripts make sure of it, see "Infrastructure files" under Updating ErezAgent. ### Grafana access behind a domain Once `DOMAIN` is set (Step 5 above) and Caddy is fronting the stack, `/grafana/*` requires your existing ErezAgent login -- Caddy checks your session against `GET /api/auth/verify-session` via `forward_auth` before ever proxying to Grafana, and Grafana's own anonymous-access setting (`GF_AUTH_ANONYMOUS_ENABLED`) is off by default now too. Log into ErezAgent normally first, then open `https://yourdomain.com/grafana/`. Without `DOMAIN` set (a local-only deployment reaching Grafana directly at `http://localhost:3001`), this proxy layer doesn't apply. ### Sandbox network mode `SANDBOX_NETWORK_MODE` (set in `.env`) controls network access for agent/skill-triggered shell, git, and build commands inside the isolated sandbox container: - `default` (the default) -- unrestricted egress. - `scoped` (new in v4.49.0) -- outbound access is allowed only to a fixed allowlist of git hosts (GitHub, GitLab, Bitbucket) and package registries (PyPI, npm), everything else blocked at the kernel level (iptables, real DROP, not just DNS filtering). Covers the common case -- `git clone`/`git push` to a real host and `pip`/`npm install` all keep working -- while closing off arbitrary outbound network access from generated/agent-run code. Verified live: an allowlisted request succeeds, everything else times out at the connection level. - `isolated` -- no network access at all -- `git clone`/`git push` and any package install genuinely stop working in that mode. Only set this for a deployment whose sandboxed workloads are pure local build/test with no real network need. Switching to `scoped` or `isolated` after an existing `default` deployment requires a fresh build of the sandbox-executor utility image (`docker compose build sandbox-executor` or equivalent for your deployment) before restarting -- an old cached image built under `default` mode won't have the tooling the new modes need. ### Default credentials are refused outside development Outside `APP_ENV=development`, the API now refuses to start if `POSTGRES_PASSWORD`, `REDIS_PASSWORD`, `GRAFANA_ADMIN_PASSWORD`, `DATABASE_URL`, or `REDIS_URL` still resolve to the placeholder values `env.example` ships with. If a deployment fails to start after an upgrade with a message about default credentials, it means `.env` was never fully edited during initial setup -- set real values for each of those before starting outside development mode. --- ### Outbound access for provider reporting (optional, since v4.94.0) An owner can connect the installation to a managed-service provider (Settings > Provider / Managed Service). Nothing is sent until they do. When connected, the **api** container (enrolment, owner actions) and the **worker** container (the report, about every 15 minutes) make outbound HTTPS requests to the single address in the enrolment string, port 443 by default. If your firewall restricts egress, allow those two containers to reach that address; nothing else is needed. The request uses the system CA store, does not follow redirects and **ignores the `HTTP_PROXY` / `HTTPS_PROXY` environment variables**, so an installation that can only reach the internet through a corporate proxy cannot use this feature. The address must be a public https address; localhost, private networks, cloud metadata addresses and this stack's own service names are refused. Keep the server clock accurate (NTP): a request whose timestamp is more than 5 minutes off is rejected by the provider. ## Monitoring ### Container health ```bash docker compose ps docker compose logs api --tail=50 -f ``` ### Resource usage ```bash docker stats ``` ### Disk usage Images take about 20 GB with the previous version kept for rollback (about 15 GB more with Android builds). Upgrades check free space on Docker's disk before downloading and stop with nothing changed when it is not enough (`--force` / `-Force` continues anyway). Only what must be downloaded is counted: an image that is unchanged from the one already on the server is not (this needs the registry to answer; when it does not, the full size is assumed). After a successful upgrade the images of older ErezAgent versions are removed, keeping the current and the previous version and the version the installation was running before the upgrade; `--no-prune` / `-NoPrune` keeps them. Rollback to the previous version works without internet access while its images are still on the server. On Docker Desktop (Windows, macOS) freed space is available to Docker again, but the disk image file on your computer does not shrink by itself. ```bash df -h du -sh /path/to/erezagent/data/ ``` Grafana (`http://localhost:3001`, `admin` / `GRAFANA_ADMIN_PASSWORD` from `.env`) ships with 6 dashboards covering 35+ Prometheus metrics; Loki collects logs from every service. --- ## Troubleshooting | Symptom | Cause | Fix | |---|---|---| | API won't start | Missing or invalid required `.env` variable | `docker compose logs api` -- look for `ValidationError` | | Frontend shows a blank page | API unreachable from the browser, or wrong build-time API URL | Check `VITE_API_BASE_URL` in `.env`, verify `docker compose ps` shows `api` healthy, then `curl http://localhost:8000/api/health` | | Obsidian connector fails | Obsidian not running, or wrong URL | Confirm Obsidian is open; use `host.docker.internal:27123`, not `localhost:27123` | | Obsidian connector fails on Linux specifically | `host.docker.internal` is not resolved by default on Linux | Add `extra_hosts: ["host.docker.internal:host-gateway"]` to the `api` service in `docker-compose.yml` | | Caddy can't get an SSL certificate | Domain not pointing at this server yet | Check DNS propagation: `dig yourdomain.com`; confirm ports 80/443 are open | | `docker compose exec api alembic upgrade head` fails | Migration state conflict | `docker compose exec api alembic current` to check the current revision, then compare against `EXPECTED_MIGRATION_HEAD` on the System Health page | | `column ... does not exist` / `relation ... does not exist` in worker/connector/security logs right after a fresh install or upgrade | `worker`, `connector_service`, and `security_service` started before `api` finished migrations | `docker compose pull && docker compose up -d` -- the current compose file makes those three services wait on `api: condition: service_healthy` before starting | | Locked out (forgot password or lost authenticator) | -- | Use **Forgot password?** on the login page, or `docker compose exec api python -m scripts.reset_credentials ` -- see "Account Recovery" in `ADMIN_GUIDE.md` | | API or worker refuses to start with a message about a secret or `LICENSE_MODE` | In production the secrets must be at least 32 characters and not placeholders, and `LICENSE_MODE=dev` is not allowed | The install and upgrade scripts correct these values in `.env` and print the API's reason when it still cannot start. To fix it by hand: set long random values for `JWT_SECRET_KEY` and `EREZAGENT_API_KEY` (`openssl rand -hex 32`) and a real `LICENSE_MODE`, then `docker compose up -d --force-recreate api worker` | | A `docker compose` command stops with "EREZAGENT_VERSION is not set" | `.env` has no `EREZAGENT_VERSION` line; the compose file has no default version and refuses to start unpinned images | Add `EREZAGENT_VERSION=` to `.env` (the install and upgrade scripts write it), then run the command again | | An upgrade said it succeeded, but the app (and the guides) still show the old version | An installer folder older than v4.91.5 wrote the new version into the `VERSION` file while `.env` had no `EREZAGENT_VERSION` line, so Docker kept starting the old images (tagged `latest`) | Refresh the installer folder once (see "Updating ErezAgent"), then run the upgrade again: it reads the version from the running API, finds that the services are behind, and completes the upgrade. By hand: add `EREZAGENT_VERSION=` to `.env`, then `docker compose pull` and `docker compose up -d --force-recreate` | | The app still shows the old version after an upgrade | A shell variable or an override file pins another tag for the `erezagent-` images | Check `docker compose config` lists the version you downloaded for the `erezagent-` images, remove the variable or override, then `docker compose up -d --force-recreate`. The upgrade script stops with a message if something else decides the tag | | API stops at the database-migration step on a new install | `CREDENTIAL_ENCRYPTION_KEY` is not a valid Fernet key | Generate one with the Fernet key command shown in the CHANGELOG, set it in `.env`, then `docker compose up -d --force-recreate api worker` | | Upgrade fails with `Invalid version format: Not a valid semver string: 'unknown'` | No `VERSION` file exists yet in the install directory, so the upgrade script can't determine what's currently installed | Run `docker compose pull && docker compose up -d && docker compose exec api alembic upgrade head` once manually -- this also writes the `VERSION` file -- then use the upgrade script normally for future upgrades | | Uploaded documents or vault notes fail to save with a permission error after an upgrade | A data volume created by an older release is owned by another user | Run the next upgrade with `installer/upgrade.sh`, which repairs ownership, or `chown -R appuser:appgroup ` from a one-off root container of the `api` image | | Data retention seems paused (attention signal "Data retention paused: no recent backup") | No backup completed within `RETENTION_REQUIRE_BACKUP_WITHIN_HOURS` | Run a backup from Backup Center, or mark backups as managed externally | | Database connection error | `POSTGRES_PASSWORD` in `.env` doesn't match what the `db` volume was initialized with | `docker compose logs db`; note that editing `POSTGRES_PASSWORD` after first install has no effect on an existing data volume -- see the password-rotation guidance in `ADMIN_GUIDE.md` | | System Health shows a migration-head mismatch (`Current: N, Expected: M`) | Database migrations and the version you're running are out of step | `docker compose exec api alembic upgrade head`, then `docker compose pull && docker compose up -d` to make sure you're on matching images | --- ## Support | Plan | Support channel | |---|---| | Starter, Builder, Agency, Business, Business Plus | Email support | | Enterprise and Provider (custom-priced) | SLA-backed support | Response times and SLA terms are not defined by the product; see your plan agreement.