# ErezAgent -- Admin Guide

**Current version: v4.94.5**

This guide covers everything a platform owner or administrator needs to configure, operate, and
govern ErezAgent. It's organized by administrative domain, not by release -- every feature is
described as it works today, with a small `(since vX.Y.Z)` tag where the version matters. For
installing or upgrading ErezAgent itself, see `docs/DEPLOYMENT_GUIDE.md` -- this guide assumes a
running deployment and focuses on what you configure and govern inside it.

---

## How this guide is organized

1. [Access Management](#1-access-management)
2. [Tenant & Venture Setup](#2-tenant--venture-setup)
3. [Company Foundation & Constitution](#3-company-foundation--constitution)
4. [Persona & Workforce Identity Administration](#4-persona--workforce-identity-administration)
5. [Agent Policies & Autonomy](#5-agent-policies--autonomy)
6. [Departments & Cost Centers](#6-departments--cost-centers)
7. [Connectors & Integrations](#7-connectors--integrations)
8. [Backups & Disaster Recovery](#8-backups--disaster-recovery)
9. [Security Policy](#9-security-policy)
10. [Data Retention & GDPR](#10-data-retention--gdpr)
11. [Licensing](#11-licensing)
12. [System Health & Diagnostics](#12-system-health--diagnostics)
13. [Human/AI Workforce Unification Administration](#13-humanai-workforce-unification-administration)
14. [Installation & Upgrade](#14-installation--upgrade)
15. [Environment Variables Reference](#15-environment-variables-reference)
16. [Background Jobs & Automation](#16-background-jobs--automation)
17. [Observability & Monitoring](#17-observability--monitoring)
18. [API & Automation Access](#18-api--automation-access)
19. [Troubleshooting](#19-troubleshooting)
20. [Business Operations Runtime Administration](#20-business-operations-runtime-administration)
21. [Business Pulse & Operating Doctrine Administration](#21-business-pulse--operating-doctrine-administration)
22. [Operational App Runtime Administration](#22-operational-app-runtime-administration)
23. [Business Twin (Build My Business) Administration](#23-business-twin-build-my-business-administration)
24. [Provider / Managed Service Administration](#24-provider--managed-service-administration)

**Navigation note:** the sidebar opens in Core view (10 screens) for new accounts; use the All screens toggle for the full
list. The sidebar is organized into 13 groups (Getting Started, Executive, Command,
Strategy & Governance, Research & Signals, Knowledge & Learning, Decisions & History, Cost &
Finance, Revenue & CRM, Cloud Ops, Governance, Workspace, Platform). Where this guide names a
screen, that's the current name and group -- see `docs/UI_GUIDE.md` for the full screen-by-screen
reference if you want to confirm a location.

---

## 1. Access Management

### Roles

Every user has exactly one role:

| Role | Can do |
|---|---|
| **Owner** | Everything -- including approving high-risk actions, issuing platform licenses, managing signing credentials, and every action below that says "owner-only." There is normally exactly one owner per deployment. |
| **Operator** | Day-to-day work -- create and edit missions, tasks, notes, drafts, conversations and memories; manage customers, customer-success accounts, sales leads and growth campaigns/assets; soft-delete departments; cancel goals and AI plans; archive strategic lessons. Cannot approve actions, except a step assigned to a business approver role they hold, and cannot execute agents (owner-only). Cannot use owner-only controls (below); those appear disabled with an "Owner role required" note, and a failed action rolls back and shows the reason. |
| **Observer** | Read-only. Can view dashboards, reports, and history but cannot create, edit, or approve anything. |

**Owner-only (v4.87.0):** hard-deleting or cancelling, and deactivating/disabling/revoking/rotating, the following
require the Owner role: missions (delete and cancel), projects, contracts, revenue deals, sales companies, finance
subscriptions, FinOps budgets, resources, strategy plans and autonomous rules, security pentest targets, agent
policies, attribution chains, workflows, licensing API keys, venture custom domains (claim and delete), voice
profiles, delegation cancellation, connector-action cancellation, business-policy circuit-breaker reset,
capability-engine and configurable-connector deactivation, and secret/key and deploy-provider rotation. The full
matrix is in `docs/12_AUTH_AND_SESSIONS.md`.

**Also owner-only (v4.88.0):** changing anything in Build My Business (operators and observers can read it), granting or
revoking AI-analysis consent, creating or editing venture staff role definitions, changing an agent's status, budget,
model, enabled flag or prompt override for your organization, restoring an earlier workflow version, changing whether a workflow requires approval, and approving a
Business Twin build (the approval itself can only be given by an owner). Revoking a skill grant that
the shared core agents come with removes it for your organization only (it can be granted again), and also needs an owner.

**Agent budgets:** the Legal agent's documented budget (10,000 tokens/day, $8/month) is set on new installs only; on installs created earlier it shows as unset until an owner applies it through an approved improvement proposal (agent budgets never change automatically).

Manage users from **Users** (Platform group). Adding a team member creates their account with a
role you choose; you can change a user's role or deactivate their account at any time from the
same screen.

### Provider staff (v4.94.0)

On the platform owner's installation, the owner can invite provider staff from the Fleet page (NOC / Operations, Support engineer, Security operator, Billing operator, Auditor). They sign in through the normal login page, see only the Fleet page, and see only the customers the owner assigns. Their access is re-read on every request; ending it takes effect on the next one. A person cannot be both provider staff and venture staff.

### Venture Staff (v4.84.0)

A separate, lighter tier from the three roles above, scoped to one venture instead of your whole
account: from a venture's own **Staff** panel (Ventures page -> open a venture -> **Staff**),
invite an employee as a Staff Member or Staff Manager. They set their own password from an emailed
invite link and log in through the normal login page -- their account technically carries the
`observer` role underneath (so if you ever look them up in Users, that's expected), but their real
access is entirely governed by their venture membership, not that role: they can read and write
that one venture's business records, documents, support cases, appointments, and tasks, and have
no access at all to any other venture, even one you also own. Revoking access from the Staff panel
takes effect on their very next request, not after their session expires.

Staff seats are capped: a tenant-wide total (part of your subscription plan) plus an optional
per-venture limit if you want to divide seats unevenly across ventures -- set from the venture's
own settings.

**Where staff work (v4.85.0):** a staff session reaches only the five areas explicitly wired for
it -- Business Records, Document Collection, Support Cases (Incidents), Appointments, and Tasks --
plus My Work and the Operational Dashboard, automatically scoped to the venture(s) they're a member
of. Every other route in the system rejects a staff token by default, the same structural rule that
already rejects a password-reset or MFA-pending token as a session -- broadening staff reach to a
new area of the product is a deliberate future change to that specific router, never an accidental
side effect of this feature.

**Staff workspace (v4.86.0, refined v4.87.0):** after sign-in a staff member lands in a focused workspace
(My Work, Operational Dashboard, Business Records, Document Review, Incidents, Appointments) with its own sign-out.
Incidents and Appointments work with the staff member's own access: the customers and ventures they may choose
come from a staff-specific lookup, owner-only appointment setup controls are hidden, and staff can run AI outcome
extraction on appointments in their own ventures (rate-limited, audited, and a draft until someone confirms it).

**Staff role definitions (v4.88.0):** each venture can define named roles, managed from the venture's **Staff** panel
(**Roles** tab, owner-only to create, edit or archive). A role lists, per record type, whether holders can view, create
and edit records, which stages they may move a record to, and whether they see all records or only those assigned to
them, plus the pages and actions they may use and, through the record type's field settings, which fields they can see.
Editing a role saves a new version and takes effect on the member's next request. Assign a role when inviting someone or
from the member's row on the **Team** tab. Staff with no role keep the original full staff access, so existing accounts
are unchanged. Archiving a role leaves its members with no access until you assign them another role. Role changes are
written to the Audit Log.

**Read-only Observer (v4.86.0):** the Observer role is read-only everywhere. Any attempt to create, change, or
delete returns a clear "Observer role is read-only" message; chat, voice, and previews remain available. Uploads
are limited to 25 MB for documents, 10 MB for avatars, and 5 MB for CSV imports.

### Sessions

Login issues a JWT session token. Session lifetime is controlled by `JWT_TTL_HOURS` (default 168
hours / 7 days) -- reduce this for a higher-security environment. Rotating `JWT_SECRET_KEY`
immediately invalidates every existing session.

**Since v4.49.0**, every user carries a server-side session version. Logging out, changing your
own password, or having an owner reset your password or change your role immediately invalidates
every session token issued before that action -- a token that's been logged out (or a stale copy
of it) is rejected on its very next request, not just deleted client-side. Passwords require 12+
characters, must not be purely numeric, and must not be one of the most common weak passwords.
Password hashing uses Argon2id; any account still on the older hash format upgrades transparently
the next time that user logs in successfully -- no visible change, no forced reset.

### Two-factor authentication (TOTP MFA)

Every user can enable optional per-user TOTP MFA (RFC 6238), compatible with Google Authenticator,
Authy, 1Password, Bitwarden, and any standard authenticator app.

**Enable it:** Settings -> Two-Factor Authentication -> **Enable MFA** -> scan the QR code -> enter
the 6-digit code to confirm.

**Login with MFA on:** enter username + password, then a second screen asks for the current
6-digit code.

**Disable it:** Settings -> Two-Factor Authentication -> **Disable MFA** -> confirm with your
password and current code.

MFA is optional and per-user -- there is no policy to force it across every account. Every MFA
secret is Fernet-encrypted at rest using `CREDENTIAL_ENCRYPTION_KEY`, the same key that protects
connector credentials.

### Account recovery

If a user is locked out, two paths are available:

**Self-service email reset** (works only when email delivery is configured -- `EMAIL_PROVIDER` set
to `sendgrid` or `gmail`, not the default `mock`): click **Forgot password?** (or **Lost your
authenticator?** on the MFA screen), enter the account email, and a reset link valid for 30 minutes
arrives if the address matches an active account. The reset page can also clear MFA in the same
step.

**CLI recovery** (always available, no email required -- the real safety net for a self-hosted,
typically single-owner deployment):
```bash
docker compose exec api python -m scripts.reset_credentials <username>
docker compose exec api python -m scripts.reset_credentials <username> --reset-mfa
```
The script prompts for the new password interactively (never as a CLI argument, so it never lands
in shell history) and hashes it the same way the API does; `--reset-mfa` also clears MFA so the
user can re-enroll after logging back in.

A self-service reset link is now single-use -- redeeming it invalidates it immediately, and
requesting a new one invalidates any earlier link you hadn't used yet. On the Users page, an owner
changing another user's role, active state, or password is now asked to re-confirm their own
current password first (step-up re-authentication) before the change is applied.

---

## 2. Tenant & Venture Setup

### The multi-tenancy model

ErezAgent is multi-tenant by design. Every tenant-scoped table carries a `tenant_id` column and a
PostgreSQL Row-Level Security (RLS) policy that filters every query and write to the caller's own
tenant. The application connects to the database as a dedicated, non-superuser role
(`erezagent_app`) specifically because PostgreSQL superusers are not subject to RLS policies -- running as a non-privileged role is what makes tenant isolation a real, database-enforced
guarantee rather than an application-level convention. `TenantContextMiddleware` extracts
`tenant_id` from the JWT on every request and sets it as a transaction-local Postgres setting
(`app.current_tenant_id`); every RLS policy checks that setting before allowing a row to be read or
written. Background workers and migrations run with no tenant setting, which is the deliberate,
audited convention this system uses for "operate across all tenants" -- never a bypass of a real
per-request boundary.

A single-tenant deployment (the common case for a self-hosted install serving one business) never
needs to think about this at all -- one tenant is seeded automatically from the first owner user.

**Per-tenant AI workforce settings (since v4.88.0):** the core agents (CEO, CFO and the rest) are shared templates, but each
tenant keeps its own settings for them -- enabled or not, status, daily token budget, default model, an optional prompt
override and department. `GET /api/agents` returns your tenant's merged values, with `is_shared_template`,
`has_tenant_config` and `has_prompt_override` flags and the `department_name`. An owner changes them with
`PATCH /api/agents/{id}/config` (budget, default model, enabled, prompt override) and `PATCH /api/agents/{id}/status`; both
are owner-only and apply to your tenant only, never to another tenant or to the template. A prompt or permission change is
still applied through an owner-approved improvement proposal, as in section 5; changing the shared template itself is
reserved for the platform owner.

Tenant isolation is enforced by the database itself (row-level security on every table that carries a tenant), and
every release is checked against the live schema before shipping. As an owner, confirm overall platform health from
**System Health -> Live Health** and the **Readiness Check** (section 12). The detailed per-table isolation report,
`GET /api/tenants/rls-status`, is a platform-operator diagnostic: it is restricted to platform operators and returns
403 for a customer owner account.

### Moving older vault notes into each tenant's area (shared installs)

Before v4.89.0 the document vault was one shared folder. On a shared install, a note written for a client other than the
home tenant before that release still sits at the top level of the vault, so that client cannot read it through
Knowledge. Use **`POST /api/obsidian/relocate`** to move such files into `tenants/<tenant-id>/`. It is for the home
tenant's owner, signed in with a real account (the static API key is refused).

- **Dry run first.** The default (`apply=false`) changes nothing and returns `moved` (what would move), `skipped` (with a
  reason) and `conflicts`.
- **Only what the database proves.** A file moves when every note or document record that names its path carries the same,
  single, non-home tenant, and nothing exists at the destination. Files of the home tenant, files claimed by more than one
  tenant, records with no tenant, missing files, symbolic links and paths that leave the vault stay where they are and are
  listed. Nothing is overwritten.
- **Apply.** `apply=true` moves the files and updates the stored paths in the same step (a failed update puts the file
  back), then writes an audit entry (`vault.relocated`) with your account as the actor.
- **Batches.** Each call looks at up to `limit` files (default 100, at most 500). Repeat with `after=<next_after>` until
  `next_after` is null. Running it again moves nothing that already moved.

### Creating and scoping a venture

A **venture** is a product or business ErezAgent builds and operates for you -- it has its own
wallet, budget, P&L, connected accounts (optionally scoped independently of your other ventures),
and its own lifecycle stage (idea -> validated -> active -> scaling -> complete). Create one from
the **Ventures** screen (Revenue & CRM group) or through Launch Center.

**Locale and market overrides:** each venture can override `currency_override`, `locale_override`,
and `market_override` independently of your company-wide defaults (set in `company_locale_config`,
configurable from Platform Management -> Localization). `NULL` means inherit the company default.
A venture's market also gates market-specific skill packs -- for example, the 42 Israel-market
skills only fire for a venture (or a whole deployment) configured to the `il` market profile.

**Scoping the dashboard to one venture:** the sidebar's Venture Switcher lets you focus every
venture-aware screen (Command Center's mission list, the 3D Brain, and more) on one venture at a
time; the selection persists across navigation and reload. Selecting "All Ventures" shows
everything. You can also enter this scope from a venture's card on the Ventures page via **Drill
In**.

**Per-venture connector credentials:** payment providers, CRM connectors, and most OAuth
connectors resolve credentials per venture -- a venture-scoped credential wins if one is
configured, falling back to your owner-level connection, falling back to `.env` settings. Configure
one from the Connect dialog's **Scope** dropdown on the Connectors screen (works for both API-key
connectors and OAuth ones, including Gmail/Slack/Calendar/Drive).

**Self-service custom domains (since v4.42.0):** a venture can claim its own custom domain for its
booking or support page without you touching the server. From the venture's Support workspace,
open **Self-Service Domains**, enter a hostname, and pick a purpose (Booking, Support, or Both).
Once the venture (or you, on their behalf) points one DNS A record at this server's public IP
(`PUBLIC_SERVER_IP` must be set in `.env`) and adds the ownership record described below, the domain
activates automatically with a real Let's Encrypt certificate -- no restart, no manual Caddy edit.
Status (`pending_dns` -> `active`, or `failed`) is re-verified every 30 minutes and shown on the System
Health page. Claiming and deleting a domain are owner-only (since v4.87.0).

**Ownership record (since v4.91.0):** pointing a name at this server shows where it points, not who
controls it, so each claim also needs one DNS TXT record. After you claim a domain, its card shows the
exact record, each part with a copy button:

| Field | Value |
|---|---|
| Type | `TXT` |
| Name | `_erezagent-challenge.<hostname>` (for `book.example.com`: `_erezagent-challenge.book.example.com`) |
| Value | `erezagent-verify=<token>` (shown on the card; unique to this claim) |

Add it at the DNS provider next to the A record, then press **Verify now**. The card shows each half
separately (DNS points here / ownership proven) and says in plain words what is still missing. DNS changes
can take a few minutes to appear; ErezAgent also checks pending claims about every 30 minutes. Once the
record is verified the domain is ready and its certificate is issued on the first visit to it (a visit before
that gets no certificate). Details:

- Only an owner sees the record value. It stays valid for seven days; **Regenerate token** issues a new one
  and the previous one stops working. A claim that is not proven within seven days shows "Ownership record
  expired" (regenerate to continue) and is removed a day later, which frees the name.
- The first organisation to prove a name owns it. Several can hold an unproven claim on the same name (each with
  its own record), so a claim made by someone who does not control the DNS never blocks the real owner. When one
  claim is proven the others are removed. A name another organisation already owns cannot be claimed, and an
  organisation holds one claim per name.
- The record can stay in DNS or be removed after the domain is proven. Pointing is still re-checked every
  30 minutes.
- **Domains that were active before v4.91.0 keep working** with no change and no downtime. They are shown as
  "Verified before ownership proof existed"; press **Prove ownership** on one to get its record, add it, and
  press **Verify now** to upgrade it. Until then it is not accepted as the target of a Support workspace
  custom-domain DNS update (that update needs a proven domain).
- The API container needs ordinary outbound DNS (the same resolver it already uses for the A-record check).
  Names that are an IP address, a single label, or end in `.local`, `.internal`, `.localhost`, `.lan`,
  `.invalid`, `.onion`, `.arpa`, `.svc`, `.cluster`, `.corp`, `.home`, `.private` or `.intranet` cannot be claimed.

---

## 3. Company Foundation & Constitution

Company Foundation is the identity layer the Executive Board reads before every deliberation. It
has two parts, stored per tenant:

1. **Company Constitution** -- your invariant identity: who you are, what you value, what you will
   never compromise. Changes rarely, only on a genuine strategic pivot.
2. **Active Mission** -- your current goal: what you're trying to achieve right now, measured by a
   specific metric, with a deadline. Changes quarterly or when achieved/pivoted.

The Constitution and Mission are injected as the very first context every executive sees in a
board deliberation -- board recommendations are grounded in your organization's real identity, not
generic business advice.

### Setting it up

**During installation:** the installer's Company Foundation step walks you through the minimum
required fields (Company Name, Company Identity, Mission Headline) right after the health check
passes.

**From the app:** Company Foundation (Strategy & Governance group) -> Constitution tab -> **Create
Constitution** -> fill in your details -> **Create & Activate**. Then switch to the Active Mission
tab -> **Set Current Mission**.

### Constitution fields and their board impact

| Field | Board impact |
|---|---|
| Company Identity | Appears first in every deliberation |
| Long-Term Vision | Prevents short-term recommendations that undermine your long game |
| Target Market | CMO and CRO frame their positions around this |
| Value Proposition | CTO and CPO frame their positions around this |
| Business Model | CFO uses this to assess financial viability |
| Competitive Moat | Used in strategic assessments |
| Core Values | Board checks alignment against these |
| Non-Negotiables | Board challenges any recommendation that would cross these |

### Mission fields

Headline, Context, Primary Metric, Target Value, Current Value, Deadline, Key Results (3-5
observable success signals), and Constraints (budget/headcount/technical limits the board must
respect). A mission moves through `draft -> active -> achieved / pivoted / abandoned`; only one
mission is active per tenant at a time.

### Amending

Amending the Constitution creates a new version and supersedes the previous one -- full history is
preserved and auditable from Constitution -> History. Use the **Board Preview** tab to see exactly
what the Executive Board reads at the start of every deliberation, so you can verify formatting
before convening the board on something that matters.

### Chat-to-Company-Mission Bridge (v4.63.0)

Every Company Foundation write action -- creating, editing, activating, amending, or pivoting a
Constitution or Mission -- now requires the `owner` role, not just any authenticated account.
This closes a real pre-existing gap where any operator-level account could change these
identity-defining records.

A new **Set as Company Mission** button in Executive Office chat lets an owner attach a strategy
document and extract a structured Mission draft from it (headline, primary metric, target,
deadline, key results, constraints) via `POST /api/company-foundation/mission/extract-from-document`.
It always creates `status='draft'` and never activates automatically or touches the current
active Mission -- review it in Company Foundation's Mission tab (drafts are now visible there
even before any mission has ever been active) and either **Edit** its fields or **Activate** it
directly, same explicit review step as every other draft in this system.

### Multi-business support

Each tenant gets its own, fully isolated Constitution and Mission, enforced by RLS plus a partial
unique index allowing only one active record per tenant. If you're running ErezAgent for multiple
businesses that need complete data isolation from each other, each is a separate tenant with its
own Company Foundation, created from Company Foundation -> Create once you're scoped to that
tenant.

### Venture Foundation (v4.64.0) -- multiple businesses under ONE tenant

The more common case: one owner, one tenant, running several ventures (Ventures page) that
don't need separate accounts but DO each have their own identity -- a different mission, target
market, and business model per venture. Each venture can have its own **Venture Foundation**,
opened from a **Foundation** button on that venture's card, injected into board deliberations,
individual agent actions, and customer-facing communications for that venture alongside (never
instead of) your Company Foundation. A venture's Foundation is often created automatically: when
the Executive Board researches and approves a path that results in a governed `venture.create`
action, a draft Foundation is derived from that same approved recommendation for you to review --
it never activates on its own. See `docs/70_VENTURE_FOUNDATION.md`.

---

## 4. Persona & Workforce Identity Administration

*(since v4.44.0; self-service admin UI new in v4.46.0)*

### What a Persona is -- and the one thing it can never be

A Persona is a name, avatar, tone, voice, and language -- for ErezAgent itself (**System**), for
one of your internal agents (**Internal Agent**), or for a customer-facing AI representative
(**External Representative**, scoped to a venture and optionally a specific customer or project).

**A Persona is identity and presentation only.** Assigning or changing one never changes what an
agent is allowed to do, which skills it can invoke, which connectors it can reach, or how much you
trust its decisions -- every authority decision still comes from Agent Policies, skill permission
grants, and trust scores, completely independent of identity. This separation is enforced at the
code level (the permission-gating logic in the skill and agent execution paths never reads a
Persona table) and verified by a dedicated automated test on every build, so it holds as a
structural guarantee, not just a convention.

### Where you configure identities

**Persona Management** (Knowledge & Learning group, new in v4.46.0) is the self-service home for
every identity in your organization. Personas are grouped by scope -- System, Internal Agents,
External Representatives -- and each card expands in place into a full edit form: display name,
title, short intro, avatar, primary language and additional languages, tone, formality,
communication style, technical depth, greeting/closing style, and approved/prohibited terminology.
A **"Configure My Whole Team"** button proposes an identity for every agent that doesn't have one
yet, all in one batch.

You can also configure identities conversationally, anywhere you already talk to ErezAgent
(command bar, Executive Office chat, Agent Chat): *"give my Security Agent a professional
identity,"* *"create a Hebrew and English support representative for Venture X,"* or *"create
identities for my entire internal workforce."*

### Governance: every change is a real, approved proposal

Whether you edit a Persona from the self-service page or ask for one in chat, the request goes
through the same governance path as any other high-risk change: a real `Action` with
`requires_approval=True` and a matching `Approval` record. Submitting an edit from Persona
Management shows a **Pending Approval** state right on the card, with inline Approve/Reject buttons
calling the same endpoint the Approvals page uses -- a self-service edit never has to leave the
page to be approved.

- **Approving is owner-only.** `POST /api/approvals/{id}/approve` independently enforces the
  `owner` role. An operator can propose a Persona change or a new System/External Representative
  Persona for review; only an owner can approve it.
- **Drafting an identity for a specific agent, or for your whole workforce at once, is owner-only at
  the endpoint itself.** Because that draft row is inert (invisible anywhere until it separately
  goes through its own `persona.apply` approval), there's no separate approval gating the draft step
  -- so the endpoint requires the owner role directly instead.
- **The bulk "Configure My Whole Team" proposal never overwrites a Persona you've already
  customized**, and approving the batch only creates drafts -- each individual identity still needs
  its own separate approval before it goes live anywhere.

Once approved, a Persona change is live immediately, everywhere that identity is shown -- Agent
Network, Command Center's 3D/2D Brain, global chat, the entity inspector, and (for an External
Representative) your Support and Booking Portals -- with no page refresh required.

### Cross-tenant safety on External Representative Personas

Creating an External Representative Persona lets you scope it to a real venture, customer, or
project. Before writing that record, the system independently verifies that the venture, customer,
and project you referenced actually belong to your own tenant -- checked explicitly at write time,
as a deliberate layer on top of Row-Level Security's own read-time scoping (RLS governs what a
query can see; this check governs what values a write is allowed to reference, which is a distinct
guarantee). In practice: you can never create a representative identity that points at another
tenant's venture, customer, or project, even by supplying its ID directly.

### Avatars

Upload your own image or have one generated from a short prompt. Every upload is validated
server-side (size-capped before decode, format-verified, dimension-capped, and forcibly re-encoded
to PNG) before it's ever stored -- an invalid or malicious file is rejected outright. Either way, a
new avatar sits pending until the same approval that applies the rest of the identity change is
granted; nothing is shown anywhere until then.

### Where identities show up

| Surface | What you see |
|---|---|
| Agent Network | An agent's real avatar next to its department icon (or in place of initials in a team roster); a Persona header in the inspector panel |
| Command Center's 3D/2D Brain | The agent's real name and avatar on its sphere/node |
| Executive Board (Objective Center) | An executive's avatar next to their role badge |
| Global chat / Agent Chat / command bar | The real agent's identity instead of a generic bot icon, plus a "consulted: X" note only when a real delegation happened that turn |
| Support and Booking Portals | A configured External Representative's name and avatar instead of the generic "Support" label -- falls back to the generic identity when none is configured |

### Hebrew / RTL

Text renders right-to-left automatically wherever a Persona's own configured language is Hebrew,
Arabic, Persian, or Urdu -- scoped to chat message text, the Support/Booking Portal conversation
text, and system attention signal cards. The rest of the application (navigation, Settings, layout
chrome) is unaffected.

### Voice

Each Persona can be assigned its own voice, chosen from the voice profiles you've already set up in
Settings -> Voice Profiles (see section 1's MFA note for the general Settings location; Voice
Profiles is covered under Connectors & Integrations below). A Persona with no assigned voice falls
back to your tenant's default voice profile.

---

## 5. Agent Policies & Autonomy

### Per-agent policy

**Agent Policies** (Strategy & Governance group) is where you define what each agent can and
cannot do autonomously: which connectors it can access, what risk tier it can self-approve up to,
and what topics are out of scope. Set this up during onboarding for any new agent, and revisit it
whenever you change how much you trust an agent's judgment.

The **Send Emails**, **Use Internet**, and **Touch Customers** capability toggles on this page are
enforced at the moment an agent actually tries to invoke a skill (since v4.49.0), not just recorded
as configuration -- turning one off genuinely blocks the matching class of skill invocation instead
of only describing an intended restriction. Since v4.92.1 the **Create Memories**, **Execute Runbooks** and
**Create Missions** toggles and the allowed and denied connector lists are enforced the same way, and an approval mode of
**Always** sends the skills a policy governs through approval. Since v4.93.0 **Read CRM**, **Create Deals**, **Send WhatsApp**, **Spawn Sub-Agents** and **Modify Agents** are enforced too. They start ticked on a new policy, so you untick what an agent must not do, and a policy saved earlier was set to ticked, so nothing changes for an agent until you untick one. **Modify Budgets** is retired and no longer shown: no skill changes a budget, and budget changes go through approved change proposals. **Read CRM** covers every skill that reads deals, customers or the CRM pipeline; **Spawn Sub-Agents** covers an agent's own decision to hand work to another agent, not the delivery and support workflows or a delegation you create yourself. Saving a policy with only some fields changes only those fields. Each workspace keeps its own
policy for a shared agent, and a shared agent used with no workspace in scope is refused.

Agent Policies lists the agents whose policy has all five of Read CRM, Create Deals, Send WhatsApp, Spawn Sub-Agents and
Modify Agents switched off, with **Allow all five** per agent and a confirmed **Allow all**. If you did not choose that
setting, allow them; the change is recorded in the audit log per agent. The page waits for the policy list to load before
an agent can be selected or a policy saved.

When an agent reaches its daily token budget, the message names the agent, the tokens used against the budget and where the budget is raised: an owner opens **Agents**, selects the agent and chooses **Edit settings**.

### Risk tiers and the approval gate

Every governed action carries a risk level. The rule that never bends: **any action rated Medium
risk or above requires your explicit approval** before it executes. Low-risk actions can be
configured to auto-approve. Approvals appear in **Action Center** and **Approvals**, and (if
Telegram is configured) as an inline-button notification on your phone.

**Autonomy tiers (T0-T3)** govern execution at a finer grain: T0 actions run immediately, T1/T2
actions create an approval and pause until you act, and T3 actions are refused outright and route
to whatever fallback policy is configured. Five global policies (covering contract actions,
payment actions, code deploy, code commit, and content delete) are seeded by default and should
not be removed -- they exist specifically to keep the highest-risk action categories from ever
running autonomously. Review or adjust them from **Autonomy Policy Console** (Command group) --
monitoring is by action, not covered here in duplicate; adjust policy from Agent Policies /
Autonomous Rules instead.

### Auto-approve policies are owner-only, and can't waive critical-risk review

Creating, editing, or deleting an auto-approve rule (**Autonomy Policy Console**) requires the
owner role -- an operator-level account can propose changes elsewhere in the system, but cannot
touch what auto-approves itself. A policy also can't be set to auto-approve Critical-risk actions
at all; that tier always reaches you as a real approval, regardless of policy configuration.

### Emergency kill switch

**Settings > Feature Flags** has a `system_kill_switch` toggle, off by default and owner-only.
Turning it on immediately treats every mission and pending action across the whole system as
paused -- new step dispatch stops, and any action already approved but not yet run stays deferred
until you turn it back off. This includes the two scheduled dispatch jobs (approved outreach
email, approved cloud actions) that otherwise run on their own five-minute schedule, independent
of a specific mission (v4.48.0). Work already past its own checkpoint at the moment you flip it
isn't force-killed mid-step; the effect is on what happens *next*, not an interrupt of what's
already running. Turn it off the same way to resume everything that was deferred.

### Pausing a mission actually stops it

Pausing a mission (**Mission Control**) now genuinely halts its execution -- both new step
dispatch and any action tied to that mission that was already approved but not yet run. Resuming
re-drives everything that was deferred while it was paused. If the kill switch above is on,
resuming one paused mission does not bypass it; that mission's deferred actions stay deferred
until the kill switch itself is turned off.

### Autonomous Rules

**Autonomous Rules** (Strategy & Governance group) is the rulebook for reactive automation --
"if the customer health score drops below 40, trigger a check-in email," evaluated continuously
against live data. Rules can operate at any risk tier; a rule that would trigger a Medium-or-above
action still creates a real approval rather than executing silently.

### Autonomy Score

**Autonomy Score** (Strategy & Governance group) is a single composite score (0-100) showing what
fraction of your organization's decisions are made and executed without your manual intervention,
broken out by domain, with trend over time -- a quick read on whether agents are operating
effectively or constantly waiting on you.

### Execution substrate

Under the hood, every governed action runs through a typed action registry
(`GET /api/execution/registry`) and creates an `execution_run` record with a full audit trail. A
gated (T1/T2) run creates a real, linked approval; approving it resumes execution automatically. To
verify the registry or policies directly:
```bash
curl -H "Authorization: Bearer $EREZAGENT_API_KEY" http://localhost:8000/api/execution/policies
curl -H "Authorization: Bearer $EREZAGENT_API_KEY" http://localhost:8000/api/execution/stats
```

---

## 6. Departments & Cost Centers

**Custom Departments** (Governance group; formerly labeled "Departments" -- renamed in v4.46.0 to
distinguish it from the fixed workforce department pages like Engineering Department, elsewhere in
the sidebar) is your dynamic cost-center hierarchy: a real place to group agents, track trailing
30-day AI spend against a monthly budget, and set an alert threshold.

### Creating and managing a department

Create a department with a name, cost-center code, monthly budget, and optional parent department
(departments can nest). Each card shows agent count, 30-day spend, and a budget utilization bar
(green/amber/red based on how close spend is to budget and your alert threshold).

**Full admin controls (since v4.46.0):** expand any department card to rename it, adjust its budget
and alert threshold, delete it (with confirmation), and manage which agents belong to it -- including
setting a real **department head** with a star toggle. The head shown here is the same one that
drives the crown badge on Agent Network.

A name-collision guard stops you from creating a cost center named after one of the fixed workforce
department pages (Engineering, Sales, Growth, Product, Finance) and points you to the right place
instead.

### Budget status

| Status | Meaning |
|---|---|
| `ok` | Spend below your alert threshold |
| `warning` | Spend at or above the threshold but under 100% of budget |
| `over_budget` | Spend exceeds the monthly budget |
| `no_budget` | No monthly budget set |

### Chargeback reporting

**AI Spend Intelligence** (Cost & Finance group) includes a per-department chargeback view --
gross spend, savings attributed, net cost, and each department's share of total spend, filterable
by period (7d/30d/90d/12m/all).

---

## 7. Connectors & Integrations

**Connectors** (Platform group) is where you connect and manage every external account ErezAgent
can use -- email, calendar, CRM, payments, and social platforms among them. Every connector card
shows real connection status, what it's actually authorized to do, and a manual sync/reconnect
option. Credentials are encrypted at rest with Fernet (`CREDENTIAL_ENCRYPTION_KEY`) -- never stored
in plaintext.

### Setting up the most common connectors

**Telegram** (approval notifications with inline Approve/Reject buttons):
1. Message **@BotFather** on Telegram -> `/newbot` -> copy the bot token.
2. Message **@userinfobot** to get your Chat ID.
3. Connectors -> Telegram -> **CONNECT** -> enter the token and Chat ID.
4. For the inline Approve/Reject buttons, set `TELEGRAM_OWNER_CHAT_ID` (your Telegram user ID from step 2) and
   `TELEGRAM_WEBHOOK_SECRET` (a random string you choose) in `.env`, restart the API, then register the webhook with
   Telegram, passing the same value as `secret_token`:
   `curl -X POST "https://api.telegram.org/bot<TOKEN>/setWebhook" -d "url=https://your-domain/api/connectors/telegram/callback" -d "secret_token=<TELEGRAM_WEBHOOK_SECRET>"`.
   Without the secret the callback is refused, and without the owner ID button presses are ignored; the readiness
   check shows what is missing. Approvals that need a named approver (chain steps, role-gated steps) and refunds are
   decided in the web app.

The bot also accepts `/remember <text>`, `#memory`/`#save` hashtags, and forwarded messages as
memory captures.

**Gmail (OAuth2):**
1. Google Cloud Console -> create a project -> enable the Gmail API -> create OAuth2 credentials
   (Web Application).
2. Add redirect URI `https://your-domain/api/connectors/oauth/callback` (or the localhost
   equivalent for local development).
3. Set `GMAIL_CLIENT_ID` / `GMAIL_CLIENT_SECRET` in `.env`, restart the API.
4. Connectors -> Gmail -> **CONNECT** -> authorize in the browser.

**GitHub:** create a fine-grained personal access token with `Contents` (read), `Issues`
(read/write), and `Pull requests` (read/write) scoped to the repos you want -- then Connectors ->
GitHub -> **CONNECT**.

**Obsidian Local REST API** (see section on Knowledge integration below for vault structure):
install the Local REST API community plugin in Obsidian, copy its auto-generated API key, set
`OBSIDIAN_API_URL=http://host.docker.internal:27123` and `OBSIDIAN_API_KEY` in `.env`, then connect
from the Connectors screen with the same key. Obsidian must stay open on the same machine as Docker
for the integration to work -- it does not work against a headless cloud server.

**Stripe:** create a webhook endpoint at `https://your-domain/api/connectors/stripe/webhook` for
`payment_intent.succeeded`, `invoice.payment_succeeded`, and `customer.subscription.created`; set
`STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET`, `STRIPE_ENABLED=true`; connect from the Connectors
screen. For plan billing and Revenue Center deal payments, enter the gateway keys in Billing Setup instead
(section 11); keys already in `.env` keep working as a fallback.

**WhatsApp** (v4.78.0, real inbound document intake for Customer Document Collection): in Meta's
App Dashboard, create a WhatsApp Business app, note its **phone_number_id** and generate a
permanent (system user) access token. Connectors -> WhatsApp -> **CONNECT** -> paste the access
token and phone_number_id (venture-scoped if this venture has its own WhatsApp Business number, or
account-wide otherwise). Then, to receive inbound messages/documents, register the webhook
`https://your-domain/api/webhooks/whatsapp` in the same Meta app, subscribed to the `messages`
field. Set `WHATSAPP_VERIFY_TOKEN` (any string you choose, must match what you enter in Meta's
webhook-verification dialog) and `WHATSAPP_APP_SECRET` (from the app's Basic Settings -- used to
verify every inbound webhook's signature; an unconfigured secret fails closed and rejects every
inbound message, it never silently trusts an unsigned one) in `.env`, then restart the API.

**DocuSign** (v4.82.0, e-signature on Business Records): in DocuSign's Developer Console, create an
integration app for OAuth 2.0 and note its Client ID/Secret. Connectors -> DocuSign -> **CONNECT**
-> authorize in the browser (defaults to the demo/sandbox environment -- set `DOCUSIGN_ENV=production`
deliberately once you're ready to send real, billable, legally-binding envelopes). Then, to receive
real-time signature status (delivered, signed, declined), create a Connect subscription in your
DocuSign account for the `Envelope Sent`/`Envelope Completed`/`Envelope Declined`/`Envelope Voided`
events, pointing at `https://your-domain/api/connectors/docusign/webhook`, and set `DOCUSIGN_
WEBHOOK_SECRET` in `.env` to the same HMAC key you configure on that Connect subscription -- an
unconfigured secret disables the webhook endpoint entirely rather than accepting unsigned events.
Every signature request still requires owner approval before it's actually sent, same as
every other governed connector write.

**All other API-key connectors** (Slack, LinkedIn, Instagram, TikTok, YouTube, HubSpot,
AWS, Zoom, Perplexity, Airtable, Linear, Twitter/X, WordPress, Outlook, Google Calendar, Google
Drive, Facebook, and more): obtain the key/token from the provider's developer portal, then
Connectors -> find the connector -> **CONNECT** -> paste it in. The connect dialog shows the exact
field name and a hint for each one.

### Social connectors read as well as post

Facebook, Instagram, Twitter/X, TikTok, YouTube, and LinkedIn go beyond publishing: once connected,
ErezAgent reads inbound comments and mentions automatically, classifies them, and routes real
leads, support questions, and product feedback to the right place -- no separate setup step.
Facebook, Instagram, and Twitter/X also support direct messages (an already-connected account
needs a one-time reconnect to pick up the new permission scope). Every drafted reply still goes
through Approvals before anything posts.

### Google Calendar push and YouTube status (v4.87.0)

Google Calendar change notifications are registered automatically for installs with a public `https` `API_BASE_URL`;
each notification must carry its per-channel token, and channels created before this check are renewed once when the
worker starts (the replaced channel is stopped). If a renewal fails, the 15-minute polling sync keeps appointments up to
date. A `403` from YouTube is now classified (quota exceeded, API not enabled, insufficient scope, or forbidden) and shown
in the connector and channel status so you know what to fix.

### Voice Profiles

Settings -> Voice Profiles lets you clone your own voice from a short recording, or pick a curated
preset voice, for ErezAgent to speak with. TTS/STT providers supported: ElevenLabs, OpenAI, and
Voicebox (a free, open-source, local voice server you run yourself -- set `TTS_PROVIDER=voicebox` /
`STT_PROVIDER=voicebox` and point `VOICEBOX_API_URL` at your own instance). One profile is active
at a time system-wide; a Persona (section 4) can also be assigned its own distinct voice from the
same set of profiles.

### Automation (n8n)

Connect an n8n instance (Connectors -> n8n) to trigger automation workflows from agent decisions.
Requires `N8N_BASE_URL`; `N8N_WEBHOOK_SECRET` is optional and, when set, signs the outbound trigger
call and verifies the inbound completion callback for faster status detection than the default
10-minute poll.

Webhook triggers and every call of a custom (configurable) connector or MCP server go to the address that was
checked when the request is sent, and a redirect answer is returned as a failure instead of being followed.

---

## 8. Backups & Disaster Recovery

### Backup Center

**Backup Center** (Governance group) is full backup management:

- **Overview** -- last backup status, total jobs, storage used, failed count.
- **Jobs** -- every backup job with status; a completed/verified job has a **RESTORE** button with
  a dry-run option that validates the archive without writing any data, and a **DELETE** button (owner only).
  Deleting a completed or verified backup asks for confirmation, deletes its archive file as well, and cannot be
  undone; the backup can no longer be restored. A failed or pending job is removed without a prompt.
- **Repositories** -- local path or S3 bucket destinations.
- **Schedule** -- preset cron expressions (daily/6h/weekly/monthly) or a custom one.

Settings -> Backup & Restore gives the same core actions (last 10 jobs, inline restore,
**Backup Now**) without leaving Settings.

**Retention.** A scheduled backup is deleted, archive file included, once it is older than its schedule's
retention days. A completed backup with no schedule (from **Backup Now**, or from a schedule you deleted) is kept
while it is among the newest 10 or younger than 30 days, and deleted by the daily prune after that
(`BACKUP_MANUAL_KEEP_MIN` and `BACKUP_MANUAL_KEEP_DAYS`; `0` for both keeps them until you delete them). The newest
completed backup is never deleted automatically. Backup Center shows the rule and how many backups the next run
will delete. Before v4.91.2, backups with no schedule were never deleted.

Backups of a large database run for up to an hour before they are stopped. A scheduled backup that fails is retried
after 30 minutes, and Grafana alerts when the last backup failed or none has completed in 36 hours.

### What's covered

A full backup covers the database, the Obsidian vault, configuration, and every real binary asset
volume your deployment uses -- including `creative-assets`, `generated-documents`, and
`persona-assets`. Each is archived under its own real volume name, with a combined archive and a
SHA-256 checksum, and can be pushed to S3 and/or an NFS share (`BACKUP_NFS_*` settings) in addition
to local storage.

### Triggering and restoring manually

```bash
# Trigger a full backup
curl -X POST http://localhost:8000/api/backups/trigger \
  -H "Authorization: Bearer $EREZAGENT_API_KEY" -H "Content-Type: application/json" \
  -d '{"backup_type": "full"}'

# Dry-run restore first
curl -X POST http://localhost:8000/api/backups/<job_id>/restore \
  -H "Authorization: Bearer $EREZAGENT_API_KEY" -H "Content-Type: application/json" \
  -d '{"dry_run": true}'

# Then restore for real
curl -X POST http://localhost:8000/api/backups/<job_id>/restore \
  -H "Authorization: Bearer $EREZAGENT_API_KEY" -H "Content-Type: application/json" \
  -d '{"dry_run": false}'
```

A `db`/`full` restore replaces the live database outright via a direct `psql` replay against the
running instance -- disruptive and potentially long-running on its own (minutes, against a real
database), independent of whether any data would actually be lost. So both the dry-run and the real
restore response include a `reconciliation` block: row counts, per key table (tenants/missions/
actions/approvals/customers), created since the backup was taken -- exactly what a restore right now
would permanently erase. The real restore (`dry_run: false`) always requires `"force": true` -- even
when `reconciliation` shows nothing at risk -- or it's refused with `needs_confirmation: true` and
does **not** touch the database. Review `reconciliation` first, then resubmit with `force`.

Since v4.92.0 the replay runs as one transaction that stops at the first error, replaces what is in the database, and checks the archive's checksum first. A restore that fails changes nothing and reports `restored: false` with the reason. A backup without a stored checksum needs `"allow_unverified": true`. A backup taken before v4.92.0 restores onto an empty database; to replace a database that already has data, restore a new backup.

### Disaster-recovery secrets escrow

`LICENSE_PRIVATE_KEY` and `CREDENTIAL_ENCRYPTION_KEY` cannot be regenerated, and are deliberately
never bundled into the backup archive -- an unencrypted archive is the wrong place for live
secrets. Retrieve them once and store them in a password manager, off this server. In Backup Center choose
the secrets card and enter your own password (the static API key cannot do this). Each reveal is
recorded in the audit log (who, tenant, source address) and works once; `GET
/api/backups/dr-secrets-escrow` returns only its state (`confirmed_at`, `revealed_at`,
`reveal_available`). Via the API:
```bash
curl -X POST -H "Authorization: Bearer $OWNER_SESSION_TOKEN" -H "Content-Type: application/json"   -d '{"password":"..."}' http://localhost:8000/api/backups/dr-secrets-escrow/reveal
```
Returns `license_private_key`, `credential_encryption_key`, `jwt_secret_key`, and a `confirmed_at`
timestamp (`null` until you confirm). A second reveal is refused until you re-arm the escrow
(`POST /api/backups/dr-secrets-escrow/rearm`, password again); rotating an escrowed secret re-arms
it automatically because the saved copy is then out of date. On a shared (multi-client) install
only the home tenant's owner can use Backup Center and the escrow. Do this before you need it -- losing the server without these
saved elsewhere makes every stored connector credential permanently undecryptable and blocks
issuing any new customer license. Once you've genuinely saved them somewhere safe, confirm it:
```bash
curl -X POST -H "Authorization: Bearer $EREZAGENT_API_KEY" http://localhost:8000/api/backups/dr-secrets-escrow/confirm
```
This is tracked as a real readiness item (**Command Center**'s readiness banner) until confirmed --
re-confirming later just updates the timestamp, so it's safe to run again after a real key rotation.

### Verify and rehearse

Verify weekly that scheduled backups are actually completing (Backup Center -> Overview). Test a
real restore quarterly, using the dry-run option first against a non-production environment when
possible.

---

## 9. Security Policy

### Tenant isolation as a standing guarantee

Every tenant-scoped table is protected by PostgreSQL Row-Level Security, enforced through the
dedicated, non-superuser `erezagent_app` database role (see section 2) -- a real database-level
boundary, not an application-layer filter that depends on every code path applying it. A
permanent automated CI check fails the build if a new table is ever added without tenant scoping,
so this guarantee holds going forward as the schema grows. The official installers always
provision and connect as this role; a manually assembled deployment that skips it would fall back
to the database's superuser connection instead, which is not subject to Row-Level Security -- this
is checked and surfaced as a readiness item (**Command Center**'s readiness banner) so it's never
silent.

### Security Assurance Division

**Security Assurance** (Research & Signals and Governance groups) runs continuous, multi-scanner
security review:

- **Bandit** -- Python static analysis
- **pip-audit** -- Python dependency CVE scan
- **npm-audit** -- frontend dependency CVE scan
- **detect-secrets** -- secret-leak scanning
- **Dockerfile static analysis**
- **OWASP passive pentest checks** against explicitly authorized targets only (the same allowlist
  discipline used for every adversarial test category -- nothing is ever probed just because it
  exists). A run that reaches its target marks earlier open findings it no longer reports as resolved (findings you
  acknowledged, dismissed or granted an exception for are left alone); a run that cannot reach the target resolves nothing.

**Gate rules** (used as a CI/deploy gate):

| Condition | Gate |
|---|---|
| Any critical finding open | `blocked` |
| 3+ high findings open | `blocked` |
| 1-2 high findings open | `warning` |
| Otherwise | `pass` |

The **Settings > Security** toggles for blocking on critical CVEs and on secret leaks (both on by default) enforce this at the
action level: a critical CVE blocks `deploy`/`commit`; a detected secret leak blocks `publish`/
`push`. Governed AI-generated repair proposals can be requested per finding and require your
approval before anything is applied.

### Security Passport (Cyber Defense & Secure Delivery, since v4.40.0)

Every protected asset -- ErezAgent itself, and every customer or venture product you deliver --
carries a real Security Passport: one honest status covering threat modeling, code/dependency/
secret/container scanning, software bill of materials, release signing, adversarial tenant-
isolation testing, open findings, and runtime monitoring. "Never scanned" and "clean" are always
shown as distinct states. Browse every asset's passport from Security Assurance's **Portfolio**
tab.

**Security exceptions** are real, audited, time-limited grants -- a reason, a severity, and a
mandatory expiry, owner-role only:
```bash
curl -X POST http://localhost:8000/api/security-assurance/exceptions \
  -H "Authorization: Bearer $EREZAGENT_API_KEY" -H "Content-Type: application/json" \
  -d '{"protected_asset_id": "UUID", "finding_id": "UUID", "reason": "Vendor patch due next sprint",
       "severity": "medium", "expires_at": "2026-10-01T00:00:00Z"}'
```
A background job expires exceptions automatically once `expires_at` passes -- there is no
indefinite, silent dismissal of a finding; an unrenewed exception reverts the finding to counting
against the asset's Security Passport status.

**Containment actions always require approval.** Revoking a credential, disabling a user, rotating
a secret, disabling/re-enabling a route, or rolling back a deployment are all real, governed
actions -- an incident can propose one, but every one of them becomes a real Approval before
anything executes.

### Real gates on AI-generated customer code

Every real project build automatically triggers a project-scoped Bandit/pip-audit/npm-audit/
secret/Dockerfile scan against the customer's own generated code before a human is ever asked to
accept or publish it. Formal acceptance and external publish both include a real security
certification criterion -- a blocked certification prevents acceptance or publish unless an
explicit, owner-recorded waiver is given, with the real certification status always shown before
anyone approves. Secure-coding standards (no hardcoded secrets, no SQL injection, no insecure
defaults) are injected directly into every code-generation prompt used to build customer projects.
A drafted skill's prompt template goes through an LLM-based safety review (prompt-injection
language, out-of-category tool access, secret exfiltration, instructions that skip approvals) before
its proposal can even be created.

### Credential encryption and .env hardening

All connector credentials, signing keystores, and `mfa_secret` are Fernet-encrypted at rest with
`CREDENTIAL_ENCRYPTION_KEY`.

`.env` itself holds 30+ plaintext secrets. Every installer and upgrade path restricts its file
permissions automatically (three principals only: the installing account, Administrators/root, and
SYSTEM). To re-harden an already-deployed `.env` at any time, with zero downtime:
```bash
./installer/harden-env.sh /path/to/.env          # Linux / macOS / Git Bash
.\installer\harden-env.ps1 -Path C:\path\to\.env  # Windows PowerShell
```
Both are idempotent and print before/after permissions.

### Authentication hardening (since v4.49.0)

**Password hashing** uses Argon2id. Any account still on the older PBKDF2 hash upgrades
transparently to Argon2id the moment that user next logs in successfully -- nothing you need to
do, no forced reset.

**Session revocation** is now real: logging out, changing a password (self-service or an
owner-initiated reset), or an owner changing a user's role all bump that user's server-side
session version, which immediately rejects every token issued before that moment. Rotating
`JWT_SECRET_KEY` (see below) has the same effect for every session at once.

**Credential rotation tools** (Settings > System, owner-only, each writes an audit-log entry):

| Secret | What it does | Completion step |
|---|---|---|
| JWT signing secret | Generates a new `JWT_SECRET_KEY`, shown once | Set it in `.env`, restart `api`/`worker` -- every active session (including your own) is invalidated the moment you restart |
| Internal API key | Generates a new `EREZAGENT_API_KEY`, shown once | Set it in `.env`, restart -- any script/automation using the old key stops working immediately |
| Credential encryption key | Takes a new key value, backs up the database first, then re-encrypts every stored connector credential and MFA secret in one transaction (all-or-nothing) | Set the same value as `CREDENTIAL_ENCRYPTION_KEY` in `.env`, restart -- do this promptly, since the running process keeps encrypting new writes with the old key until you do |
| License signing key (founder instance only) | Generates a new Ed25519 keypair and registers it as the next signing-key version -- licenses already issued under the previous version keep verifying, nothing needs reissuing | Set the new private key as `LICENSE_PRIVATE_KEY` in `.env`, restart |

None of these endpoints ever store the newly generated secret anywhere -- it's returned once, to
the caller who triggered the rotation, and never logged or persisted. Treat the credential
encryption key rotation as a maintenance-window operation.

**Sandbox network egress**: `sandbox_network_mode` (sandbox-executor's own setting, per-deployment
via environment, not per-tenant) controls what network access an agent/skill-triggered shell, git,
or build command gets inside the isolated sandbox container. `default` (the default) gives normal
internet egress; `isolated` runs with no network access at all -- a real tradeoff, not a free
hardening switch, since `git clone`/`git push` and similar operations genuinely stop working in
that mode. Set it only for a deployment whose sandboxed workloads are pure local build/test with no
real network need.

### Sign-in throttling, trusted proxies and request limits

**Account lockout.** Repeated failed sign-ins for one username lock that username for a progressively
longer time (1 minute after 5 failures, 5 minutes after 8, 15 minutes after 10), regardless of which
address the attempts come from. The same response is returned for an unknown username, so it does not
reveal which accounts exist. A successful sign-in clears the count. Six-digit authenticator codes are
limited the same way per user (5 failures locks code entry for 5 minutes). If Redis is unavailable
the lockout is skipped and logged, and the per-address limits still apply.

Failed sign-ins for a username that does not exist are recorded without the username: only a short digest of it, the
source address and the time are kept (90 days), and the security detection check counts them like any other failed
sign-in, so a burst from one address or against one name raises the same alert.

**Rate limits** are keyed on the signed-in user (or the static API key) only when the credential
actually verifies; otherwise on the client address. Sign-in, MFA, password-reset, public portal, booking,
support, document and webhook routes are always limited per client address.

**Trusted proxies.** The API accepts `X-Forwarded-For` only from loopback and private-range peers
(`127.0.0.0/8`, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`) -- the Docker network and an in-cluster
ingress -- and uses the right-most address that is not one of those hops as the client address. A
reverse proxy or load balancer on a public address is not trusted by default, so it appears as the
client address in rate limits and audit rows.

**Static API key.** `EREZAGENT_API_KEY` acts as an owner only on installs with a single tenant (and on
health and system-status routes). Once a second tenant exists the key is rejected for tenant data;
use a signed-in user session instead. This includes the event stream's `?token=` option and a seed call to
Company Foundation that carries only the key. The weekly route coverage scan uses the key too, so on a shared install
it checks only the health and system routes, shows every other route as untested, and says so in its result.

**Startup checks.** With `APP_ENV` other than `development`, the API and worker refuse to start if
`DISABLE_AUTH=true`; with `APP_ENV=production` they also refuse to start when `APP_DATABASE_URL` is empty
(row-level security would not be enforced). The
installers populate `APP_DATABASE_URL`; on an older install run the upgrade script to provision it.
This applies to Kubernetes pods too: set `appDatabase.host` in the Helm values and the chart provisions the
role and `APP_DATABASE_URL` (see `KUBERNETES.md`, "Row-level security"), or add `APP_DATABASE_URL` to the
`erezagent-secrets` Secret before upgrading.

**Request size.** JSON and form requests larger than 10 MB are rejected with `413`; webhook endpoints accept 1 MB (4 MB for SendGrid, 25 MB for DocuSign); file uploads keep
their own per-route limits.

### Emergency containment

Block an API route instantly via Redis, no redeploy required (health/auth/metrics paths are always
exempt):
```bash
curl -X POST http://localhost:8000/api/security-assurance/containment \
  -H "Authorization: Bearer $EREZAGENT_API_KEY" -H "Content-Type: application/json" \
  -d '{"rule_type": "route_disable", "target": "/api/some-endpoint", "reason": "CVE mitigation"}'
```
Supports method-specific blocking, an incident link, and an auto-expiry timestamp. List, audit, or
lift a rule from the same `/api/security-assurance/containment*` family of endpoints.

---

### What changed for administrators in v4.91.0 (security and isolation)

- **Secret scan.** The scheduled scan (Security Assurance) checks a copy of the source files, finishes in a few minutes and shows
  as **failed**, never clean, when it cannot finish. Credentials found in notes inside the document vault now appear as findings:
  rotate them and remove them from the note. Only the platform owner's workspace sees the scan events.
- **Approvals.** A request for approval takes its risk, action type and agent from the action itself. Approvals are decided
  once and not after they expire, and batch approval records the approver and applies the same refund rules as a single
  approval. A project **Deploy** now waits in Approvals before the build starts. During a chat an agent decides an approval
  only when the person talking to it names that approval.
- **Prompts.** A change to a shared agent's system prompt applies to your workspace only, and an approved prompt change can be
  used once, for exactly the text that was approved.
- **Workflows.** A workflow run approval covers the definition as it was when the run was requested; edit it and the run needs a
  new approval. n8n completion callbacks must carry the run id (`run_id`) in the signed body. An n8n webhook target is a
  webhook path, a URL on your own n8n, or a public https address.
- **Accounts.** Creating a user asks you to confirm your own password (wrong confirmations count toward the sign-in lockout);
  the static API key cannot create users. Every workspace keeps at least one active owner, and an enrolled authenticator is
  replaced only after it has been disabled. Key rotation (Settings > Data & Privacy > Keys) now re-encrypts every stored secret and
  reports what it covered.
- **Install-wide settings.** Diagnostics, preflight and the support bundle, model routing overrides, revisions of shared
  platform skills, the licensing product and edition catalogue, Value Hunter scanner settings, and switching off the shared
  action catalog are changed by the instance administrator only (the owner, on a single-workspace install). A model routing
  override or a shared-skill revision must be requested while signed in as a person, is sent for approval, and changes only
  after the owner approves it in Self-Improvement; it is applied once, and not if the policy or skill changed in the meantime.
- **Connectors and hosts.** Settings that name a host are checked when saved and when used: provider addresses must belong to the
  provider; WordPress sites need https; SSH deploy targets and custom hosts must be public addresses. Connector status and sync
  messages show a status code or a short message, not the provider's response.
- **Project folders and sandbox.** Project folders are separate per workspace; a project name that another workspace already uses
  gets a distinct folder name. Each sandbox project runs under its own compose name.
- **Lead webhooks on a shared install.** Twilio, Meta Lead Ads and LinkedIn webhooks use
  `https://<your-domain>/api/webhooks/<provider>/<tenant-id>` (the Google Ads webhook already did). The bare URL works only on a
  single-workspace install. A WhatsApp number or a Facebook/Instagram page can be connected by one workspace only.
- **Payments.** Delayed payments (bank debits) count as received only once they settle; Stripe amounts use the currency's own
  decimal places; PayPal deals close on the completed capture; Pelecard confirmations must be approved and for the full price.
- **Startup checks.** In production the API and worker require secrets of at least 32 characters and refuse `LICENSE_MODE=dev`.
  The install and upgrade scripts (shell, PowerShell, Python) and the web installer correct these values in an existing `.env`: `LICENSE_MODE=dev` becomes `evaluation` (`local` when a licence file is configured), `DISABLE_AUTH=true` becomes `false`, and a `JWT_SECRET_KEY` or `EREZAGENT_API_KEY` that is shorter than 32 characters or a placeholder is replaced. When the API still cannot start, they print its reason.

## 10. Data Retention & GDPR

### Retention schedule

A daily background job (`enforce_data_retention`, 04:00 UTC) purges data past its retention window:

| Data | Retention |
|---|---|
| `events` | `EVENT_RETENTION_DAYS`, default 90 days |
| `audit_logs` | `AUDIT_LOG_RETENTION_DAYS`, default 730 days |
| Dismissed or rejected attention signals | 30 days (a signal that was promoted into governed work is kept) |
| `email_events` | 1 year |
| Failed sign-ins for unknown usernames (digest, address, time) | 90 days |
| Deal contact email (`deals.contact_email`) | anonymized 6 months after `closed_lost`, or 1 year after `closed_won` -- the deal record itself, and its financial history, is preserved; only the contact address is cleared |
| Business Twin source files (uploaded originals and their extracted text) | `TWIN_SOURCE_RETENTION_DAYS`, default 90 days after a bundle finishes; `0` keeps them. Removed by a separate daily job (`purge_expired_twin_sources`, 03:40 UTC). The evidence quotes stored with each model item are kept. |

The job deletes in batches and only runs when a backup completed within the last 48 hours
(`RETENTION_REQUIRE_BACKUP_WITHIN_HOURS`) or backups are marked as managed externally in Settings;
otherwise it deletes nothing, raises a "Data retention paused" alert in the Attention Feed, and tries again at the next
daily run -- so keep scheduled backups healthy (section 8). Batch size is `RETENTION_DELETE_BATCH_SIZE` (default 5000).
Billing and usage records (the usage ledger, statements, invoices, payments and their audit entries) are financial records: this job does not remove them, and erasing a contact does not delete them. Rendered billing documents print the customer's name or reference, so the export lists them and an erase removes their stored files (they can be generated again). Billing webhook events and delivery logs are pruned automatically after 180 days. Push subscriptions are not purged by age; a dead subscription is removed when a push to it fails. Other records
(conversations, tasks, model-call and token-usage history, approvals, decision records) are not removed by this job.
The audit-log default is 730 days; set `AUDIT_LOG_RETENTION_DAYS` explicitly if you need a shorter window.

The audit trail (`Audit Log`, Governance group) cannot be edited, and no API path deletes it;
only this scheduled purge removes entries older than the audit retention window (730 days by default).

### GDPR posture

- **Right of access:** `GET /api/v1/data-export` (owner-only) produces a ZIP of structured CSV/JSON
  exports covering every table that holds third-party personal data: deals, email events, revenue
  events, decision records, customer contacts, leads, support cases and messages, appointments,
  document submissions and files, business records and signature envelopes, invoices and payments, usage
  statements, rendered billing documents, reseller and license customers, user and staff accounts, and Business Twin source text.
- **Right to erasure:** `DELETE /api/v1/contacts/{email}/erase` (owner-only; optional `?phone=` widens
  the match to a phone number) anonymizes that person's personal identifiers across the same tables,
  for your tenant only. The response lists how many rows each table changed. This is a data-controller
  tool for you to act on a request, not a self-service end-user flow, and it removes personal
  identifiers without deleting the underlying business/financial record. The deployment owner account
  is never erased through this path. Free text in Business Twin sources cannot be matched to one person;
  delete the bundle (or let retention remove it) instead.
- **Stays complete as the product grows (since v4.88.0):** the list of covered tables lives in one place,
  and an automated check fails a release if a new table with personal-looking columns is neither covered
  nor explicitly reviewed.
- **Data minimization:** PII-redaction utilities are applied to logging paths that could otherwise
  capture an email address or similar identifier, including payment-webhook failure logs.

For the full data policy -- what's collected, why, and your obligations as the data controller --
see `docs/31_DATA_POLICY.md`.

### Full system wipe

If you're decommissioning a deployment entirely: stop all services, delete the Docker volumes
(destroys all data), delete backup files, and revoke every external credential (Anthropic, OpenAI,
Google OAuth app, and any other connected provider) from that provider's own console. This is
irreversible -- confirm you have anything you need preserved (an export, a final backup you intend
to keep) before starting.

---

## 11. Licensing

ErezAgent has two independent licensing layers -- keep them mentally separate.

### Layer 1 -- your own platform license

Controls what your own ErezAgent instance can do. `LICENSE_MODE=evaluation` (the built-in default, written by the installers and
by upgrades that add the key) applies the trial countdown. `LICENSE_MODE=dev` is for development machines
only: it skips trial/expiry enforcement entirely and is never written by an installer or an upgrade; an installer or upgrade that finds it
in an existing `.env` changes it to `evaluation` (`local` with a licence file), because the API refuses to start with it. It does not
change the capacity limits for ventures, projects and users, which still follow the subscription or trial.
`LICENSE_MODE=local` uses a signed, Ed25519
license file (`LICENSE_FILE_PATH`, `LICENSE_PUBLIC_KEY`) -- with trial enforcement in three states:
no active trial (`HTTP 402` on every request), expired trial (`HTTP 423`), or an active
trial/license (normal operation).

**Installing a purchased license (since v4.89.0).** The owner opens Settings -> Platform License -> **Enter License Key**
and pastes the signed license file. The signature is verified before anything is saved, a forged, edited,
unsigned or expired file is refused with a clear message, and a valid one takes effect immediately with no
restart (also under `LICENSE_MODE=evaluation`). The install is recorded in the audit log. Once licensed, the
trial screen no longer appears; a lapsed license shows the license-expired screen instead, and a 30-day offline
grace period applies after the license expiry date. The file is stored in the `license_data` volume at
`/app/license/local.json` (the location used while `LICENSE_FILE_PATH` is blank), so it survives restarts,
container recreation and upgrades. To supply the file yourself, set `LICENSE_FILE_PATH` to its path (for example a
read-only mount); that path takes precedence. On Kubernetes it is stored under the `license` subpath of the shared
assets volume. A license applies to the whole installation: every tenant on it receives the license's limits, each
tenant counted separately.

**Trial reminders, expiry and renewal (since v4.91.1).** While a trial runs, a banner appears in the app once 14 days or
fewer remain, and the owner also gets a reminder by email and Telegram (when configured) at 14, 7, 3 and 1 day(s) before the
end. Email is only sent when an email provider is set up (see *Email delivery* in section 15); with the default `mock`
provider the reminders appear in the app and on Telegram only, and the worker log says so once per run. The banner appears from 14 days before the end, so a 14-day trial shows it throughout (the 14-day reminder itself is not sent for a trial that is 14 days long). A reminder counts as sent once Telegram or the email provider delivered it; if every configured channel failed it is tried again on the next daily run, until it is delivered or the window passes. The email goes to the address the trial was started with, otherwise to `OWNER_EMAIL`. When
the trial ends, the app is locked until a licence is entered; your data is kept untouched. To renew, open
Settings -> Platform License, copy the **Machine Fingerprint**, and email it to the vendor contact, or visit the renewal
page; the signed licence you receive is pasted into the same section. The contact and the page are set with
`VENDOR_CONTACT_EMAIL` (default `erez.rozenbaum@gmail.com`) and `VENDOR_RENEWAL_URL` (default `https://www.erezagent.com`, a
placeholder until the vendor site is live; `http` or `https` only). Leave them as they are unless you resell ErezAgent and
want your own customers to contact you; a blank or invalid value uses the default (the address must be one plain `name@domain.tld`, and the URL must not contain a user name or password). They are never taken from `OWNER_EMAIL`.
Licences close to their end are warned about separately, from 30 days before the licence's own end date.

**What a license grants.** One entitlement decision is used everywhere, in this order: the platform-owner
instance (the vendor's own installation: every limit unlimited, every feature on, shown as "Platform owner
instance - everything enabled"), a verified installed license, an active subscription, an active trial. A limit
that an older license does not list is treated as unlimited. Every license includes all feature areas; only the
optional add-ons (white label, OEM, multi-tenancy, local LLM, metering export, API access) depend on what the
file lists. `GET /api/commercial/entitlements/snapshot` shows the source, plan, limits and current usage, and
`GET /api/system/license` shows the installed license.

**Verification key.** `LICENSE_PUBLIC_KEY` ships pre-filled with the vendor public key (the same key is built into
the images); a blank value behaves the same, and a value you set overrides it. Install and upgrade fill a blank
value and never change one you set. After the vendor rotates its signing key, a new license carries a certificate
chain and still verifies with the original key; installs older than v4.89.0 verify only licenses signed directly
by the original key.

**Limits of a license.** The `scope` values of a license (maximum tenants, resale, white label) and its optional
features are shown to you but are not enforced technically. A license is not tied to one machine. Revoking a license
at the vendor does not reach an installation that is offline; it is bounded by the license's expiry date plus the
offline grace period. Signing-key certificates have no expiry, so removing a signing key that is no longer wanted
means shipping a new trust anchor in a release.

Generate a local license with `python tools/generate_license.py` (keygen / sign / verify). Check
live status at **Platform Management -> Licensing**, or `GET /api/platform/diagnostics`.

### Layer 2 -- Licensing Department (customer license issuance)

**This is for you as the platform provider distributing ErezAgent to your own customers, not for
managing your own instance.** **Licensing Department** (Revenue & CRM group) covers Dashboard, Products,
Licenses, Customers, Activations, API Keys, Provider Console, Events and Audit Log. Issue and track customer
licenses, monitor activations, and review license events, all scoped to the customers you're licensing this
product to. A purchase webhook that is delivered twice (same event id or payment confirmation) answers
`duplicate` and issues one license, which belongs to the tenant the webhook was configured for.

**Platform Licenses** (owner-only tab within Licensing Department) is where you issue unlimited or
special licenses to trusted partners (the **Open License** card issues an unlimited licence with every
feature and add-on, forever or for a number of days, as a signed file the customer installs in Settings --
Platform License) -- deliberately restricted to the owner role given the
financial and legal weight of that action. Platform Licenses also hosts the **Commercial Catalog**
editor -- Plans, Add-ons, and Discounts sub-tabs for the price book every install's own Plan &
Billing section reads from. Visible only on your own platform-owner install, never on a customer's. Platform Licenses has three views:
**Licenses** (issue, download and revoke), **Customers** and **Signing key**.

**Platform customers and vendor-side billing (since v4.89.0).** In **Platform Licenses -> Customers** you keep a
record per customer organization (owner only, platform-owner installation only): contracts and the licenses
issued under them, invoice drafts, payments, renewals, license updates and usage reports. A license is valid for
its billing term (monthly 31 days, annual 366 days, lifetime has no expiry) plus a 30-day offline grace period.
A **renewal** issues a new license with a new expiry, drafts the renewal invoice and revokes the old license as
superseded in one step, after you confirm. A **license update** (more capacity, for example) shows a preview first;
limits can only rise, and the prorated invoice draft is created with the new license. A customer's data can be
exported or erased from the same screen. A manual issue creates or links a platform customer, and a purchase from
the marketing site is recorded once per payment confirmation, with the sale on your own workspace. These invoices
are drafts, not tax invoices.

**Provider Console** (Licensing Department tab, since v4.51.0, any install's own owner -- not
restricted to the platform owner) is where a Provider/MSP configures downstream distribution: a
Signing Keys panel (current Ed25519 key version, rotate on demand -- previously issued signed
licenses stay valid; see the rotation procedure below), a **Volume tier charge** preview (the tier from your
provider contract that applies to your active managed installs; a preview, not an invoice), a Purchase Webhook panel (configure the HMAC secret your marketing site uses
to trigger real license issuance, and see delivery status), a form to register a downstream managed
install, and the managed-install list with a deactivate action that revokes that install's signed
license. Each managed install has an owner-only form to record a usage report (customers, ventures, active users)
and a history of the reports recorded so far.

Semi-public endpoints (no ErezAgent auth -- for a customer's own SDK to call): `POST
/api/licensing-dept/validate`, `POST /api/licensing-dept/activate`, `POST
/api/licensing-dept/deactivate`.

### License signing key rotation (platform owners)

Only the platform owner's installation rotates the signing key (owner role, signed-in account). A new key is
certified by the key your customers' installs already trust, so no customer action is needed and every license
already issued keeps working. At most three certified generations can follow the original key; after that a new
release must ship a new trust anchor.

1. Upgrade your customers to v4.89.0 or later first. Older installs verify only licenses signed directly by the
   original key and would not accept a license signed by a rotated key.
2. Take a backup (Backup Center). `LICENSE_PRIVATE_KEY` and `CREDENTIAL_ENCRYPTION_KEY` are never part of a backup
   archive, so keep your own copy.
3. In Platform Licenses -> Signing key (or `POST /api/system/rotate-license-key`), rotate. The new private key is
   shown once and is never stored; `GET /api/system/license-signing-key` shows the active version without key material.
4. Copy the private key into `LICENSE_PRIVATE_KEY` in `.env`, then restart `api` and `worker`. Until you do,
   issuing licenses is refused.
5. Issue licenses as usual; each one carries the certificate chain customers need.

If a rotation is refused, the message says why (for example, `LICENSE_PRIVATE_KEY` does not match the active key).

### Plan & Billing (self-service subscription, since v4.51.0)

**Settings -> Plan & Billing** is where an install's own owner manages that install's commercial
subscription: usage bars per entitlement dimension (users, ventures, projects, client workspaces),
a monthly/annual/lifetime billing-term switcher, plan upgrade/downgrade cards with live proration,
add-ons, promo-code redemption, billing history, and (since v4.77.0) a **Cancel Subscription**
control that cancels at the end of the current billing period -- full access continues until then,
at which point a daily background job applies the real cancellation. Every write action here is
owner-gated on the backend, even though the section itself renders for anyone who can reach
Settings. Add-ons are available only on subscription installs: a new add-on is pending payment and does not count
until a payment is recorded; on a licensed install they are provided by your vendor through an updated license.
A subscription that is pending payment grants nothing.

An unpaid subscription billed through a payment gateway is handled by a daily job. Once its paid period has ended
and the last payment failed, it enters a **grace period** of 14 days (the plan stays active and Plan & Billing
shows the end date), then it is **suspended** and the plan's limits stop applying. You get an attention signal
for each step; a confirmed payment restores the subscription and closes them. Your data is never deleted or changed.
Metering-billed and lifetime subscriptions are not affected. While the plan is suspended, actions that need capacity
answer that no plan is active and point to Plan & Billing.

Pelecard has no billing of its own, so ErezAgent charges the saved card. A declined Pelecard charge is tried again
about 3 and 6 days later during the grace period (three charges at most per period), each retry recorded as its own
invoice. If the card is still declined, or a charge result is unclear, no further charge is made: settle the payment
with the customer, then the platform owner reconciles the invoice as paid
(`POST /api/commercial/admin/pelecard-invoices/{invoice_id}/reconcile`, outcome `paid`) to restore the subscription.

### Billing method, Billing Setup and invoicing (since v4.89.0)

Billing is a setting, for the workspace or per venture: **metering** (no payment gateway; you record payments
yourself) or **Stripe, PayPal, Rapyd or Pelecard**. Pelecard cannot collect plan payments (it still confirms
Revenue Center deal payments), and Rapyd and Pelecard are not available for Venture Commerce. Changing the method
is an owner approval, is kept in a history and can be withdrawn while pending. Switching from metering to a gateway
needs the customer to complete a checkout; switching back cancels the provider subscription at period end and you
renew manually. Existing invoices keep their provider tag, and refunds go to the original provider. Stripe, PayPal,
Rapyd and Pelecard: signature verification and event handling are covered by automated tests; none has been verified
against a live account in this release. Billing Setup shows "Not yet verified with a live account" on each gateway;
run Test connection and a small real payment before relying on one.

**Billing Setup** (Platform Management) is where the owner enters, tests and removes gateway credentials. They are
stored encrypted per workspace or venture and never shown again; keys already in `.env` keep working as a fallback.
Each workspace has its own webhook addresses to give the provider (`/api/webhooks/commercial/{tenant_id}/{provider}`
for plan billing and `/api/webhooks/revenue/{tenant_id}/{provider}` for Revenue Center deal payments); a request
whose signature does not match is rejected. A gateway that is not configured answers `409`; simulated provider
responses exist only in development.

The installation's administrator also sees a **Pelecard status codes** card in Billing Setup. It lists which Pelecard
status codes count as a card refusal (the card was not charged, so an unpaid renewal is retried after the retry delay)
and which of those are never retried because only a new card can fix them. The defaults are 001-199 and 006, 033, 036,
039; they apply to the whole installation. Adding a refusal code asks for an explicit confirmation, because a retry after
a refusal charges the card again. Every change is recorded in the audit log with the lists before and after.

The chain is **usage ledger -> usage statement -> invoice draft -> payment**. Statements and invoice drafts are
not tax invoices; the documents say so.

- **Usage ledger** -- an append-only record of AI usage and cost, actions, agent runs, missions, capacity snapshots
  and venture-defined units, attributed to venture and project where known. Owners add dated corrections as
  adjustments; entries are never edited.
- **Statements and invoice drafts** -- generate a period statement, finalize it (a finalized statement is
  immutable; revise creates a new one), then draft an invoice. Prices come only from the unit prices you set; an
  unpriced line is flagged, never guessed. Invoice numbers are sequential per workspace and assigned at finalize;
  void and credit note need an owner approval. A recurring draft is created each period and is never finalized
  automatically. Foreign-currency invoices need a stored exchange rate (there is no built-in rate table).
- **Payments** -- record amount, method, date, reference and an optional evidence note, or import a bank CSV
  (preview, then confirm). Marking an invoice paid requires an amount, a reference and the date received. Each
  payment is processor-confirmed, owner-attested or imported; partial payments and reversals are supported. A fully
  paid invoice activates its subscription or pending add-on.
- **Documents** -- PDF or DOCX in English, Hebrew or Arabic, with a content hash and a "not a tax invoice" notice.
  Document settings hold your payment instructions, issuer name and details, and document language. Sending is a
  governed email action; Gmail attaches the file, other connectors state that it was not attached.
- **Access** -- billing changes need an owner signed in with a real account (the static API key is refused);
  operators can read; observers and venture staff have no access. Writing billable usage (`POST /api/usage/events`,
  adjustments) is owner only; operators can refresh the ledger. Every money action is written to the audit log.

### Billing integrations

Your accounting system, ERP or usage-billing platform can receive or read the same data. Full field reference:
`docs/88a_EXTERNAL_BILLING_CONTRACT.md`.

- **Signed webhooks** (Revenue Center -> Billing -> Integrations, or `/api/billing/webhooks`; owners manage,
  operators read). Events: `invoice.finalized`, `invoice.paid`, `statement.finalized`, `payment.recorded` and
  `payment.reversed`. Each request carries `X-ErezAgent-Signature: t=<unix seconds>,v1=<hex>`, where `v1` is the
  HMAC-SHA256 of `"{t}."` followed by the raw request body, keyed by the endpoint's signing secret (shown once; you
  can rotate it). Verify against the raw bytes, compare in constant time and reject a timestamp more than 5 minutes
  off. Delivery is at least once and retried with growing delays (about 1, 2, 4, 8 and 16 minutes) up to six
  attempts, after which the delivery is marked dead and an attention signal is raised; use the event id to
  de-duplicate. Triggers are manual, on finalize or on a schedule. Send queues the delivery (`202`); Test sends a
  signed test event. A new endpoint only receives events that happen after it was created.
- **Private networks.** Targets must use `https` and are checked on every delivery. An endpoint on a private
  network is allowed only after an owner approval (`billing_webhook.allow_private`) that names the exact host, port
  and addresses; deliveries go only to those, and a host that later resolves elsewhere needs a new approval. The
  permission is removed if the URL changes. This installation's own networks, services and ports stay unreachable,
  and the delivery record keeps the status code and size of a private target's response, not its text. When
  upgrading from a release before v4.89.0, an endpoint that was already allowed shows "Re-approval required" and sends
  nothing private until you choose "Request approval again" and approve it; refused sends say why on the delivery.
- **Scoped tokens** (`/api/billing/tokens`, owner). Prefix `eab_`, shown once, scopes `billing:read` and
  `usage:ingest` (a venture-bound token that reports venture-defined usage). Rotate with an overlap period or revoke
  at once. A token opens only the read API under `/api/billing/v2/`.
- **Exports** -- `GET /api/billing/export/{statements|invoices|payments|ledger}.{json|csv}` with filters, paging and
  an optional spreadsheet byte-order mark; owners can mark exported rows. The earlier meter downloads still work and
  are deprecated.
- **If you used the earlier push**, its payload and signature changed in v4.89.0; update your receiver.

### Usage reports for your vendor (self-hosted installs)

If your license comes from a vendor who bills by usage, open Revenue Center -> Billing -> Usage and choose
**Export usage report for my vendor** for a past month. The file holds aggregate figures only (no customer names), a
checksum and your attestation, and is never sent automatically: you send it to the vendor, who imports it.

The collapsed **Advanced: Contract Overrides & Value Share** panel records negotiated
Enterprise/Provider terms (reseller margin, revenue share) and Business-Operations-Runtime
value-share contracts, agreed directly with the platform owner and entered here afterward. It's
self-referential to this install's own subscription and available to the install's own owner on a
single-workspace installation, since every install (including the platform owner's own) tracks a
subscription about itself. On the platform owner's installation, the **Workspace** list in this panel lets the platform owner create, attach, detach
or end overrides for another workspace of the installation; the change is recorded in that workspace's billing audit log and
in the platform owner's own. Each active override has an **Attach** button: the attached override's custom limits
(and a provider's volume tiers) apply to this install; **Detach** returns to the plan's own limits, and **End** retires the override for good. Owner only; on an installation with several workspaces only the platform owner creates, attaches or ends overrides. Reseller margin and revenue share apply to provider overrides only. When you create an override you can set the five capacity
limits (Ventures, Active projects, Active users, Client workspaces, Staff seats): leave a field empty to keep the plan's
limit, enter a whole number to replace it, or enter "unlimited". Only an override that is active and your own applies.

A value-share contract is activated only through an owner approval pinned to that contract version. A scheduled
job then proposes one charge per completed measurement window from the strongest real evidence available (payment
processor records, then a connector-verified outcome, then an outcome attested by a signed-in owner or finance
approver) -- never an AI estimate. The owner confirms, disputes or reverses each charge, and a confirmed charge
becomes an invoice line.

---

## 12. System Health & Diagnostics

### Live Health

**System Health** (Governance group) -> **Live Health** tab shows a real-time, 20-component
health dashboard: API, database, Redis, worker, connectors, license validity, security service,
sandbox executor, document renderer, Support channel reachability, self-service domain status, and
more -- green/amber/red per component, refreshed continuously.

```bash
curl -X POST http://localhost:8000/api/system-health/refresh
# Look for "overall_status": "healthy"
```

**Critical components** (any failure drops overall status to `critical`): `redis`, `postgresql`,
`api`. Everything else degrades independently without taking the whole dashboard down.

Container health checks: the API answers `/health` (checks PostgreSQL and Redis) and `/health/live` (process only).
The worker, connector and security checks confirm the service is running its loop, not only that Redis answers, so
`docker compose ps` shows `unhealthy` for a hung worker. On stop, the API gets 30 seconds and the worker 2 minutes to
finish what it is doing.

### Diagnostics

The **Diagnostics** tab is your setup checklist and preflight verification -- confirms every
required environment variable is configured, every connection is reachable, and migrations are at
the expected head. Run this during initial setup and after any major configuration change.

### Readiness Check (v4.65.0)

The **Readiness** tab, next to Live Health and Diagnostics, answers a different, deeper
question than the Diagnostics checklist above: not "is everything configured" but **"is
this installation actually ready to operate."** Every item is one of five real states --
VERIFIED (a real check ran and passed just now), CONFIGURED BUT NOT VERIFIED, NOT
CONFIGURED, FAILED, or NOT APPLICABLE -- never a bare pass/fail. Nothing is ever reported
ready just because its database row exists.

Covers 8 areas: core platform, AI providers (a real, minimal connectivity call to each
provider you've configured -- the first real ping anywhere in the system, since previously
only a key's *format* was ever checked), the Executive Board's configuration, execution and
approvals (a real, zero-side-effect action is created, approved, and executed to prove the
approval path genuinely works), capabilities, notifications, security, and backup.

Click **Run Check Now** (owner-only -- it makes real AI-provider calls and creates a real
approval, so it's never triggered by a non-owner role) to run one on demand. It also runs
automatically once after a fresh install and once after a real upgrade -- never on a
recurring schedule. A NOT READY result is surfaced clearly but does not block starting a
mission in this release.

### Readiness banner (Operational Readiness Audit)

A small amber banner may appear at the top of every page listing anything still needed before your
company is fully operational -- Company Foundation, timezone, access/domain, backup protection,
owner communication, or platform health. Each item links straight to the page that resolves it and
the banner disappears once everything is resolved. This is a separate, simpler onboarding checklist
from the Readiness Check above -- the banner confirms basic setup items exist; the Readiness Check
proves core systems genuinely work.

### Platform Management tools

**Platform Management** (Platform group, 16 tabs) rounds out day-to-day platform administration:

| Tab | Use it for |
|---|---|
| Overview | License status and system vitals at a glance |
| Usage | Resource consumption vs. plan limits |
| Update Planner | Compares your current version to the latest -- estimated downtime, backup readiness, rollback availability, migration requirement, risk level, breaking changes |
| Support Bundle | Downloads a ZIP of diagnostics, self-score, migrations, license info (no private keys), redacted config, recommendations, and preflight results -- safe to hand to support |
| Benchmark | Live CPU, memory, DB latency, Redis latency, disk I/O, external API latency |
| Recommendations | Proactive, rule-based suggestions from your self-score, DB health, Redis pressure, and connector state |
| Installer Report | Post-installation summary -- version, license, image tags, migration status, service health, credential configuration, backup status |
| Mail Log | Confirms outbound email delivery |
| Localization | Read-only view of locale config and market profiles |

---

## 13. Human/AI Workforce Unification Administration

*(since v4.45.0)*

A task or mission step can be owned by a real person or a real AI agent, interchangeably -- humans
and AI are equally valid owners of the same unit of work.

### Waiting-on and hand-off

Either side can be marked as genuinely **"waiting on"** the other, with a stated reason -- between
two AI agents, or between a person and an agent, in either direction.

When an AI agent needs human judgment or an action it genuinely can't take itself, it can hand that
work directly to a real person: this creates a real task assigned to them and sends **exactly one**
correctly-targeted notification -- never a broadcast to every registered device. The agent
automatically picks the thread back up once the person marks the work done.

### The one governance rule that never bends

**Moving a task's ownership between a human and an AI agent is always a governed proposal that
requires human approval.** An AI agent can never silently reassign a person's work, and a person
reassigning work to an agent still goes through the same approval discipline as any other
ownership change. This is enforced the same way every other high-risk change in this system is --
a real `Action` + `Approval` pair, never a direct write.

### What this looks like day to day

- **Ask in plain language**: *"what's waiting on me,"* *"what's blocked by \<person\>,"* or *"who's
  overloaded"* -- five dedicated skills answer with real, disclosed counts, never a fabricated
  score.
- **Blocked across the workforce**: the Executive Office page's Waiting For panel has a tenant-wide
  list, separate from and never merged into your own private Waiting-On list, surfacing overdue
  human-owned work and the AI work genuinely stalled behind it anywhere in your organization.
- **Waiting-For commitments can link to a real known person or agent**, not just free text, so a
  commitment's real owner is always resolvable rather than only described.

### What to watch as an administrator

Review the "Blocked across the workforce" list periodically the same way you'd review a stalled
mission -- it's the fastest way to see where human bandwidth, not AI capability, is the actual
bottleneck. Approval requests for ownership reassignment appear in your normal Approvals queue like
any other governed action.

---

## 14. Installation & Upgrade

Installation and upgrade procedures -- prerequisites, the Web Installer Wizard, manual setup, SSL,
Kubernetes/Helm deployment, and every documented upgrade path -- live in `docs/DEPLOYMENT_GUIDE.md`.
This guide intentionally doesn't restate those steps; follow that document for a fresh install or a
version upgrade, and come back here for what you configure and govern once ErezAgent is running.

A few pointers relevant to administrators specifically:

- **Rollback**: `installer/rollback.sh` (Linux/macOS) and `installer/rollback.ps1` (Windows) switch back to an earlier
  version, optionally restoring a Backup Center checkpoint; they accept only a plain `MAJOR.MINOR.PATCH` version. Since
  v4.89.0 a rollback to an older version also reverts the database schema to that version's migration head, which
  removes the tables and columns newer migrations added and the data in them (leaving v4.89.0 removes the usage ledger,
  statements, invoices, payments and rendered billing documents). The script lists what will be removed, asks you to
  type `confirm` (or takes `--yes`; `--dry-run` only prints the plan). It then pulls and verifies the target images
  while the running version keeps serving, stops the application services, writes a database dump into the backups
  folder, reverts the schema and only then switches the version. If the schema revert fails, the script stops before
  any image or version is changed, starts the current version again and prints the command that restores the dump.
  Statements or tenant rows that the older schema cannot hold are copied to `customer_usage_meters_downgrade_archive`
  before they are removed. A failed upgrade reverts the database schema only to the revision recorded before its
  migrations ran.
- **Infrastructure files**: install and upgrade bring `infrastructure/` (the Caddy, Prometheus, Loki, Promtail, Tempo and
  Grafana files and the docker-socket-proxy template `docker-compose.yml` mounts) in step with the release. A missing file is created. A file that is a copy
  an earlier release shipped is replaced after its previous copy is kept under `infrastructure.backup-<timestamp>/`.
  A file you changed (for example an extra site block in the Caddyfile) is kept as it is; this release's copy is written
  next to it as `<file>.new` and one warning lists those files. Merge what you need into yours, or replace it with the
  `.new` copy, then restart its service (`docker compose restart <service>`). Your own additional files are not touched.
  The services that read a replaced file are restarted unless they were started after the files were written. A rollback
  keeps the compose file, so it keeps these files too.
- **License after an upgrade**: a license pasted in the UI of v4.88 or earlier is carried over to the license volume by
  `upgrade.sh` / `upgrade.ps1`. If the license screen shows expired after upgrading, paste your license again.
- **Version validation**: every documented upgrade script checks
  `GET /api/platform/upgrade/validate?from_version=X&to_version=Y` before pulling any images, and
  aborts up front if the path isn't supported.
- **Image signature verification** (since v4.49.0): after `docker compose pull`, install/upgrade
  verifies each pulled image's cosign signature when the release channel publishes a signing public
  key (`installer/cosign.pub`). Until that key is published the scripts print an informational line
  ("Image signature verification is not enabled for this release") and continue; the images are then
  trusted by tag and registry only (this is informational until the release channel publishes a key). Once the key is published and `cosign` is not installed, they print
  "Image signatures were NOT verified" and continue. Once the key is present, a verification failure blocks a normal
  install/upgrade outright; the emergency rollback path only warns and continues, since getting back
  to a known-good state takes priority there. Do not bypass a real verification failure.
- **Update Planner** (Platform Management, see section 12) tells you what an available update
  actually involves before you run it -- downtime estimate, backup readiness, migration
  requirement, and risk level.
- **Migrations** run automatically on container start (`entrypoint.sh` calls `alembic upgrade
  head`) for both a fresh install and a normal upgrade -- manual migration commands are rarely
  needed outside of troubleshooting (see section 19).
- **Data-volume ownership** (since v4.87.0): install, upgrade, and rollback scripts repair ownership of the data
  directories the app writes to (`/obsidian-vault`, `/repos`, `/creative-assets`, `/generated-documents`,
  `/persona-assets`) automatically -- no manual `chown` step. Upgrade scripts also append any environment keys
  missing from your `.env` (the shell, PowerShell, and web installers share one identical list of known keys) without touching values you already set, and an `.env` with no trailing newline is handled correctly.
- **Kubernetes shared storage** (since v4.87.0): the Helm chart now provisions shared `ReadWriteMany` volumes
  `<release>-vault` (documents and vault notes) and `<release>-assets` (creative assets, generated documents,
  persona images) in addition to `<release>-repos`, because `api` and `worker` run in separate pods. They need an
  RWX-capable storage class; check that your storage class supports `ReadWriteMany` before `helm upgrade`
  (`vaultStorage` / `assetsStorage` in `values.yaml`, see `KUBERNETES.md`).
- **Backups and retention**: the daily retention job (section 10) waits for a completed backup, so a failing backup
  schedule also pauses retention. Backup Center shows the last completed backup.
- **Initial plan selection** (since v4.51.0): every installer now prompts for a real commercial
  plan (Starter through Enterprise) instead of the old placeholder tier list. This only sets a
  starting preference for the Plan & Billing screen (section 11) -- your trial always gets full,
  generous limits regardless, and you (or any customer you're setting this up for) can change
  plans, or receive a free/custom grant, at any time afterward without conflict.
- **Android Customer Delivery** needs roughly 11 GB of additional disk for its build image (about 4 GB to
  download). Install and upgrade no longer download it on every run: it is downloaded on request
  (`--with-android` / `-WithAndroid`, or the install prompt), when it is already in use, or automatically the first
  time an Android app is built, and not again while its toolchain is unchanged -- see `docs/DEPLOYMENT_GUIDE.md` for the real
  `/dev/kvm` hardware-acceleration prerequisite (Android builds still work without it, just slower).
  Generated apps use Expo SDK 57 (React Native 0.86, Android API 36 toolchain); an Android build image left
  over from an earlier release is detected as out of date and rebuilt on the next Android build (10-20 minutes)
  if the upgrade could not pull the current one.
  Signing-credential management is owner-only: `POST /api/mobile/projects/{project_id}/signing`
  stores a project's Android keystore Fernet-encrypted at rest and tenant-isolated; every read
  returns only a `fingerprint_sha256`, never the raw key.
- **Upgrading to v4.94.5:** migration 566 runs on start and adds `sales_leads.venture_id`; existing leads stay without a venture. The upgrade scripts refresh their files only from the install folder, and the web installer's upgrade keeps running when its page is closed.
- **Upgrading to v4.94.4:** no migration. The upgrade scripts and the web installer can fetch the release with a read-only GitHub token (`EREZAGENT_DEPLOY_TOKEN`) when the `ErezAgent-Deploy` repository is private for you; see "Updating ErezAgent" in the Deployment Guide.
- **Upgrading to v4.94.3:** no migration. The upgrade scripts refresh the installer folder from the release they install and check that every core service runs it; see "Updating ErezAgent" in the Deployment Guide. A script older than v4.94.3 needs the one-time refresh described there.
- **Upgrading to v4.94.2:** migration 565 runs on start. Starter's annual price in the catalog becomes 990 USD as a new plan version; existing subscriptions keep the version they bought.
- **Upgrading to v4.94.1:** no migration. On the platform owner's own installation, the website forms can post leads to `/api/public/site-leads`; other installations answer 404.
- **Upgrading to v4.94.0:** migrations 555-564 run on start. Every container is recreated once because the log tag option changes (container names then appear in Loki). The Docker socket proxy now allows only container details, daemon info, version and ping; promtail reads the container log files instead of the Docker API, so a Docker data root other than `/var/lib/docker` needs the same path for promtail as for cAdvisor. If you replace `docker-compose.yml` by hand, replace the `infrastructure/` folder with it. The installer compose file requires `EREZAGENT_VERSION` (the install and upgrade scripts set it). Connecting to a provider is optional and off by default (section 24). Retention now also covers log tables (model calls, runs, error, notification and audit rows) using the event and audit windows; for the first 7 days it only reports what it would delete, and after that the monthly model-call summary keeps no history older than 400 days. A workspace uses the installation's own provider accounts only when it is the only workspace or the home workspace; on an installation with several workspaces, the other workspaces connect their own HubSpot, Stripe, Rapyd, Pelecard, PayPal, Pipedrive, Salesforce, Hetzner, Vercel, Cloudflare, PostHog and LEON accounts. Data contracts are now applied to every read and write of an interface that has one.
- **Upgrading to v4.93.0:** migrations 553 and 554 run on start. Policies: the Read CRM, Create Deals, Send WhatsApp, Spawn Sub-Agents and Modify Agents flags now apply; every saved policy was set to ticked, so untick what an agent must not do. Subscriptions: a gateway-billed subscription that stays unpaid after its period ends enters a 14-day grace period and is then suspended (data is kept; a confirmed payment restores it). Events: each event now belongs to a workspace; on a shared installation, events that cannot be attributed are kept but not shown to workspaces. YouTube: reconnect existing YouTube connections once to grant the comment permission. Docker: cAdvisor and promtail now read containers through a filtered `docker-socket-proxy` service; the upgrade script brings the `infrastructure/` files in step with the release (copies of an earlier release are replaced and kept under `infrastructure.backup-<UTC timestamp>/`; a file you changed is kept and the release's copy is written next to it as `<file>.new`). Docker Compose 2.17 or newer is required. Contract overrides: on an installation with more than one active workspace only the platform owner creates, attaches or ends them.
- **Upgrading to v4.92.3:** no migration. On the platform owner's own installation, Platform Licenses has an Open License card that issues a signed licence (every limit unlimited, every feature and add-on on) forever or for a number of days. Customer installations are not affected and do not show the Platform Licenses screens.
- **Upgrading to v4.92.2:** no migration. Tell owners that executing an agent task now waits for its result (a few minutes for a task with several steps) instead of reporting a timeout, and that a task run shows its tokens and cost. Asking the Executive Office to launch a plan the Board recommended reaches the CEO agent; the documentation search skill answers in container installations.
- **Upgrading to v4.92.1:** three migrations run automatically. Agent policies become one per workspace. The Google resource's Pro-tier capabilities move to **Gemini 4 Argon** (a row you edited is left as it is; agent model assignments are not changed -- pick it in an agent's model setting if you want it); it is priced $2 / $10 per million tokens until 30 November 2026 and $4 / $20 from 1 December, and it is not offered on the Google AI Pro plan. The Chief of Staff tables get strict workspace isolation (rows without a workspace go to the only workspace on a one-workspace installation) and the audit log can no longer be edited or deleted by the application. Rolling back to an earlier version removes the extra workspaces' agent policies (the oldest per agent is kept) and restores the earlier policies and privileges; the rollback plan lists it, and `--allow-unverified` / `-AllowUnverified` restores a backup that has no stored checksum. On Kubernetes the sandbox executor and the document renderer get their own token Secret instead of the application Secret. The sandbox executor creates the `erezagent_projects` network on start, and `docker compose down` no longer removes it. Android builds still reach the public internet in the default and scoped network modes (`isolated` mode cuts build egress), and the first sandbox use after the upgrade rebuilds the utility image, which needs internet access. Release assets include a `SHA256SUMS` file. On an installation with more than one workspace, platform diagnostics cannot be read with the install key (the upgrade continues with a notice), and a restore from a backup job needs an instance administrator session.
- **Upgrading to v4.92.0:** no migration. The upgrade scripts now take the pre-upgrade backup through the Backup Center and stop if it cannot be created (`--skip-backup` / `-SkipBackup` continues without one; an installation with more than one workspace takes the backup from the Backup Center and uses that flag). Tell owners that **Open** on an objective now opens its own page, that approvals list the requesting agent, the reason, and the mission and goal, and that an agent task's send or change step needs a connector authorised for it: without one it is shown as blocked, with the reason, instead of raising an approval that could not be carried out.
- **Upgrading to v4.91.6:** nothing manual (no migration). Long conversations that use tools complete normally, and answers an agent saves to memory are saved under the right workspace.
- **Upgrading to v4.91.5:** nothing manual (no migration). Re-download the installer folder first. If an earlier upgrade ended without changing the version shown in the app, your `.env` probably has no `EREZAGENT_VERSION` line: add `EREZAGENT_VERSION=<version>` to `.env` and run `docker compose up -d --force-recreate`. The upgrade and rollback scripts now add the line themselves and check which version Compose will start before stopping anything.
- **Upgrading to v4.91.4:** nothing manual. Migration 549 removes the example business records early releases created in every installation (deals, a capital plan, a revenue goal, assets, contracts, competitor items, predictions, calibration scores); a row you edited or linked to your own data is kept. After it, the Board and the dashboards show only your own data, so a new workspace shows empty states until you add records. Executive Board roles on Claude Sonnet 5.5 are no longer cut off at the output limit.
- **Upgrading to v4.91.3:** nothing manual (no migration, the application is unchanged). Re-download the installer folder first. The install and upgrade scripts and the web installer now correct an `.env` the API refuses to start with: `LICENSE_MODE=dev` becomes `evaluation` (`local` with a licence file), `DISABLE_AUTH=true` becomes `false`, and a `JWT_SECRET_KEY` or `EREZAGENT_API_KEY` under 32 characters or a placeholder is replaced (a new `JWT_SECRET_KEY` signs everyone out; update any script that used the old API key). When the API still cannot start, they print its reason.
- **Upgrading to v4.91.2:** nothing manual (no migration). Re-download the installer folder first (`lib/` has new files). The upgrade now checks Docker's free disk space before downloading (`--force` / `-Force` continues anyway), stops cleanly with nothing changed if a download fails, and removes older ErezAgent images afterwards, keeping the current and the previous version (`--no-prune` / `-NoPrune` keeps them); the first run can free a lot of space. Completed manual backups beyond the newest 10 that are also older than 30 days are deleted by the daily backup job from now on (`BACKUP_MANUAL_KEEP_MIN`, `BACKUP_MANUAL_KEEP_DAYS`, `0` for both keeps everything); set them before upgrading if you want to keep old manual backups.
- **Upgrading to v4.91.1:** nothing manual (no migration). Two optional settings, `VENDOR_CONTACT_EMAIL` and `VENDOR_RENEWAL_URL`, are added blank and use the built-in defaults. To have trial and license reminders emailed, set `EMAIL_PROVIDER=sendgrid` with `SENDGRID_API_KEY` and `EMAIL_FROM_ADDRESS` (or connect Gmail); the Readiness Check shows whether delivery works. Lines for the 14 removed settings in an existing `.env` are ignored.
- **Upgrading to v4.91.0:** nothing manual on a single-workspace install (migration 548 runs automatically; existing custom domains keep working and can be proven with a DNS TXT record). Re-download the installer folder first:
  `installer/lib/compose-health.sh` and `.ps1` are new, and the upgrade script now waits for every core service to be healthy
  and rolls back if one is not. In production the API and worker refuse to start when `JWT_SECRET_KEY` or `EREZAGENT_API_KEY`
  is shorter than 32 characters or a placeholder, or when `LICENSE_MODE=dev`; from v4.91.3 the install and upgrade scripts correct these in an existing `.env`, before that check `.env` before upgrading. Installs with more
  than one workspace must move Twilio, Meta Lead Ads and LinkedIn lead webhooks to the workspace-specific URL and make custom
  n8n workflows echo `run_id` in the completion callback. Kubernetes installs: the chart defaults changed (email and voice
  providers start as `mock`, sender name ErezAgent, 7-day sessions), so set what you rely on in your values before
  `helm upgrade`, and set `configMap.ALLOWED_ORIGINS` for CORS (see KUBERNETES.md).
- **Upgrading to v4.90.0:** nothing manual. Migration 547 runs automatically: Claude Haiku 5.5 becomes the fast tier, replacing Haiku 4.5, 
  for the routing rows and built-in catalog entries that still hold the previous default. Anything you changed yourself and every agent's
  own model assignment stay as they are; Haiku 4.5 remains selectable. An existing `.env` keeps its `HEALTH_NARRATIVE_MODEL` value.
  OpenAI voice: speech-to-text uses `gpt-transcribe` and text-to-speech uses `gpt-realtime-2.1-mini` (OpenAI retires `whisper-1` on
  26 Feb 2027 and `tts-1` on 6 Jan 2027). Spoken replies from OpenAI cost somewhat more; ElevenLabs and the
  local voice option are unchanged. `OPENAI_STT_MODEL`, `OPENAI_TTS_MODEL` and their `*_FALLBACK_MODEL` settings are optional
  overrides (the upgrade adds them to `.env` with these defaults; leave a fallback empty to disable it).
- **Upgrading to v4.89.3:** nothing manual. Migration 546 runs automatically. Kubernetes (Helm) installs: set
  `appDatabase.host` (or keep `APP_DATABASE_URL` in your own Secret) before `helm upgrade`; see KUBERNETES.md. After
  upgrading, `POST /api/obsidian/relocate` (dry run by default) can move older vault notes of non-home tenants into their
  own area on a shared install. `docker compose up -d` recreates Prometheus once so it picks up its healthcheck.
- **Upgrading to v4.89.2:** nothing manual. If `CREDENTIAL_ENCRYPTION_KEY` in `.env` is not in the required format (an
  installation made by an earlier Windows or web installer), the upgrade script sets it; a key already in the right format
  is never changed. If a new installation's API does not become healthy at the database-migration step, set the key in
  `.env` to one made with `python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"` and
  run `docker compose up -d --force-recreate api worker`.
- **Upgrading to v4.89.0:** nothing manual. Migrations 529-545 run automatically, and the upgrade scripts fill a
  blank `LICENSE_PUBLIC_KEY` with the vendor key (a value you set is kept). Things you may notice afterwards: billing
  changes are owner only, operators can read billing data and observers can read the usage ledger and capacity only; marking an invoice paid needs an
  amount, a reference and a date; a billing webhook receiver built for the earlier push must be updated (new payload
  and signature); Pelecard no longer collects plan payments; payments in other currencies need a stored exchange rate;
  and the API container can take about two minutes to become healthy on a cold start (its health-check grace period is
  180 seconds). Kubernetes installs that run the bundled PostgreSQL or Redis must add `--set postgresql.enabled=true` (and `--set redis.enabled=true`)
  to every upgrade, or move to an external database (see KUBERNETES.md). Platform owners should rotate the license signing key only
  after their customers have upgraded.
- **Upgrading to v4.88.0:** nothing manual. Migrations 508-528 run automatically on start. The `api` and `worker` images
  carry the document readers Build My Business uses (including `pypdfium2` for scanned PDF pages); no system packages
  or host changes are needed. The upgrade scripts append the four new `TWIN_*` keys to `.env` with their defaults (section 15), and
  `helm upgrade` adds the same four keys to the ConfigMap. Things you may notice afterwards: AI analysis of documents
  is off until the owner grants consent (Settings > Data & Privacy), and until then photographed customer documents wait in the
  Review Queue; document text sent to AI providers is masked by default; changing an agent's status is owner-only; workflows
  that require approval pause for it, and only an owner can change that setting; a workflow with an unknown step type is rejected; and the core
  agents are shared templates with budgets and status kept per tenant (section 2).

---

## 15. Environment Variables Reference

All variables belong in `.env` -- never in code, never committed to git. This section covers the
ones an administrator is most likely to touch directly; `env.example` is the complete, authoritative
list.

Since v4.91.1 the templates no longer list keys that no service ever read: `AUTO_BLOCK_CRITICAL_CVE`,
`AUTO_BLOCK_SECRET_LEAK`, `SECURITY_SCAN_ENABLED`, `SECURITY_MIN_SCORE`, `BUDGET_WARNING_PERCENT`,
`BUDGET_CRITICAL_PERCENT`, `WORKER_MAX_JOBS`, `WORKER_JOB_TIMEOUT`, `WORKER_KEEP_RESULT`, `SLACK_ENABLED`,
`SLACK_CHANNEL_ID`, `SLACK_COMMAND_CHANNEL_ID`, `METRICS_ENABLED` and `AWS_SES_REGION`. A line left in an existing `.env` is
ignored and harmless; delete it whenever convenient. The real controls: blocking on critical CVEs and secret leaks and the
budget warning and critical thresholds are per-workspace settings (Settings); Slack is connected in Connectors; metrics are
always on; worker concurrency is fixed in the worker.

### Core (required)

| Variable | Description |
|---|---|
| `EREZAGENT_API_KEY` | Master API key -- every API call requires `Authorization: Bearer <this>` |
| `JWT_SECRET_KEY` | Signs session tokens. Rotating expires every session immediately. |
| `CREDENTIAL_ENCRYPTION_KEY` | Fernet key encrypting connector credentials, MFA secrets, and signing keystores at rest. Must be a valid 32-byte base64-urlsafe Fernet key. |
| `ADMIN_USERNAME` / `ADMIN_PASSWORD` | Initial login credentials |

### Database & cache

| Variable | Description |
|---|---|
| `DATABASE_URL` | AsyncPG connection string |
| `POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB` | PostgreSQL credentials |
| `POSTGRES_APP_USER` / `POSTGRES_APP_PASSWORD` / `APP_DATABASE_URL` | The non-superuser `erezagent_app` role's credentials -- what actually enforces tenant Row-Level Security (see section 2 and section 9). Auto-generated by every installer/upgrade path. |
| `REDIS_URL` | Redis connection |

**Since v4.49.0**, the API refuses to start outside development mode (`APP_ENV` not `development`)
if `POSTGRES_PASSWORD`, `REDIS_PASSWORD`, `GRAFANA_ADMIN_PASSWORD`, `DATABASE_URL`, or `REDIS_URL`
still resolve to the placeholder/known-default values `env.example` ships with. Set real values
before deploying outside development -- an unedited `.env` now fails to start instead of running on well-known credentials.

### AI providers

| Variable | Description |
|---|---|
| `ANTHROPIC_API_KEY` | Primary model provider |
| `OPENAI_API_KEY` | GPT family + embeddings |
| `GOOGLE_API_KEY` | Gemini family |
| `MODEL_PROVIDER_OVERRIDE` | Force `mock`, `anthropic`, `google`, or `ollama` |
| `OLLAMA_BASE_URL` / `OLLAMA_DEFAULT_MODEL` | Local model serving |
| `ANTHROPIC_ADMIN_API_KEY` / `OPENAI_ADMIN_API_KEY` (since v4.77.0) | Separate Admin keys (never the inference keys above) powering real spend visibility on the AI Resources page |
| `XAI_MANAGEMENT_API_KEY` / `XAI_TEAM_ID` (since v4.77.0) | xAI Management API credentials, for a real remaining prepaid balance instead of just spend |

Available Claude models (select in Settings -> AI Model): `claude-fable-5-1`, `claude-fable-5`, `claude-opus-5`,
`claude-opus-5-5`, `claude-sonnet-5-5` (recommended for everyday work), `claude-sonnet-5`, `claude-sonnet-4-6`,
`claude-opus-4-8`, `claude-haiku-5-5` (the fast tier since v4.90.0: fastest and lowest-priced),
`claude-haiku-4-5-20251001`. Gemini and the full GPT lineup -- including new
(v4.61.0) `gpt-6-sol` and `gpt-6-luna` alongside `gpt-6-astra` -- are also available with the
corresponding key set. Model prices, context windows and seeded capability defaults were re-verified against each
vendor in v4.87.0 (for example Claude Fable 5.1 is now priced at its real rate); migration 502 refreshes seeded defaults on
installed systems without changing any value you customized, and Gemini 2.0 models (shut down) are replaced by
`gemini-3.1-flash-lite` defaults. Selecting a new model here is available immediately; it does not change any
agent's existing default model assignment.

### Application

| Variable | Description |
|---|---|
| `APP_ENV` | `development` or `production` |
| `LOG_LEVEL` | `DEBUG` / `INFO` / `WARNING` / `ERROR` |
| `ALLOWED_ORIGINS` | CORS allowlist |
| `API_BASE_URL` | Public API base URL, used for OAuth callbacks |
| `OWNER_NAME` / `OWNER_EMAIL` | Briefing/approval-notification identity |
| `JWT_TTL_HOURS` | Session lifetime (default 168) |
| `DISABLE_AUTH` | Never `true` in production |

### AI budget

| Variable | Description |
|---|---|
| `DAILY_AI_BUDGET_USD` | Per-agent daily cap (default 10.00) |
| `MONTHLY_AI_BUDGET_USD` | System-wide monthly cap -- alerts at 70%/90%, hard-blocks execution at 110% (default 150) |

### Licensing and trial

| Variable | Description |
|---|---|
| `LICENSE_MODE` | `evaluation` (default), `local` (only with a signed license file and public key) or `dev` (development machines only) -- see section 11 |
| `LICENSE_FILE_PATH` / `LICENSE_PUBLIC_KEY` | Signed license file and its verification key. The key defaults to the vendor public key and a blank value behaves the same; set it only to trust a different issuer |
| `TRIAL_TYPE` | `trial_14` or `trial_30` (default `trial_30`; the web installer offers the 30-day trial) |
| `INITIAL_PLAN_CODE` | Since v4.51.0. Which commercial plan the installer prompted for at setup -- a cosmetic starting preference for the Plan & Billing screen, never a binding subscription. See section 14. |

### Security and rate limiting

| Variable | Description |
|---|---|
| (Settings > Security) | Blocking on critical CVEs and on secret leaks is a per-workspace toggle, on by default -- see section 9 |
| `RATE_LIMIT_ENABLED` | Default `true` |
| `RATE_LIMIT_GLOBAL_PER_MINUTE` / `RATE_LIMIT_GLOBAL_PER_HOUR` | Global request caps |
| `OAUTH_REFRESH_LEAD_MINUTES` | How early OAuth tokens proactively refresh (default 15) |

### Observability

| Variable | Description |
|---|---|
| `GRAFANA_ADMIN_PASSWORD` | Change from the default in production |
| `PROMETHEUS_RETENTION_DAYS` | Default 15 |
| `LOKI_RETENTION_HOURS` | Default 168 |
| `SENTRY_DSN` | Optional error tracking |

### Email delivery

| Variable | Description |
|---|---|
| `EMAIL_PROVIDER` | `mock` (default: email is only written to the log, never sent), `sendgrid` or `gmail`. Any other value behaves as `mock` and logs a warning |
| `SENDGRID_API_KEY` | SendGrid key with the Mail Send permission (required for `sendgrid`) |
| `EMAIL_FROM_ADDRESS` / `EMAIL_FROM_NAME` | Sender address (must be verified in SendGrid; the installers accept letters, digits and `._%+-` before the `@` and a dotted domain after it) and display name |
| `GMAIL_CLIENT_ID` / `GMAIL_CLIENT_SECRET` | For `gmail`: also connect the Gmail account in Connectors |

Without a real provider, trial and licence reminders, approval alerts, briefings and password-reset emails are not delivered by email; the
reminders and alerts still appear in the app and on Telegram when it is set up. The installers ask for this as an optional step
(SendGrid key and sender address; the key is never shown or logged), and it can be changed later in `.env` followed by a restart of the
api and worker. The Customer Readiness Check (System Health, Readiness tab) shows an **Email delivery** item: not configured, configured
but not verified, verified (the provider accepted the credentials; no email is sent by the check) or failed. **Platform Mgmt -> Mail Log ->
Send Test Email** sends one real message to `OWNER_EMAIL` and logs the result; with the `mock` provider it reports that nothing was sent, and with `OWNER_EMAIL` empty it says so and sends nothing.

### Voice

| Variable | Description |
|---|---|
| `STT_PROVIDER` | `openai` (`gpt-transcribe`, falling back to `whisper-1`), `voicebox`, or `mock` (the default in `env.example`) |
| `TTS_PROVIDER` | `elevenlabs`, `openai`, `voicebox`, or `mock` (the default in `env.example`) |
| `OPENAI_STT_MODEL` / `OPENAI_STT_FALLBACK_MODEL` | OpenAI speech-to-text model and its one-time fallback (defaults `gpt-transcribe` / `whisper-1`; leave the fallback empty to disable it) |
| `OPENAI_TTS_MODEL` / `OPENAI_TTS_FALLBACK_MODEL` | OpenAI text-to-speech model and its one-time fallback (defaults `gpt-realtime-2.1-mini` / `tts-1`) |
| `ELEVENLABS_API_KEY` / `ELEVENLABS_VOICE_ID` | ElevenLabs TTS |
| `VOICEBOX_API_URL` | Your own self-hosted Voicebox instance |

### Push notifications

| Variable | Description |
|---|---|
| `VAPID_PUBLIC_KEY` / `VAPID_PRIVATE_KEY` / `VAPID_EMAIL` | Web Push -- see section 17 |

### Data retention (since v4.87.0)

| Variable | Description |
|---|---|
| `EVENT_RETENTION_DAYS` | Days of `events` kept (default 90) |
| `AUDIT_LOG_RETENTION_DAYS` | Days of `audit_logs` kept (default 730) |
| `RETENTION_DELETE_BATCH_SIZE` | Rows deleted per committed batch by the daily retention job (default 5000) |
| `RETENTION_REQUIRE_BACKUP_WITHIN_HOURS` | The job runs only if a backup completed within this many hours (default 48) or backups are marked external (section 10) |
| `BACKUP_MANUAL_KEEP_MIN` / `BACKUP_MANUAL_KEEP_DAYS` | Backups with no schedule are kept while among the newest N (default 10) or younger than D days (default 30); the daily prune deletes the rest. `0` for both turns it off (section 8) |

### Business Twin (since v4.88.0)

| Variable | Description |
|---|---|
| `TWIN_MAX_FILES_PER_BUNDLE` | Files allowed in one bundle (default 25) |
| `TWIN_MAX_BUNDLE_MB` | Total size of one bundle in MB (default 200; a single file is also capped at 50 MB) |
| `TWIN_ANALYSIS_COST_CONFIRM_USD` | Estimated analysis cost above which the owner must confirm before analysis starts (default 5.00) |
| `TWIN_SOURCE_RETENTION_DAYS` | Days uploaded originals and extracted text are kept after analysis (default 90; `0` keeps them) |

Also used by this feature: `OLLAMA_BASE_URL` / `OLLAMA_DEFAULT_MODEL` for local-only analysis (section 23). On Kubernetes
the same four keys are in the chart's `configMap` values.

### Deployment and domains

| Variable | Description |
|---|---|
| `DOMAIN` / `ACME_EMAIL` | Public hostname and Let's Encrypt contact -- enables auto-SSL via Caddy |
| `PUBLIC_SERVER_IP` | Required for automated publish and self-service venture custom domains (section 2) |
| `BACKUP_HOST_PATH` / `BACKUP_LOCAL_PATH` | Backup mount configuration -- see section 8 |

For every connector-specific variable (payment providers, infrastructure providers, social
platforms, and the rest), see `env.example` directly -- it's kept current with every release and is
the canonical source, not duplicated here field-by-field.

---

## 16. Background Jobs & Automation

The ARQ worker runs a large, fixed set of scheduled jobs, spread across the hour so that no minute starts more than
ten of them, and runs up to 20 at a time. The ones most relevant to day-to-day administration:

| Job | Schedule | What it does |
|---|---|---|
| `enforce_data_retention` | Daily 04:00 UTC | Purges data past its retention window (section 10) |
| `backup_policy_runner` | Every 5 min | Runs each Backup Center policy when it is due (the default policy is daily): DB, vault, config, and every asset volume (section 8) |
| `run_daily_security_scan_job` | Daily 03:00 UTC | Bandit + pip-audit, upserts findings |
| `run_weekly_pentest_job` | Saturday 04:00 UTC | OWASP passive checks against authorized targets only |
| `governance_trigger_sensor` | Hourly :24 | Evaluates governance triggers, convenes the board automatically when one fires |
| `token_budget_sensor` | Every 30 min | Fires budget warning (80%) / critical (95%) attention signals |
| `compute_trust_scores` | Daily 02:00 UTC | Recomputes agent trust scores from decision accuracy |
| `refresh_customer_health` | Monday 07:02 UTC | Rescores customer health from live incident data |
| `nightly_recommendations` | Daily 02:15 UTC | Scale/pause/shutdown/continue recommendations per venture |
| `run_system_review` | Mon-Fri 06:02-06:10 UTC | Generates self-improvement proposals (cost, prompt quality, budget, security, connector health) |
| `weekly_strategic_lessons_job` | Sunday 06:12 UTC | Extracts institutional-memory lessons from recent decisions |
| `weekly_autonomous_learning_job` | Sunday 07:00 UTC | Scores autonomous rules against real outcomes, adjusts confidence |
| `poll_social_engagement` | Every 10 min | Reads new comments/mentions across connected social accounts, routes leads/support/feedback |
| `morning_intelligence_job` | Daily 06:00 UTC | Generates the Executive Office morning briefing |
| `analyze_business_bundle` | On demand | Reads an uploaded Business Twin bundle after the owner starts analysis |
| `execute_twin_change_set` | On demand | Runs an approved Business Twin build step by step |
| `resume_twin_change_sets` | Every 10 min | Resumes a build that was interrupted (does nothing while the kill switch is on) |
| `detect_business_twin_drift_job` | Every 6 hours | Compares built items with live configuration and raises an attention signal when something was changed outside the Twin |
| `purge_expired_twin_sources` | Daily 03:40 UTC | Deletes uploaded Business Twin files and extracted text past `TWIN_SOURCE_RETENTION_DAYS` |
| `evaluate_time_based_rules_job` | Every 30 min | Evaluates business-record rules that use days-in-stage, days-since-update or days-since-created conditions |
| `rollup_usage_ledger` / `snapshot_capacity_usage` / `backfill_usage_ledger` | Hourly :12 / hourly :17 / daily 02:40 UTC | Write usage and capacity entries to the usage ledger; the backfill repairs the last 7 days |
| `compute_value_share_charges` | Daily 04:50 UTC | Computes value-share charges for active contracts from real evidence; each charge waits for owner confirmation |
| `roll_tenant_subscription_periods_job` / `roll_venture_subscription_periods_job` | Hourly :05 / hourly :10 | Roll subscription billing periods (venture subscriptions also apply cancel-at-period-end) |
| `lapse_unpaid_subscriptions_job` | Daily 02:45 UTC | Moves unpaid gateway-billed subscriptions into a 14-day grace period, then suspends them (entitlement only; data is kept) |
| `apply_due_billing_method_changes_job` | Every 30 min | Applies an approved billing-method change when it is due |
| `retry_approved_invoice_changes_job` | Every 10 min | Applies an owner-approved invoice void or credit note whose apply failed (5 attempts, then an attention signal) |
| `generate_recurring_invoice_drafts_job` | Hourly :25 | Creates recurring invoice drafts (never finalizes them) |
| `deliver_billing_webhooks_job` / `prune_billing_events_job` | Every minute / daily 03:35 UTC | Delivers queued billing webhooks with retries; removes billing events older than 180 days |

Check worker health at a glance: `docker compose logs worker --tail=100`, or the `worker` component
on System Health. Every scheduled job that can fail retries transient errors automatically;
exhausted-retry failures are visible via `GET /api/admin/dead-jobs` (owner-only).

Manually trigger the most common on-demand jobs:
```bash
curl -X POST http://localhost:8000/api/self-improvement/reviews/trigger -H "Authorization: Bearer $EREZAGENT_API_KEY"
curl -X POST http://localhost:8000/api/security/scan -H "Authorization: Bearer $EREZAGENT_API_KEY"
curl -X POST http://localhost:8000/api/exchange-rates/refresh -H "Authorization: Bearer $EREZAGENT_API_KEY"
```

---

## 17. Observability & Monitoring

Prometheus, Grafana, cAdvisor, node_exporter, Loki, and Tempo are all bound to `127.0.0.1` on the
host -- reachable from the server itself (or an SSH tunnel) but not from the open internet, since
none of them sit behind their own authentication. **Since v4.49.0**, the `api` and `frontend`
containers' host ports are loopback-only too, matching this same pattern -- Caddy remains the sole
public ingress for every service, local dev/test access via `127.0.0.1` is unaffected.

### Grafana

`http://localhost:3001` on the server itself, or `https://your-domain/grafana/` from anywhere, via
Caddy. Login: `admin` / `GRAFANA_ADMIN_PASSWORD`.

**Since v4.49.0**, once a public `DOMAIN` is configured, reaching `/grafana/*` through Caddy
requires your existing ErezAgent session -- Caddy checks it via `forward_auth` before ever proxying
to Grafana, and Grafana's own anonymous-access setting is now off too (defense in depth). You no
longer need, and can no longer use, a separate anonymous Grafana login once a domain is set; log
into ErezAgent normally first, then open `/grafana/`.

Pre-built dashboards: System Overview, Agent Workforce, Token & Cost, Business Intelligence, Docker
Health, and a full-text Logs Explorer over Loki.

Alert rules routed to Telegram when triggered: a critical CVE found, the API down for 2+ minutes,
budget at 90% of the monthly limit, a high-risk approval waiting 4+ hours, an elevated worker error
rate, an overdue decision verification, root disk space below 10% free, a mission stuck executing
with no step activity for 2+ hours, or a core service restarting more than twice in 30 minutes.

### Unhandled API errors

Any unhandled exception on the API is now recorded to a real, queryable log instead of only
reaching container logs:
```bash
curl -H "Authorization: Bearer $EREZAGENT_API_KEY" http://localhost:8000/api/admin/api-errors
```
Owner-only, same shape as the background-job dead-letter queue above (`GET /api/admin/dead-jobs`,
section 16). Mark one resolved once
you've looked into it: `POST /api/admin/api-errors/<id>/resolve`.

**Tenant scoping (since v4.76.0):** on a shared/multi-tenant install, this now correctly shows only
your own tenant's attributed errors plus errors with no tenant context yet (an exception in shared
middleware before routing resolves a tenant) -- never another tenant's own errors. Resolving an id
that isn't visible to you (or doesn't exist) now returns 404 instead of a false success.

### Container resource limits

`api`, `worker`, `postgres`, and `redis` each have a real CPU/memory ceiling (`API_CPU_LIMIT`/
`API_MEM_LIMIT_MB` and the matching `WORKER_*`/`POSTGRES_*`/`REDIS_*` pairs in `.env`) so a runaway
agent loop can't consume the whole host. Defaults are sized for the Recommended tier in
`docs/DEPLOYMENT_GUIDE.md`'s system-requirements table -- raise them if you're on the Large tier or
running heavier concurrent-agent workloads, or check `docker inspect <container>` if you ever
suspect a service is being throttled.

### Prometheus

`http://localhost:9090`. Key metric families: `erezagent_agent_executions_total`,
`erezagent_tokens_used_total`, `erezagent_model_cost_usd_total`, `erezagent_pending_approvals`,
`erezagent_security_findings`, `erezagent_connector_health`, `erezagent_worker_job_duration_seconds`,
`erezagent_missions_stuck`, plus standard FastAPI HTTP metrics.

### Loki log search

All services ship structured JSON logs to Loki via Promtail. In Grafana -> Explore -> Loki:
```logql
{service="erezagent-api"} |= "error"
{service="erezagent-api"} | json | agent_id != ""
{service="erezagent-worker"} |= "cron"
```

### cAdvisor

`http://localhost:8080` -- real-time container CPU, memory, network, and disk I/O, also available
in the Docker Health Grafana dashboard. **Since v4.49.0**, it runs with a small set of specific
Linux capabilities instead of full `privileged: true`. **Since v4.93.0**, cAdvisor reads container
information through a filtered socket proxy (`docker-socket-proxy`) on an internal network that
only it shares, and no longer mounts the Docker socket. The proxy answers only four read requests
(a container's details, the daemon's information and version, and the ping) and refuses every
other request, including container logs, file archives, exports, process lists, exec and every
write. A client of the proxy can still read each container's configuration, including its
environment, because the proxy cannot filter what an answer contains; that is why only cAdvisor is
on its network. Promtail does not use the Docker API: it reads the log files Docker writes under
`/var/lib/docker/containers`, which every service labels with its container name (the `tag` log
option). A container started before that option existed shows in Loki once it is recreated
(`docker compose up -d` recreates the stack's containers); the Docker data root must be
`/var/lib/docker`.

cAdvisor starts after the proxy and does not wait for it to be healthy, so an install or upgrade
completes even when the proxy cannot use the Docker socket (for example with strict SELinux on
CentOS/Fedora/RHEL, rootless Docker or a non-standard socket path). In that case `docker compose ps`
shows `docker-socket-proxy` as unhealthy and cAdvisor reports containers by cgroup ID instead of by
name. Check `docker compose logs docker-socket-proxy`, fix the socket access, then run
`docker compose restart docker-socket-proxy`; cAdvisor restarts with it.

Container names in cAdvisor also need the classic `overlay2` storage driver. On Docker 29 and Docker
Desktop with the containerd image store, cAdvisor v0.49.1 may report few or no per-container series
(its upstream issue is closed as not planned, and newer releases need a containerd socket mount that
this stack does not add). The Container Crash Loop alert still fires there: besides the core
services by name it counts restarts of every container by cgroup ID, without naming the container.

### Docker log rotation

Every core service's container logs are now capped and rotated automatically (`json-file` driver,
10MB per file, 5 files kept) instead of growing without limit on the host disk.

### Push notifications

Browser push delivers approval alerts instantly, even when the tab is backgrounded.

1. Generate VAPID keys once (`docker exec erezagent_api python -c "..."` -- see the full snippet in
   `docs/DEPLOYMENT_GUIDE.md` if you don't already have keys).
2. Set `VAPID_PUBLIC_KEY`, `VAPID_PRIVATE_KEY`, `VAPID_EMAIL` in `.env`, restart the API.
3. Settings -> Push Notifications -> **Enable** -> allow the browser permission.

Push is per-device -- enable separately on each browser/device you want alerts on. Supported:
Chrome, Edge, Firefox (desktop + Android); Safari requires iOS 16.4+ with the PWA installed.

The web interface itself needs a current browser: Chrome and Edge 111+, Safari 16.4+, Firefox 128+.

---

## 18. API & Automation Access

Every screen in ErezAgent is backed by a real, documented API -- see `docs/06_API_SPECIFICATION.md`
for the complete endpoint reference. All routes except `GET /health` and provider webhook callbacks
require:
```
Authorization: Bearer <EREZAGENT_API_KEY>
```
Browser-initiated SSE uses a query parameter instead: `/api/stream/sse?token=<EREZAGENT_API_KEY>` (single-tenant installs; on a shared install use a signed-in session).
Interactive Swagger docs are always available at `/docs` on your own deployment.

Common administrative operations you'll script or automate:

```bash
# System status
curl -H "Authorization: Bearer $EREZAGENT_API_KEY" http://localhost:8000/api/system/status

# List pending approvals
curl -H "Authorization: Bearer $EREZAGENT_API_KEY" "http://localhost:8000/api/approvals?status=pending"

# Trigger the morning briefing on demand
curl -X POST -H "Authorization: Bearer $EREZAGENT_API_KEY" http://localhost:8000/api/executive/briefing/generate

# Overall readiness (tenant isolation is covered by the platform-operator diagnostic, section 2)
curl -H "Authorization: Bearer $EREZAGENT_API_KEY" http://localhost:8000/api/system-health
```

Every write endpoint that touches a high-risk action still goes through the same governance model
described throughout this guide -- scripting against the API doesn't bypass approvals.

---

## 19. Troubleshooting

| Problem | Likely cause | Fix |
|---|---|---|
| API returns 500 on connector connect | `CREDENTIAL_ENCRYPTION_KEY` not set | Add the key to `.env` and restart |
| VMware VCD / OpenStack tabs empty | Credentials not configured -- correct production behavior | Set the relevant provider env vars and restart |
| FinOps shows $0 for all providers | No cost records and no billing API connected | Import via `POST /api/finops/costs` or connect a billing API |
| All AI responses are generic | `MODEL_PROVIDER_OVERRIDE=mock` set | Clear the var in `.env` and restart the API |
| Login fails | Wrong credentials or DB not migrated | Check `ADMIN_PASSWORD`; run `alembic upgrade head` |
| Grafana login fails | Password in volume differs from `.env` | `docker exec erezagent_grafana grafana-cli admin reset-admin-password admin` |
| Agents not responding | Worker container down | `docker compose restart worker` |
| SSE events not updating | Redis down or SSE disconnected | Check Redis; look for the stream indicator in the header |
| Telegram notifications not arriving | Bot token not set or wrong chat ID | Check env vars; `curl http://localhost:8000/api/connectors/telegram/status` |
| Obsidian sync fails | Using `localhost` inside Docker | Set `OBSIDIAN_API_URL=http://host.docker.internal:27123` |
| Database migration fails | Conflicting migration state | `docker compose run --rm api alembic current`; restore from backup if needed |
| Budget alert firing immediately | Daily budget set too low | Check `DAILY_AI_BUDGET_USD`; adjust in `.env` or per-agent in Settings |
| Worker cron not running | Worker container not started | `docker compose up -d worker`; check `docker compose ps` |
| Stripe webhook not processing | `STRIPE_ENABLED=false` | Set `true` in `.env` and restart |
| Signature status never updates after sending | `DOCUSIGN_WEBHOOK_SECRET` unset, or Connect subscription not pointed at your domain | Set the secret to match your DocuSign Connect subscription's HMAC key and restart |
| Approval Telegram buttons not working | Webhook URL not configured | Register the Telegram webhook (`setWebhook`) for `https://your-domain/api/connectors/telegram/callback` with the same `secret_token` as `TELEGRAM_WEBHOOK_SECRET`, and set `TELEGRAM_OWNER_CHAT_ID` |
| Frontend can't reach API | CORS or wrong API base URL | Check `ALLOWED_ORIGINS` and `VITE_API_BASE_URL` |
| MSP/FinOps events not in Attention Feed | Sensors run every 30-60 min | Check worker logs; events surface on the next 2-minute routing cycle |

For anything not covered here, generate a **Support Bundle** (Platform Management, section 12) and
review it before reaching out -- it packages diagnostics, config (secrets redacted), and
recommendations in one file safe to share.

---

## 20. Business Operations Runtime Administration

*(since v4.50.0)*

Some solutions your organization takes on aren't just build-and-deploy projects -- they're an
ongoing operational commitment (a pricing floor that must never be silently breached, an SLA credit
policy, a procurement rule). The Business Operations Runtime is what lets ErezAgent manage those
after delivery, not just build them once.

### Business Policies

A Business Policy is a floor, ceiling, guardrail, rate limit, or cooldown that governs a repeated
operational decision. Policies are always created as a **draft** and only ever reach **active**
status after a real, human-approved review -- an AI agent may draft a policy proposal, but it can
never activate one itself. Evaluation is deterministic: missing, stale, or conflicting input always
blocks the decision rather than guessing.

- Review and manage policies under **Business Policies** in the sidebar.
- A policy with a tripped circuit breaker (repeated failed/blocked evaluations) stays blocked until
  it's manually reset, automatically expires (if configured), or goes through a governed reset --
  never silently.

### Segregation of Duties

Some governed actions require **two distinct people** to approve, in a specific order, each holding
the right role -- configured once as an **Approval Chain Template** (Settings > Approval Chains,
owner-only). The same person can never approve two steps of the same chain, and an approval raised
against a policy version is automatically invalidated if that policy changes before it's approved.

### Verified actions and automatic rollback

A governed action taken through the Business Operations Runtime (e.g. publishing a price) is
checked against what the target system actually shows afterward. A mismatch triggers an automatic
rollback and raises a real attention signal -- it never silently reports success on an unverified
claim.

### Connector Builder

Not every third-party system you'll need to connect to is built into ErezAgent ahead of time.
Under **Connectors > Builder** (owner-only), you can define a new connection yourself -- its
address, how it authenticates, and the specific read/write actions it supports -- without waiting
on an engineering release. A new connector starts as a **draft** and must be explicitly activated
before it's used for anything real.

### Operational Workspaces

**Operational Workspaces** (sidebar) is where you monitor and act on every provisioned governed
capability -- its live queue, its full append-only decision history, and anything genuinely
needing your review (a low-confidence entity match, a rejected recommendation). A new workspace is
provisioned automatically when a delivered solution's blueprint declares it needs an ongoing
operational capability.

### What to watch as an administrator

Review Operational Workspaces the same way you'd review the Approvals queue -- a tripped circuit
breaker or a growing exception queue is the fastest sign something needs your attention before it
becomes a customer-facing problem.

## 21. Business Pulse & Operating Doctrine Administration

*(since v4.60.0)*

### Business Pulse

Business Pulse (Executive Office page, below the Chief of Staff digest strip) continuously answers
"what changed, why does it matter, does anything need to happen" -- company-wide, or for one venture
via the venture selector. It's built entirely from your existing Attention Feed signals, classified
into 7 levels from purely informational up through Critical, and shown across three tiles: **Needs
You**, **Already Handled**, and **Watch**.

As administrator, the one thing to understand: an item classified as safe for autonomous action is
now actually dispatched, not just described -- through the exact same approval-gated execution path
every other autonomous action already uses. An item that needs a decision or your approval becomes a
real entry in your task list or Approvals queue; anything serious enough for executive review becomes
a real mission. This runs on a schedule (every 15 minutes) so what you see on the page is already
current, and a signal is never acted on twice.

Dismissing a Business Pulse item dismisses the same underlying Attention Feed signal -- there's no
separate dismiss state to manage. When an item combines several related signals (see below), dismiss
clears all of them, not just one.

**Root-cause correlation and "Already Handled" (v4.66.0):** when a stalled mission's real, most
recent failure traces to a specific connector that's not operational, Business Pulse states that
plainly ("Mission 'Q4 Outbound' is blocked because Gmail is not operational") and shows a direct Fix
link, instead of surfacing several separate, unexplained symptoms of the same problem. This is built
only from data that already links a mission to what actually blocked it -- never a guessed or
probable cause. "Already Handled" now shows a real, specific description of what ErezAgent did
autonomously.

Per-venture **Health** snapshots (linked from each venture's pulse) score revenue, missions, and
customers daily -- a smaller, honest subset of the tenant-wide Company Health score's 8 domains,
since the other 5 (workforce, security, memory, connectors, assets) have no real per-venture data
source today. The snapshot discloses which domains it skipped rather than guessing at them.

### Guided First-Value Journey (v4.67.0)

A small, persistent indicator at the top of Executive Office names the single most important
next step toward a customer's first real, measurable business outcome -- activate the Company
Foundation, create a first venture or objective, connect a capability, convene the Executive
Board, or review a pending approval, each with a direct link. It disappears once a real
outcome is recorded (a real row in the underlying value-tracking table your Strategic
Objectives already write to) -- it never claims value was reached from setup activity alone.

Behind it, a "Time to First Value" ledger tracks nine real milestones (install, first login,
Company Foundation active, first Venture Foundation active, first capability connected, first
Board deliberation, first mission created, first action executed, first measurable outcome).
Each milestone is filled in from your account's own real history the first time it's read --
never fabricated, never stamped with the current time -- and once set, a milestone never
reverts, even if the thing that satisfied it is later changed or removed. This is not a new
onboarding system: every milestone and every next-step recommendation is read from Company
Foundation, Venture Foundation, capabilities, the Executive Board, missions, and approvals --
systems that already exist.

### Executive Office as the Primary Operating Interface (v4.68.0)

Executive Office chat now shares the exact same real actions the command bar (the persistent
bar at the top of the screen) already had: switch venture, approve or reject an approval,
launch a venture, and build a mobile app. Previously only the command bar could do these
without falling back to a generic AI reply -- typing "approve the deployment" into Executive
Office chat now actually approves it, the same way saying it into the command bar already did.

Five new real actions are available the same way: "run a readiness check", "why is Venture X
blocked", and "summarize the company / this venture / this mission" -- each answered from real
existing data (Customer Readiness Check, Business Pulse, Journey, Company/Venture Foundation),
never guessed by the AI. A reply that includes a real next step now shows a clickable button,
not just text. A new "Explain" button on Business Pulse items and the Getting to First Value
indicator asks the same question directly, with one click.

Approval Center and Capability Engine now support opening a specific item directly from a
link (`?id=...`), the same way Mission Control already did -- the item is highlighted and
scrolled into view.

**As administrator, the one thing to understand**: creating a mission or starting a real Board
deliberation from a chat conversation now asks for your approval first, like any other
consequential action. This reuses the same approval mechanism every other approval-gated skill
already uses; nothing new to configure.

### Full Skill Confirmation-Gate Audit (v4.69.0)

Every AI skill was reviewed for whether it requires your approval before it runs. Creating a
project scaffold or generating project code from chat asks for your approval first, the same as
creating a mission or starting a Board deliberation (v4.68.0). Two support skills that only draft
a reply (never commit to a refund or charge) do not ask for approval.

**As administrator, the one thing to understand**: if you use project scaffolding or code
generation from Executive Office chat, expect a real approval step the first time, the same
one you already see for mission creation. Nothing new to configure -- this reuses the
existing Approval Center.

### Project & Customer Ambient Context (v4.70.0)

While viewing a project (Projects) or a customer (Customer Intelligence), you can now ask
Executive Office to summarize it without naming it -- "summarize this project" or "what's
going on with this customer" resolves to whichever one you have open. A project summary
includes real status, health, requirement completion, and deployment state; a customer summary
includes real health score, churn risk, and open support cases. If nothing is open, Executive
Office asks which one you mean instead of guessing.

Projects and Customer Intelligence can now also be linked to directly from a specific item
(`?id=...`), the same way Mission Control and Approval Center already could -- the item is
selected automatically.

**As administrator, the one thing to understand**: nothing new to configure -- this reuses the
same real project and customer data already shown on those pages, and never fabricates a
summary when the data doesn't exist.

### Security cleanup, opportunity diversity, licensing fix (v4.71.0)

Opportunity Discovery (Value Hunter's "Opportunity Radar") now takes your Company Foundation
into account when researching a request, and checks a newly-discovered opportunity against your
own recent ones before adding it to the shortlist -- two independently-worded ideas that turn
out to be the same opportunity now collapse into one, keeping the stronger evidence-backed
version instead of listing both. Approving a market-discovered opportunity now goes through the
same Executive Board review step every other AI-discovered opportunity already required, and its
category label displays correctly instead of blank.

**As administrator, the one thing to understand**: nothing new to configure here either. Local
license files (`LICENSE_MODE=local`) are verified on every start.

### Full production-readiness hardening pass (v4.73.0)

**As administrator, the one thing to understand**: a subscription downgrade you schedule
completes automatically at its renewal date -- nothing to configure. Sending a test
email/Telegram notification, VMware/OpenStack power actions, deleting an Obsidian vault file and
changing a security finding's status are owner actions. Push notifications go to the intended
device. Two admin-only billing/commercial endpoints were added: a one-time professional-services
charge (onboarding, migration, training, etc.) can be billed against a subscription, and a
competitor price observation can be recorded manually so pricing recommendations can reference
it. Routine dependency updates throughout.

### Customer Success venture scoping, Marketing/Social Media reviews, WebMCP opt-in (v4.72.0)

If you run more than one venture under this account, Customer Intelligence and Customer Success
follow whichever venture you have selected consistently; with none selected, you see everything.
Customer Success also now learns from its own churn predictions over time: when a
customer flagged at-risk actually leaves, or one that wasn't flagged leaves anyway, that outcome
is recorded for future reference.

Marketing and Social Media both went through a full review this release with no gaps left open.
Social Media specifically: a LinkedIn credential that needed re-authorizing now refreshes itself
proactively instead of expiring, and you can now ask an agent to check for new social
comments/mentions/DMs right now from chat, instead of waiting for the next automatic check
(every 10 minutes).

Site scaffolding (Customer Solution Delivery) gained an optional WebMCP-annotation flag for
generated sites -- off by default, no behavior change unless you turn it on for a specific
project.

**As administrator, the one thing to understand**: nothing new to configure. If you run
multiple ventures, double-check that Customer Intelligence/Customer Success show what you expect
with a venture selected vs. not.

### Operating Doctrine

**Operating Doctrine** (Strategy & Governance group, right after Autonomous Rules) is where you
define *how* a business, venture, department, agent, or mission should operate -- communication
tone, minimum evidence requirements, research requirements, publishing restrictions, spending
boundaries -- without any code change. A rule set at a more specific level (say, one venture) always
wins over a broader one (the whole company); every resolved rule tells you exactly which level it
came from.

A fixed set of security- and approval-critical rule names -- the minimum security floor, whether
approval gates are required, autonomy tier floors, and similar -- can **never** be set through
doctrine, at any level, by anyone, including the owner role. This is enforced structurally, not as a
setting you could accidentally turn off.

Setting a spending-related doctrine rule that would conflict with a stricter limit already active
higher up (e.g. a venture trying to raise a ceiling the company level has capped lower) is rejected
outright with a clear reason. Every other kind of conflict between levels is recorded as a warning
you can see, but doesn't block the write.

Every doctrine rule is versioned -- review its full history and roll back to any earlier version
from the same page.

---

## 22. Operational App Runtime Administration

*(since v4.85.0, builds on Venture Staff, section 1)*

This is the layer that turns a venture's own operational staff (section 1) and its customers
into real, day-to-day users of Business Records, not just an owner-only feature.

### My Work and the Operational Dashboard

**My Work** (sidebar) and the **Operational Dashboard** (sidebar) are both automatically scoped:
an owner/operator/observer sees every venture they have access to; a venture staff member sees
only the venture(s) they're a member of. Nothing to configure -- scoping follows the same
membership records reviewed in a venture's Staff panel.

The Operational Dashboard's "stalled" threshold (how many days without a stage change counts a
record as stalled) is a disclosed, fixed constant, not a per-tenant setting today -- if you need
it tuned for your process, treat it as a real product request rather than a configuration gap.

### Business Record Customer Portal

Select a record type on **Business Records** and click the globe icon next to Rules to open its **Customer
Portal** panel (owner/operator). Choosing components (record fields, document checklist and upload, appointment
scheduler, the read-only task checklist, signature status and approval status, and since v4.88.0 stage progress,
required documents, a customer form, messages and requests) publishes a new, versioned portal configuration for that type, the same
draft/active/superseded lifecycle as the record type itself -- a change here never edits a
version a customer might currently be looking at; it supersedes it as of the next request. Only
fields you've separately tagged `customer`-visible in the type's field list (the same "visible
to" control used for Owner/Operator/Observer/Staff) ever reach the portal, regardless of what
components you enable. The task checklist and approval status show only tasks flagged
`customer_visible` on the record's Workspace; signature status shows only signature fields tagged
`customer`-visible. Each component's data is served only when the type's active portal
configuration includes it.

A customer reaches their own record only, at a link scoped to their venture and that one record
-- there is no browse or search surface on this portal, and ownership is re-verified on every
request, not trusted from anything in the link itself.

### Executive Office / MCP-reachable Business Record actions

Six Business Record actions -- change stage, request a signature, request documents, schedule
an appointment, send a reminder, and reassign a task -- are now real, registered skills, reachable
identically from Executive Office chat, voice, and any external MCP client you've granted the
relevant skill permission to (Capability Engine / MCP Gateway administration). Granting or
revoking one of these skills to an agent or an external MCP credential controls all three
surfaces at once -- there is no separate chat-only or voice-only permission. Reassigning a task
always creates a pending approval; it's never applied immediately regardless of which surface
requested it.

### Opening customer-submitted files (v4.87.0)

Owners and venture staff (and other users who can open the record) can open a file a customer submitted: **Document Review** has a **View file** button
on each submission, and **Business Records** has a **Download** button on file fields. Access is limited to the user's own
tenant and, for staff, their own venture; files download as attachments and every download is written to the Audit Log.

### Other v4.87.0 notes

- System Health component counts are now computed per tenant.
- The Revenue pipeline can be filtered by venture, and the Revenue Center and Operating Doctrine pages show load errors
  and no longer keep stale data after you switch ventures.
- Domain claiming for a venture's custom domain is owner-only (section 2).
- Background jobs skip a tenant that was deleted mid-run instead of failing.

---

## 23. Business Twin (Build My Business) Administration

*(since v4.88.0)*

**Build My Business** (Executive group) lets the owner give ErezAgent their business documents or a description, review
an evidence-backed model of how the business runs, approve it, and have ErezAgent configure the existing platform
(record types, rules, document checklists, portal pages, workflows, staff roles, appointment types). It never adds a
second workflow engine: every step goes through the same services and approvals as doing it by hand. The owner flow is
in `USER_GUIDE.md`; this section covers what an administrator controls.

### Who can do what

- **Owner:** all changes, including approving a build. Approval is owner-only and tied to the exact version reviewed;
  an approval for an older version is refused.
- **Operator and observer:** read-only access to every step. Venture staff cannot open the page.
- **Executive Office:** anyone with access can ask questions about how the business runs; asking for a *change* is
  owner-only and only drafts a change for review, never applies it. The same two capabilities (explain a process, propose a
  change) are registered skills, so they also appear in Skills Registry and can be granted to an external MCP client.

### AI-analysis consent

**Settings > Data & Privacy > AI analysis of your documents** (owner-only). It is **off by default** and per tenant.
Until it is granted, Build My Business will not analyze anything, and image-based classification or extraction of
customer documents (Document Collection, Business Records) sends the item to the human Review Queue instead
(reason `ai_analysis_consent_required`); a scanned PDF with no text layer goes to the Review Queue as `unreadable`. Revoking consent takes effect immediately. Two modes:

| Mode | Behavior |
|---|---|
| Redacted (default) | Document text goes to your configured AI provider after identifiers are masked: Israeli ID numbers (9 digits with check digit, including unpadded and grouped forms), IBANs, valid payment card numbers, phone numbers, email addresses and secrets (API keys, tokens, private keys). Page images cannot be masked. |
| Local models only | Only privacy-safe (locally hosted) resources are used. If none is available the step fails closed, and if the local model fails during a request the request fails; nothing falls back to an external provider. |

Two tiers apply. **Consent-gated:** Build My Business, AI classification of customer documents (photographed or
scanned pages need consent; text PDFs are read with identifiers masked), field extraction on Business Records, and
Company DNA. **Masked always, no consent needed:** features you start yourself on your own material keep working
without consent, but identifiers are masked before the text reaches the AI model. These are customer requirements
intake (extraction, coverage check, feasibility, blueprint generation), contract analysis, knowledge generation
(contradiction check, runbook, onboarding), document ingest (the text stored as a memory is the masked text), Company
and Venture Foundation extraction from a document, writing-style analysis and meeting-notes outcome extraction. With
**Local models only** selected, these features use a local model, or stop with an error (HTTP 409) when none is
available; nothing falls back to an external provider.

Chat attachments: a document attached to an Executive Office, command bar or Agent Chat message is masked on the
server before any model sees it, and the response reports how many identifiers were masked (counts only). The text you
type is never masked. At most 5 attachments per message (50,000 characters each, 150,000 in total) are accepted. With
**Local models only** selected, attachments are refused (HTTP 409, code `ai_analysis_private_resource_unavailable`)
because the chat agent can reach cloud-backed skills; remove the attachment or change the mode.

Masking is applied when a document is ingested, so knowledge stored from now on holds masked identifiers; the original
file in the vault is unchanged.

Limits of masking: Israeli ID numbers are recognized, other countries' ID formats and passport numbers are not; bank
account numbers are recognized only as IBANs; personal names and free-text addresses are not detected. Memories created
before v4.88.0 are not re-masked. Notes synced from an Obsidian vault are embedded by the external embedding provider as
written, and in Local models only mode the embedding for ingested documents is generated locally and is not semantic.
Granting and revoking consent is written to the Audit Log.

### Local-only mode and the local vision model

Local-only analysis needs a reachable Ollama server (`OLLAMA_BASE_URL`) with a text model (`OLLAMA_DEFAULT_MODEL`). Images
and scanned PDF pages need a local **vision** model: run

```bash
ollama pull qwen2.5vl
```

on the Ollama host. When Ollama is healthy and that model is present, a local vision resource is registered
automatically and shows as privacy-safe on **AI Resources**; use **Test** there to confirm. Without it, scanned or image
sources in local-only mode are skipped with a stated reason rather than sent elsewhere. Quality on Hebrew documents and
with local vision models varies by model; review the evidence labels before approving.

### Cost and limits

Before analysis the owner sees an estimate (computed locally, no model call) and must confirm it above
`TWIN_ANALYSIS_COST_CONFIRM_USD`. Intake limits come from `TWIN_MAX_FILES_PER_BUNDLE` and `TWIN_MAX_BUNDLE_MB`. Business
Twin model calls are routed by six editable task types (`twin_source_extraction`, `twin_consolidation`,
`twin_regenerate`, `twin_conversation_discovery`, `twin_change_translation`, `twin_board_review`) under
**AI Model Routing & Cost**, and every call is token-tracked like any other.

### Builds, approval and the kill switch

Submitting a build creates **one** approval (the owner approval chain) in the Approvals queue. After approval each step
runs as a governed execution run (`business_twin.apply_step`) with read-back verification. A build interrupted by a
restart resumes automatically (every 10 minutes); the emergency kill switch (section 5) pauses the build and its resume
job. Connectors a build depends on are never activated automatically; the preview lists their readiness and the owner
must accept any gaps. Rollback is a new change that goes through the same approval, and shows beforehand what cannot be
restored.

### Drift and notifications

Every 6 hours ErezAgent compares built items with the live configuration. Anything edited outside the Twin is marked
*drifted* and raises an Attention Feed signal; nothing is reverted automatically. Events raised: analysis completed or
failed, approval pending, apply failed, applied and drift detected. They reach the Attention Feed and the live event
stream for your tenant only.

### Templates

An owner can promote parts of a model to a Solution Template. Promotion removes identifiers and customer-specific data
automatically and the owner acknowledges a sanitization report before publishing. Templates provided with the platform are
readable by every tenant and cannot be changed by a tenant. Installing a template only creates a draft version for the
chosen venture.

### Storage, backups and isolation

Uploaded originals live under `twin-sources/` inside the vault volume, so the existing backup covers them; on Kubernetes
that is the shared `<release>-vault` volume (section 14). All Twin data is tenant- and venture-scoped. Executive Office,
agents and Board reviews receive only a short, capped summary of the **active, approved** model for the venture in question.

### Troubleshooting

| Symptom | Likely cause | Fix |
|---|---|---|
| Analyze stays disabled | Consent not granted, no files, cost not estimated or confirmed, or not the owner | Follow the list shown under the buttons |
| Analysis fails in local-only mode | No privacy-safe model reachable, or no local vision model for images | Check Ollama on AI Resources; `ollama pull qwen2.5vl` |
| A file shows "Could not be read" | Unsupported format (.doc, .xls, .ppt, Outlook .msg) | Re-save as .docx/.xlsx/.pptx or export as .eml |
| Build shows "Waiting for your approval" | Owner has not approved it | Approvals page; only an owner can approve |
| Build stopped part-way | A step failed, or the kill switch is on | Open the change in Build, read the failed step, **Resume build** |
| Upload rejected as too large | Bundle or file limit reached | Raise `TWIN_MAX_BUNDLE_MB` / `TWIN_MAX_FILES_PER_BUNDLE`, or split into bundles |

## 24. Provider / Managed Service Administration

If a provider manages ErezAgent for you, you can let it see the operational state of this installation and tell you when
an update exists. This is optional, off until you switch it on, and the provider cannot change anything on your server:
updates, backups and every other change stay yours to perform.

### Connecting

1. The provider gives you an enrolment string that starts with `ea-enroll-v1.`. It is valid for 7 days and works once.
2. Open **Settings > Provider / Managed Service**, paste the string and select **Check enrolment string**. The page
   shows the address that would receive the report. Confirm it is your provider's, read the two lists (shared and never
   shared), tick the consent box ("Send this report to ...") and select **Connect**. Only an owner can do this, and only
   from a signed-in account (not the static API key). On a shared installation it is the instance administrator's owner.
3. The installation registers with the provider and sends its first report. The page then shows the provider's name, the
   managed-install ID, the last time a report was accepted, the support contact, the package name the provider assigned
   and the latest version the provider knows.

There is no default provider, no environment variable and no background connection: nothing is sent before you connect.
The provider's own installation cannot be enrolled as a customer, and an installation cannot be enrolled with its own
address.

### What is shared

About every 15 minutes the installation sends one report: the application and database version and the number of
workspaces; ok, degraded or down for the core services; the security summary of the installation itself (state, open
critical and high counts, age of the last scan); the status of each AI and voice provider; the number of connected
connectors, connectors that need re-authorising and connectors in error; the readiness state; disk and memory flags; and
the licence and subscription state. **Show what is reported** displays the exact report last sent and what the next one
would contain.

Never shared: prompts, model responses, missions or any business data, documents, e-mail, CRM or customer data, names of
people, companies, workspaces, domains, ventures or projects, credentials, API keys, account labels, voice profile names,
log lines or error text.

### Pause, resume, disconnect

**Pause reporting** stops sending at once; **Resume reporting** continues. **Disconnect** (after a confirmation) stops
reporting, deletes this installation's key for the provider, removes the received notices and tells the provider that
the installation was disconnected, so it stops accepting reports for it; if the provider cannot be reached the
disconnect still completes. To connect again you need a new enrolment string. Each of these, and each answer to a
notice, is recorded in the audit log with your account.

If a later version changes the list of what is shared, reporting pauses and the page asks you to read the updated list
and confirm it before it resumes. The connection is tied to the installation it was made on: if the database is restored
from a backup or copied to another server, reporting stays stopped with a message, and you connect again with a new
enrolment string from your provider.

### Notices

The provider can send notices (an update, a security update, information or a service offer). They appear as plain text
under **Notices from your provider** and, for owners, in a strip at the top of every page. Answer each with **I will
update**, **Remind me later** (the notice returns after 7 days), **Request assistance** or **Request professional
services**; the last two open a support case on the provider's side. Nothing happens on your server because of a notice.

### If reports fail

The page shows the last problem in short words (for example `timeout`, `unauthorized` or `http 500`). After three failed
reports in a row the state reads "Connected, last reports failed" while the installation keeps retrying with growing
pauses, up to 6 hours; the first accepted report clears it, and **Resume reporting** retries at once. `unauthorized`
usually means the provider revoked the enrolment. A message that the clock differs from the provider's by a number of
minutes means the time on this server needs correcting. Outbound HTTPS from
the api and worker containers to the provider's address is required (see the Deployment Guide).

---

*© 2026 Erez Rozenbaum. All Rights Reserved.*