# ANC CRM — Rules of Engagement for AI Agents

Everything below was paid for with a production incident, a wedged chat, a wrong number in a
stakeholder's inbox, or a day of someone's time. Read before touching `crm.ancsports.net`.

Canonical living source: the `crm-knowledge` skill at
`/root/.claude/skills/crm-knowledge/SKILL.md`. Read it before, update it after.

---

## 0. Before you touch anything

- **Three separate Twenty instances. Never cross-contaminate.**
  - ANC live — `crm.ancsports.net`, workspace `d3fbc29a-a635-48b7-9d6e-250941677fd0`
  - `crm.basheer.app` — deprecated stale clone, do not build here
  - `manage.basheer.app` — Ahmad's personal CRM, workspace `be2f5450…ccd8`
- **No forking the source, no custom code in core.** Three sanctioned paths only: the API
  (GraphQL `/graphql` for data, `/metadata` for metadata), direct DB when the API is genuinely
  blocked, and Twenty Apps (`twenty-app-builder`) for UI behaviour the platform doesn't expose.
- **Salesforce is read-only and the contract ended 2026-04-30.** Reconcile against the frozen
  archive at `/root/sf-archive/2026-05-29/` (606 objects, 592k records), not live SF.
- **CRM-only work needs no deploy.** Don't push, don't touch EasyPanel. Make the change in the
  CRM and verify it there.
- **Snapshot before any bulk or destructive action.** `/root/twenty-backups/` is the house
  location. Keep the restore list in the same commit as the change.

---

## 1. The cache is the enemy — there are three layers

Twenty caches metadata in **three** places: an in-process local cache (30 min TTL), Redis
flat-maps, and the DB. A correct DB row means nothing if the layer in front is stale.

- **GraphQL mutations write through and bust their own cache. Direct SQL does not.**
  Prefer the mutation every time for an UPDATE to an existing row. Raw SQL is fine for
  inspection and for INSERTing new rows (no cache to invalidate).
- `pageLayoutWidget` config **must** go through `updatePageLayoutWidget` — a direct SQL UPDATE
  served the old chart config for hours while the DB was correct.
- `core.viewField` DB updates are served stale on v2.17 — use `updateViewField`.
- **Role and permission-flag changes need a restart:**
  `docker service update --force --update-order start-first abc_twenty` (~30–60s, start-first
  ≈ no downtime). A Redis bust alone will not drop the in-process permission cache.
- **RLS predicate changes do NOT need a restart** — DEL the two flat-map keys and INCR
  `engine:workspace:metadata:workspace-metadata-version:<ws>`.
- **Verify against the endpoint the frontend actually reads** (`getViewFields`,
  `getPageLayout`, `findManyAgents`), never against the DB row you just wrote. If the two
  disagree, look at `overrides` next — see §2.
- Redis requires AUTH and the password is not in the redis container's env. Run a small node
  TCP client **inside `abc_twenty`** using its own `REDIS_URL`.

---

## 2. The `overrides` jsonb trap

Any entity seeded by `twenty-standard-application` (`viewField`, `viewFieldGroup`, `viewFilter`,
`viewSort`, `viewGroup`, `view`, `dashboard`, `pageLayout`) has an `overrides` JSONB column, and
the DTO converter **spreads `overrides` OVER the column values**:

```js
const { createdAt, updatedAt, deletedAt, overrides, ...rest } = flatEntity;
return { ...rest, ...overrides ?? {} };   // overrides win
```

So `UPDATE ... SET position = -1` is a **silent no-op** when `overrides->>'position'` is also
set. Update both in lockstep, or clear `overrides` entirely to fall back to columns:

```sql
UPDATE core."viewFieldGroup"
SET position = N,
    overrides = jsonb_set(COALESCE(overrides,'{}'::jsonb), '{position}', to_jsonb(N))
WHERE id = '<uuid>';
```

Same lens applies to `fieldMetadata.standardOverrides`.

---

## 3. Standard vs custom metadata

- `updateOneField` **cannot** change the label on a STANDARD field — it silently no-ops. Write
  `core."fieldMetadata"."standardOverrides"` JSON directly.
- Standard **object** properties (`labelIdentifierFieldMetadataId`, `isLabelSyncedWithName`) are
  blocked by the API. DB column + `standardOverrides` — set both.
- `createOneField` rejects `isCustom` on current builds. Drop it.
- TEXT `defaultValue` must be postgres-quoted: `"''"`, not `""`.
- Composite defaults are quoted strings too:
  `{primaryPhoneNumber:"''", primaryPhoneCallingCode:"'+1'", primaryPhoneCountryCode:"'US'"}`.
- **A PHONES field with a NULL `defaultValue` silently corrupts US numbers** — the leading
  digits get parsed as a country calling code (856 → Laos, 212 → Morocco, 914 → India). Always
  set the default at field-creation time.
- Converting a TEXT field to a RELATION: delete and re-create with the same name
  (`deleteOneField` also removes its viewField references).
- The GraphQL data endpoint **returns null for unknown field names instead of erroring** — a
  successful query is not proof the field exists.

---

## 4. Changing a SELECT / enum option

Adding one option rebuilds the whole postgres enum (rename column → drop). **Check dependencies
first — do not assume:**

```sql
-- columns still on the enum type (incl. stale backup tables)
SELECT n.nspname||'.'||c.relname||'.'||a.attname FROM pg_attribute a
  JOIN pg_class c ON c.oid=a.attrelid JOIN pg_namespace n ON n.oid=c.relnamespace
  JOIN pg_type t ON t.oid=a.atttypid
 WHERE t.typname LIKE '%<fieldName>%' AND a.attnum>0 AND NOT a.attisdropped;
-- triggers naming the field
SELECT tgname FROM pg_trigger t JOIN pg_class c ON c.oid=t.tgrelid
 WHERE NOT tgisinternal AND pg_get_triggerdef(t.oid) ILIKE '%<fieldName>%';
```

Then: DROP the dependent trigger → detype backup-table columns to `text` → run
`updateOneField` → recreate the trigger verbatim. Back up the options array first.

- **Twenty AUTO-CREATES a viewGroup** on every grouped view when you add an option. If you also
  create one manually you get a duplicate — delete the auto row, or let it auto-create and just
  move it.
- **Audit every viewFilter that whitelists that field.** An `IS [whitelist]` filter silently
  drops records at the new option (Natalia's queue lost 15 deals this way). An
  `IS_NOT [blacklist]` filter leaks the new option in. Both need a pass.

---

## 5. Views

- **The label-identifier (`name`) viewField must be VISIBLE at the lowest position (`-1`).**
  Hiding it desyncs the frozen record-name column from the headers → the classic off-by-one
  "duplicate Game Date" column shift. It cannot be hidden, only narrowed or relabelled. Legacy
  views with it hidden are NOT a safe template.
- **Never sort a grouped TABLE view by a nullable field.** Cursor pagination encodes
  `{sortField, id}` and cannot advance past a null cluster — "Load more" returns 0 rows with
  records clearly remaining. Sort by `name` (non-null, unique).
- **Paged reads MUST `orderBy: {id}`.** A `createdAt` cursor silently skips and repeats — one
  audit returned 143 rows containing only 90 unique ids out of 203.
- **No two views with the same name on the same object.** That, not visibility scoping, is why
  users report "everyone sees a different view."
- Footer totals and per-group subtotals both come from `viewField.aggregateOperation` — one
  flag, two render targets.
- Field labels truncate at the END in the record rail. Two fields sharing a leading phrase
  ("Total Project …") become indistinguishable and people type into the wrong one. Name money
  fields `<Revenue|Margin> — <qualifier>`.
- Twenty auto-creates a viewField for a new field on FIELDS_WIDGET views only, never on
  TABLE/KANBAN. On those you must `createViewField` explicitly.
- `updateViewField` takes **one update key per call** — multi-key updates silently truncate.
  Position before group.
- "Can't enter a value for X" on a record page is almost always a hidden viewField, not a
  missing field. Check `isVisible` before building anything.

---

## 6. Permissions — the counter-intuitive ones

- **There is no separate CREATE permission.** The server gates `insert` on
  `canUpdateObjectRecords`. A role that must create records needs update, not just read.
- **Merge requires `canDestroyObjectRecords`** on that object (merge soft-deletes the loser).
  Single Delete only needs soft-delete. That's why a role can see Delete but not Merge.
- **Note/task creation validates READ on every polymorphic `noteTarget`/`taskTarget` target**
  (~55 `target<Object>Id` columns). One denied object breaks note creation on *every* record
  type, surfacing only as "An error occurred." 30 of 32 users were affected for three months.
- **`executeOneLogicFunction` is gated by the `WORKFLOWS` permission flag.** Admins pass only
  because `canUpdateAllSettings` bypasses the guard — every front-component AI panel breaks for
  everyone else until the flag is granted.
- **RLS predicates crash on NULL.** `DOES_NOT_CONTAIN` against a null title throws
  `Cannot convert undefined or null to object`, and since the INSERT already committed you get
  an orphan blank record. Shape the group as
  `OR( title IS_EMPTY , AND( <the real predicates> ) )`.
- **RLS is NOT applied to API-key contexts.** Never test an RLS change with an API key — it
  reads straight through and the rule will look broken when it isn't. Test with a real session.
- To test what a role can do without touching a person's account: mint a role-scoped API key
  (`core."apiKey"` + `core."roleTarget"` with `apiKeyId`, JWT signed with
  `sha256(APP_SECRET + workspaceId + 'API_KEY')`) — remembering the RLS caveat above.

---

## 7. The AI layer (models, Scout, chat)

- **Three gates must agree before a model is usable.** Env alone does nothing.
  1. `AI_PROVIDERS` env on **both** `abc_twenty` AND `abc_twenty-worker` — the registry.
  2. `core.workspace."enabledAiModelIds"` (text[]) — the per-workspace allow-list = what the
     picker renders. Same row: `smartModel` / `fastModel` / `routerModel`.
  3. `core."keyValuePair"` key `AI_MODEL_PREFERENCES` — `disabledModels` (admin kill-switch)
     plus `defaultSmartModels` / `defaultFastModels` / `recommendedModels`.
  `updateWorkspace` is user-context only ("API keys are not supported"), so gate 2 is a DB write.
- **Never bulk-prune `AI_PROVIDERS`.** `getEffectiveModelConfig` THROWS `Model with ID X not
  found` for anything missing from the registry, and `validateModelAvailability` throws when a
  model isn't workspace-enabled. Removing a provider while an agent's `modelId`, the workspace
  default, or a preference default still names one of its models wedges every thread —
  `activeStreamId` stays set, no assistant message, and the vendor endpoint streams fine under
  curl the whole time. **Order: repoint agents + workspace defaults + preferences FIRST (DB, no
  restart), then remove providers from env.**
- **Diagnostic tell:** a healthy thread clears `core."agentChatThread"."activeStreamId"` to NULL
  when the stream completes. Still set = wedged. The CRM logs nothing for these failures.
- **Probe a model with a realistic prompt and ~400 max_tokens**, never `max_tokens: 5` —
  reasoning models spend the whole budget thinking and return empty content, which reads as a
  dead model when it's fine. Several models return **HTTP 200 with empty content**: that is the
  "spins forever" failure, and it does not show up as an error anywhere.
- z.ai coding-plan keys are served from `https://api.z.ai/api/coding/paas/v4`. The standard
  `.../api/paas/v4` returns `429 code 1113 "Insufficient balance"` on the very same key.
- `databaseEventTriggerSettings.eventName` is a **single string** — "fire on create AND update"
  needs two sister logic functions. A create event has no `before` state, so any
  transition guard must handle that or it fires on every bulk import.
- Updating a logic function's source sets `isBuildUpToDate=false`; the build is triggered by the
  next `executeOneLogicFunction`. Fire it once, then verify.
- Wrap every logic function body in try/catch returning `{stage, error, stack}` — otherwise the
  executor masks the real failure with the i18n string `"Could not find flat entity in maps"`.
- `executeOneLogicFunction`'s result field is **`data`**, and its input type is
  `ExecuteOneLogicFunctionInput`. Functions hard-timeout at **90s** — budget every fetch with
  `AbortSignal.timeout()` below that.
- **Scout routes; it always needs a real tool.** When you add a skill, field, or view, update
  Scout's system prompt in the same pass, and mirror new skills to the @ANC Slack bot.
- **Currency reaches the model as micros.** Ad-hoc AI workbooks come back 1000× high. Route any
  number a human will read to a `/api/render/*-xlsx` endpoint instead.

---

## 8. Data — the rules that prevent wrong numbers

- **NEVER match CRM ↔ Salesforce by name.** Always `opportunityNumber == Opportunity_ID__c`.
  Renewal-style deals ("…2021-26" vs "…2026-29") collide on every fuzzy scheme; 11 deals were
  wrongly marked Closed Won this way and a stakeholder caught it before we did.
- **NEVER link venues or events by name.** Use `servicesId ↔ venues.id`. A `%M&T%` search
  returns 11 Quinnipiac hockey games at M&T Bank *Arena* in Connecticut — nothing to do with the
  Ravens.
- **Money is `*AmountMicros`.** Convert once, in one helper. Clear a CURRENCY field with
  `{amountMicros:null, currencyCode:null}` — passing whole-object `null` is silently ignored.
- **`amount` ≠ `totalProjectRevenue` ≠ `dealValue`.** `amount`/`margin` are the FY rollup,
  `totalProject*` mirror SF Actual, `dealValue` mirrors SF Sale_Price. Anything claiming to
  mirror the Slack win alert reads `totalProjectRevenue`/`totalProjectMargin`. They routinely
  disagree, and picking the wrong one puts a wrong number in front of finance.
- **A derived number a human watches must be computed synchronously — use a DB trigger, not an
  async logic function.** Event-driven rollups fire once per save and write the running total so
  far, so someone typing year-by-year watches the total step 500k → 1m → 1.5m, and a dropped
  event leaves it permanently short.
- **DB-direct status writes leave no timeline entry** and stakeholders notice. For any
  stakeholder-visible field change on a live record, go through the REST/GraphQL data API so
  `timelineActivity` is written. Reserve raw SQL for metadata and bulk backfills.
- **Soft-delete by default** on anything bulk; keep the id list for `restore*`.
- **Never fabricate.** No invented inventory, pricing, or "reasonable" defaults. If the real
  value doesn't exist, say so and ask — an empty field is honest, a plausible one is a landmine.
- **When a stakeholder says a record is wrong and a recent batch touched it, audit the WHOLE
  batch immediately by exact key.** One wrong row means the matching method was wrong.

---

## 9. Apps and front components

- **Never `window.open`, `window.location.href`, or `target="_blank"`.** Front components run in
  an iframe sandboxed without `allow-popups-to-escape-sandbox`, and `location` is read-only in
  the Worker context (`Cannot set property href of WorkerLocation`). Use
  `navigate(AppPath.RecordShowPage, …)` from `twenty-sdk/front-component`. For same-origin
  parent nav, `(window.top ?? window.parent).location.href` works. Cross-origin downloads need a
  plain `<a href>` or a blob URL — not a scripted open.
- **Pin `twenty-sdk` / `twenty-client-sdk` to the version the live server runs.** `latest`
  breaks install with `installApplication must not have a selection`.
- **The runtime pageLayout id is NOT the manifest UUID** — the manifest one 404s. Read the
  runtime id after install.
- App install busts its own metadata cache — no Redis bust, no restart, **no EasyPanel deploy**.
- `cp -r` of an app dir breaks yarn PnP (keyed to the old package name). Run
  `corepack yarn install` in the new dir.
- `createPageLayoutWidget` needs `type` (`GRAPH`/`FIELD`/…) **in addition to**
  `configuration.configurationType`, and it **silently drops `configuration.filter` on
  AGGREGATE_CHART widgets** — create it, then set the full configuration via
  `updatePageLayoutWidget`, then read the rendered number back and reconcile against SQL. An
  unfiltered KPI tile looks completely plausible.
- Widget ids must be real RFC 4122 UUIDs (`uuid_generate_v4()`), and `position` — not just
  `gridPosition` — must be set with `layoutMode: "GRID"`, or the whole dashboard 500s.
- **One RECORD_PAGE layout per object.** Twenty serves the newest and there is no isDefault
  flag; a second layout silently wins and a Fields widget on a non-primary tab spins forever.
  Edit the existing Default layout instead.

---

## 10. Env-var safety (hard rule)

A previous AI session wiped every env var on a production service with a write-only "add". The
only sanctioned pattern is **read → merge → verify → write**:

1. Snapshot the full env to `/root/.env-snapshots/<svc>-<ts>.json` first.
2. Merge; never full-replace.
3. Diff and assert the result is a strict superset — no var lost, only the intended key changed.
4. Write with `docker service update --env-add "KEY=VALUE" --update-order start-first <svc>`.
5. Re-read and confirm.

**CRM env changes must be applied to BOTH `abc_twenty` and `abc_twenty-worker`.** Each write
restarts the container: ~60s of 502s. Plan for it.

---

## 11. Infrastructure facts

- `abc_twenty` runs a **hand-tagged custom image** (`anc/twenty-v2170-anc-*`), not an EasyPanel
  git build. Frontend behaviour is patched by injecting scripts into `index.html`. A future
  EasyPanel "deploy" can revert it — always check
  `docker service inspect abc_twenty --format '{{.Spec.TaskTemplate.ContainerSpec.Image}}'`
  before diagnosing a regression.
- **`anc-crm-guard.service` pins web+worker to the approved image id** and auto-reverts drift.
  Approved upgrades go through `/root/anc-crm-work/crm-update/anc-crm-update.sh`, which opens and
  reseals a time-limited guard window. A slow boot that outruns that window makes the guard
  revert a perfectly healthy cutover.
- DB access: `docker exec <abc_twenty container> sh -c "psql \$PG_DATABASE_URL -c '<sql>'"`.
  Workspace schema is `workspace_cjspnkm8glh7iooo1gep8c1qo`. Pipe SQL from a file for anything
  containing JSON — nested shell quoting mangles it.
- **The DB is ahead of the image**: `core."searchFieldMetadata"."tsVectorFieldMetadataId"` is NOT
  NULL but the running server never sets it, so `createOneObject` fails workspace-wide. The
  `anc_fill_search_field_tsvector_trg` trigger unblocks it — confirm it exists before installing
  any app that adds an object.
- API rate limit is 100 calls / 60s shared across REST and GraphQL; bulk updates cap at 200
  records, creates at 50-row chunks, per-record PATCH paced ~1.5/s with backoff on 429.
- Nightly backup crons have twice PANICked the CRM's postgres with "No space left on device"
  around 9:30 PM ET. If the CRM is flapping late evening, check disk before anything else.

---

## 12. Verification — the part that is not optional

- **Log in and look.** Use the `crm-visual-verify` skill (email+password auth is enabled; creds
  at `/root/.crm-creds`). An API 200 does not mean it rendered. A DB row does not mean the user
  sees it.
- **Reconcile every number against direct SQL** before it reaches a stakeholder. An unfiltered
  aggregate is the most convincing wrong answer there is.
- **Exercise the path the way that person would** — click their button, open their export, read
  the file. Not the endpoint. The artifact.
- **A stakeholder message goes out only after that.** "Should work" is not a state.
- **Update the `crm-knowledge` skill in the same session.** If it isn't written down it will be
  rediscovered the expensive way.

---

*Compiled 2026-08-20 from the `crm-knowledge` skill, project memory, and the incident record.*