ANC CRM — Rules of Engagement for AI Agents
Everything below was paid for with a production incident, a wedged chat, a wrong number in a stakeholder's inbox, or a day of someone's time. Read before touching crm.ancsports.net.
Canonical living source: the crm-knowledge skill at /root/.claude/skills/crm-knowledge/SKILL.md. Read it before, update it after.
0. Before you touch anything
- Three separate Twenty instances. Never cross-contaminate.
- ANC live —
crm.ancsports.net, workspaced3fbc29a-a635-48b7-9d6e-250941677fd0 crm.basheer.app— deprecated stale clone, do not build heremanage.basheer.app— Ahmad's personal CRM, workspacebe2f5450…ccd8- No forking the source, no custom code in core. Three sanctioned paths only: the API (GraphQL
/graphqlfor data,/metadatafor metadata), direct DB when the API is genuinely blocked, and Twenty Apps (twenty-app-builder) for UI behaviour the platform doesn't expose. - Salesforce is read-only and the contract ended 2026-04-30. Reconcile against the frozen archive at
/root/sf-archive/2026-05-29/(606 objects, 592k records), not live SF. - CRM-only work needs no deploy. Don't push, don't touch EasyPanel. Make the change in the CRM and verify it there.
- Snapshot before any bulk or destructive action.
/root/twenty-backups/is the house location. Keep the restore list in the same commit as the change.
1. The cache is the enemy — there are three layers
Twenty caches metadata in three places: an in-process local cache (30 min TTL), Redis flat-maps, and the DB. A correct DB row means nothing if the layer in front is stale.
- GraphQL mutations write through and bust their own cache. Direct SQL does not. Prefer the mutation every time for an UPDATE to an existing row. Raw SQL is fine for inspection and for INSERTing new rows (no cache to invalidate).
pageLayoutWidgetconfig must go throughupdatePageLayoutWidget— a direct SQL UPDATE served the old chart config for hours while the DB was correct.core.viewFieldDB updates are served stale on v2.17 — useupdateViewField.- Role and permission-flag changes need a restart:
docker service update --force --update-order start-first abc_twenty(~30–60s, start-first ≈ no downtime). A Redis bust alone will not drop the in-process permission cache. - RLS predicate changes do NOT need a restart — DEL the two flat-map keys and INCR
engine:workspace:metadata:workspace-metadata-version:<ws>. - Verify against the endpoint the frontend actually reads (
getViewFields,getPageLayout,findManyAgents), never against the DB row you just wrote. If the two disagree, look atoverridesnext — see §2. - Redis requires AUTH and the password is not in the redis container's env. Run a small node TCP client inside
abc_twentyusing its ownREDIS_URL.
2. The overrides jsonb trap
Any entity seeded by twenty-standard-application (viewField, viewFieldGroup, viewFilter, viewSort, viewGroup, view, dashboard, pageLayout) has an overrides JSONB column, and the DTO converter spreads overrides OVER the column values:
const { createdAt, updatedAt, deletedAt, overrides, ...rest } = flatEntity;
return { ...rest, ...overrides ?? {} }; // overrides win
So UPDATE ... SET position = -1 is a silent no-op when overrides->>'position' is also set. Update both in lockstep, or clear overrides entirely to fall back to columns:
UPDATE core."viewFieldGroup"
SET position = N,
overrides = jsonb_set(COALESCE(overrides,'{}'::jsonb), '{position}', to_jsonb(N))
WHERE id = '<uuid>';
Same lens applies to fieldMetadata.standardOverrides.
3. Standard vs custom metadata
updateOneFieldcannot change the label on a STANDARD field — it silently no-ops. Writecore."fieldMetadata"."standardOverrides"JSON directly.- Standard object properties (
labelIdentifierFieldMetadataId,isLabelSyncedWithName) are blocked by the API. DB column +standardOverrides— set both. createOneFieldrejectsisCustomon current builds. Drop it.- TEXT
defaultValuemust be postgres-quoted:"''", not"". - Composite defaults are quoted strings too:
{primaryPhoneNumber:"''", primaryPhoneCallingCode:"'+1'", primaryPhoneCountryCode:"'US'"}. - A PHONES field with a NULL
defaultValuesilently corrupts US numbers — the leading digits get parsed as a country calling code (856 → Laos, 212 → Morocco, 914 → India). Always set the default at field-creation time. - Converting a TEXT field to a RELATION: delete and re-create with the same name (
deleteOneFieldalso removes its viewField references). - The GraphQL data endpoint returns null for unknown field names instead of erroring — a successful query is not proof the field exists.
4. Changing a SELECT / enum option
Adding one option rebuilds the whole postgres enum (rename column → drop). Check dependencies first — do not assume:
-- columns still on the enum type (incl. stale backup tables)
SELECT n.nspname||'.'||c.relname||'.'||a.attname FROM pg_attribute a
JOIN pg_class c ON c.oid=a.attrelid JOIN pg_namespace n ON n.oid=c.relnamespace
JOIN pg_type t ON t.oid=a.atttypid
WHERE t.typname LIKE '%<fieldName>%' AND a.attnum>0 AND NOT a.attisdropped;
-- triggers naming the field
SELECT tgname FROM pg_trigger t JOIN pg_class c ON c.oid=t.tgrelid
WHERE NOT tgisinternal AND pg_get_triggerdef(t.oid) ILIKE '%<fieldName>%';
Then: DROP the dependent trigger → detype backup-table columns to text → run updateOneField → recreate the trigger verbatim. Back up the options array first.
- Twenty AUTO-CREATES a viewGroup on every grouped view when you add an option. If you also create one manually you get a duplicate — delete the auto row, or let it auto-create and just move it.
- Audit every viewFilter that whitelists that field. An
IS [whitelist]filter silently drops records at the new option (Natalia's queue lost 15 deals this way). AnIS_NOT [blacklist]filter leaks the new option in. Both need a pass.
5. Views
- The label-identifier (
name) viewField must be VISIBLE at the lowest position (-1). Hiding it desyncs the frozen record-name column from the headers → the classic off-by-one "duplicate Game Date" column shift. It cannot be hidden, only narrowed or relabelled. Legacy views with it hidden are NOT a safe template. - Never sort a grouped TABLE view by a nullable field. Cursor pagination encodes
{sortField, id}and cannot advance past a null cluster — "Load more" returns 0 rows with records clearly remaining. Sort byname(non-null, unique). - Paged reads MUST
orderBy: {id}. AcreatedAtcursor silently skips and repeats — one audit returned 143 rows containing only 90 unique ids out of 203. - No two views with the same name on the same object. That, not visibility scoping, is why users report "everyone sees a different view."
- Footer totals and per-group subtotals both come from
viewField.aggregateOperation— one flag, two render targets. - Field labels truncate at the END in the record rail. Two fields sharing a leading phrase ("Total Project …") become indistinguishable and people type into the wrong one. Name money fields
<Revenue|Margin> — <qualifier>. - Twenty auto-creates a viewField for a new field on FIELDS_WIDGET views only, never on TABLE/KANBAN. On those you must
createViewFieldexplicitly. updateViewFieldtakes one update key per call — multi-key updates silently truncate. Position before group.- "Can't enter a value for X" on a record page is almost always a hidden viewField, not a missing field. Check
isVisiblebefore building anything.
6. Permissions — the counter-intuitive ones
- There is no separate CREATE permission. The server gates
insertoncanUpdateObjectRecords. A role that must create records needs update, not just read. - Merge requires
canDestroyObjectRecordson that object (merge soft-deletes the loser). Single Delete only needs soft-delete. That's why a role can see Delete but not Merge. - Note/task creation validates READ on every polymorphic
noteTarget/taskTargettarget (~55target<Object>Idcolumns). One denied object breaks note creation on every record type, surfacing only as "An error occurred." 30 of 32 users were affected for three months. executeOneLogicFunctionis gated by theWORKFLOWSpermission flag. Admins pass only becausecanUpdateAllSettingsbypasses the guard — every front-component AI panel breaks for everyone else until the flag is granted.- RLS predicates crash on NULL.
DOES_NOT_CONTAINagainst a null title throwsCannot convert undefined or null to object, and since the INSERT already committed you get an orphan blank record. Shape the group asOR( title IS_EMPTY , AND( <the real predicates> ) ). - RLS is NOT applied to API-key contexts. Never test an RLS change with an API key — it reads straight through and the rule will look broken when it isn't. Test with a real session.
- To test what a role can do without touching a person's account: mint a role-scoped API key (
core."apiKey"+core."roleTarget"withapiKeyId, JWT signed withsha256(APP_SECRET + workspaceId + 'API_KEY')) — remembering the RLS caveat above.
7. The AI layer (models, Scout, chat)
- Three gates must agree before a model is usable. Env alone does nothing.
AI_PROVIDERSenv on bothabc_twentyANDabc_twenty-worker— the registry.core.workspace."enabledAiModelIds"(text[]) — the per-workspace allow-list = what the picker renders. Same row:smartModel/fastModel/routerModel.core."keyValuePair"keyAI_MODEL_PREFERENCES—disabledModels(admin kill-switch) plusdefaultSmartModels/defaultFastModels/recommendedModels.updateWorkspaceis user-context only ("API keys are not supported"), so gate 2 is a DB write.- Never bulk-prune
AI_PROVIDERS.getEffectiveModelConfigTHROWSModel with ID X not foundfor anything missing from the registry, andvalidateModelAvailabilitythrows when a model isn't workspace-enabled. Removing a provider while an agent'smodelId, the workspace default, or a preference default still names one of its models wedges every thread —activeStreamIdstays set, no assistant message, and the vendor endpoint streams fine under curl the whole time. Order: repoint agents + workspace defaults + preferences FIRST (DB, no restart), then remove providers from env. - Diagnostic tell: a healthy thread clears
core."agentChatThread"."activeStreamId"to NULL when the stream completes. Still set = wedged. The CRM logs nothing for these failures. - Probe a model with a realistic prompt and ~400 max_tokens, never
max_tokens: 5— reasoning models spend the whole budget thinking and return empty content, which reads as a dead model when it's fine. Several models return HTTP 200 with empty content: that is the "spins forever" failure, and it does not show up as an error anywhere. - z.ai coding-plan keys are served from
https://api.z.ai/api/coding/paas/v4. The standard.../api/paas/v4returns429 code 1113 "Insufficient balance"on the very same key. databaseEventTriggerSettings.eventNameis a single string — "fire on create AND update" needs two sister logic functions. A create event has nobeforestate, so any transition guard must handle that or it fires on every bulk import.- Updating a logic function's source sets
isBuildUpToDate=false; the build is triggered by the nextexecuteOneLogicFunction. Fire it once, then verify. - Wrap every logic function body in try/catch returning
{stage, error, stack}— otherwise the executor masks the real failure with the i18n string"Could not find flat entity in maps". executeOneLogicFunction's result field isdata, and its input type isExecuteOneLogicFunctionInput. Functions hard-timeout at 90s — budget every fetch withAbortSignal.timeout()below that.- Scout routes; it always needs a real tool. When you add a skill, field, or view, update Scout's system prompt in the same pass, and mirror new skills to the @ANC Slack bot.
- Currency reaches the model as micros. Ad-hoc AI workbooks come back 1000× high. Route any number a human will read to a
/api/render/*-xlsxendpoint instead.
8. Data — the rules that prevent wrong numbers
- NEVER match CRM ↔ Salesforce by name. Always
opportunityNumber == Opportunity_ID__c. Renewal-style deals ("…2021-26" vs "…2026-29") collide on every fuzzy scheme; 11 deals were wrongly marked Closed Won this way and a stakeholder caught it before we did. - NEVER link venues or events by name. Use
servicesId ↔ venues.id. A%M&T%search returns 11 Quinnipiac hockey games at M&T Bank Arena in Connecticut — nothing to do with the Ravens. - **Money is
*AmountMicros.** Convert once, in one helper. Clear a CURRENCY field with{amountMicros:null, currencyCode:null}— passing whole-objectnullis silently ignored. amount≠totalProjectRevenue≠dealValue.amount/marginare the FY rollup,totalProject*mirror SF Actual,dealValuemirrors SF Sale_Price. Anything claiming to mirror the Slack win alert readstotalProjectRevenue/totalProjectMargin. They routinely disagree, and picking the wrong one puts a wrong number in front of finance.- A derived number a human watches must be computed synchronously — use a DB trigger, not an async logic function. Event-driven rollups fire once per save and write the running total so far, so someone typing year-by-year watches the total step 500k → 1m → 1.5m, and a dropped event leaves it permanently short.
- DB-direct status writes leave no timeline entry and stakeholders notice. For any stakeholder-visible field change on a live record, go through the REST/GraphQL data API so
timelineActivityis written. Reserve raw SQL for metadata and bulk backfills. - Soft-delete by default on anything bulk; keep the id list for
restore*. - Never fabricate. No invented inventory, pricing, or "reasonable" defaults. If the real value doesn't exist, say so and ask — an empty field is honest, a plausible one is a landmine.
- When a stakeholder says a record is wrong and a recent batch touched it, audit the WHOLE batch immediately by exact key. One wrong row means the matching method was wrong.
9. Apps and front components
- Never
window.open,window.location.href, ortarget="_blank". Front components run in an iframe sandboxed withoutallow-popups-to-escape-sandbox, andlocationis read-only in the Worker context (Cannot set property href of WorkerLocation). Usenavigate(AppPath.RecordShowPage, …)fromtwenty-sdk/front-component. For same-origin parent nav,(window.top ?? window.parent).location.hrefworks. Cross-origin downloads need a plain<a href>or a blob URL — not a scripted open. - Pin
twenty-sdk/twenty-client-sdkto the version the live server runs.latestbreaks install withinstallApplication must not have a selection. - The runtime pageLayout id is NOT the manifest UUID — the manifest one 404s. Read the runtime id after install.
- App install busts its own metadata cache — no Redis bust, no restart, no EasyPanel deploy.
cp -rof an app dir breaks yarn PnP (keyed to the old package name). Runcorepack yarn installin the new dir.createPageLayoutWidgetneedstype(GRAPH/FIELD/…) in addition toconfiguration.configurationType, and it silently dropsconfiguration.filteron AGGREGATE_CHART widgets — create it, then set the full configuration viaupdatePageLayoutWidget, then read the rendered number back and reconcile against SQL. An unfiltered KPI tile looks completely plausible.- Widget ids must be real RFC 4122 UUIDs (
uuid_generate_v4()), andposition— not justgridPosition— must be set withlayoutMode: "GRID", or the whole dashboard 500s. - One RECORD_PAGE layout per object. Twenty serves the newest and there is no isDefault flag; a second layout silently wins and a Fields widget on a non-primary tab spins forever. Edit the existing Default layout instead.
10. Env-var safety (hard rule)
A previous AI session wiped every env var on a production service with a write-only "add". The only sanctioned pattern is read → merge → verify → write:
- Snapshot the full env to
/root/.env-snapshots/<svc>-<ts>.jsonfirst. - Merge; never full-replace.
- Diff and assert the result is a strict superset — no var lost, only the intended key changed.
- Write with
docker service update --env-add "KEY=VALUE" --update-order start-first <svc>. - Re-read and confirm.
CRM env changes must be applied to BOTH abc_twenty and abc_twenty-worker. Each write restarts the container: ~60s of 502s. Plan for it.
11. Infrastructure facts
abc_twentyruns a hand-tagged custom image (anc/twenty-v2170-anc-*), not an EasyPanel git build. Frontend behaviour is patched by injecting scripts intoindex.html. A future EasyPanel "deploy" can revert it — always checkdocker service inspect abc_twenty --format '{{.Spec.TaskTemplate.ContainerSpec.Image}}'before diagnosing a regression.anc-crm-guard.servicepins web+worker to the approved image id and auto-reverts drift. Approved upgrades go through/root/anc-crm-work/crm-update/anc-crm-update.sh, which opens and reseals a time-limited guard window. A slow boot that outruns that window makes the guard revert a perfectly healthy cutover.- DB access:
docker exec <abc_twenty container> sh -c "psql \$PG_DATABASE_URL -c '<sql>'". Workspace schema isworkspace_cjspnkm8glh7iooo1gep8c1qo. Pipe SQL from a file for anything containing JSON — nested shell quoting mangles it. - The DB is ahead of the image:
core."searchFieldMetadata"."tsVectorFieldMetadataId"is NOT NULL but the running server never sets it, socreateOneObjectfails workspace-wide. Theanc_fill_search_field_tsvector_trgtrigger unblocks it — confirm it exists before installing any app that adds an object. - API rate limit is 100 calls / 60s shared across REST and GraphQL; bulk updates cap at 200 records, creates at 50-row chunks, per-record PATCH paced ~1.5/s with backoff on 429.
- Nightly backup crons have twice PANICked the CRM's postgres with "No space left on device" around 9:30 PM ET. If the CRM is flapping late evening, check disk before anything else.
12. Verification — the part that is not optional
- Log in and look. Use the
crm-visual-verifyskill (email+password auth is enabled; creds at/root/.crm-creds). An API 200 does not mean it rendered. A DB row does not mean the user sees it. - Reconcile every number against direct SQL before it reaches a stakeholder. An unfiltered aggregate is the most convincing wrong answer there is.
- Exercise the path the way that person would — click their button, open their export, read the file. Not the endpoint. The artifact.
- A stakeholder message goes out only after that. "Should work" is not a state.
- Update the
crm-knowledgeskill in the same session. If it isn't written down it will be rediscovered the expensive way.
Compiled 2026-08-20 from the crm-knowledge skill, project memory, and the incident record.