Drops the trailing sentence about returning later for another check-in (now redundant with the post-submit CTA) and matches the rest of the UI's 'survey' terminology.
Each stage-transition activity now becomes a horizontal range on its lane: start = the activity's date, end = the next activity with a higher stage rank, or extending to today if still in effect. Milestone date-field dots overlay on top of the ranges. Activity dots at each range start carry a tooltip with the activity subject. Added computeStageRanges in lib/stageRank.ts and switched DateTimeline to take activities directly (deriving both the rank resolver and the ranges internally). Wide year-spanning timelines now show their full extent even when no milestone dates have been entered.
Each date-field event is now plotted on the lane corresponding to the Framework Stage the co-op was in on the field's stored date, resolved from the activity stream's stage transitions. Date_Opened is pinned to Stage 5 as the journey anchor. Events that pre-date any known transition fall back to the field's section rank so they still surface somewhere. Extracts the STAGE_RANK map (previously inline in ReportView) into lib/stageRank.ts alongside the new buildStageRankAtDate resolver factory.
Subtitle now tells respondents they can skip fields they do not know. Post-submit CTA says Submit an update since the same form is the ongoing update channel, not a one-off survey.
DateTimeline now skips section rank 0 (intake) and renders stage lanes
top-to-bottom 5→1, matching how the framework actually numbers stages.
MembershipChart's latest-Actual label was placed at the circle's
x + 6, which overflowed the SVG when the most-recent point landed near
the right padding. Anchor it to xOf(maxT) - 4 with textAnchor="end",
mirroring the Goal label so both endpoint labels stay inside the chart.
Every JSON-based upload path on this Civi stores the `content` field
verbatim on disk — confirmed against both APIv4 File.create AND APIv3
Attachment.create (both came back as base64 text in hex dumps). The
multipart `file` part to /civicrm/ajax/rest is also a dead end: APIv3
Attachment.create on this install doesn't see $_FILES (rejected with
"Mandatory key(s) missing: id or content or options.move-file").
The one path Civi honors is APIv3 Attachment.create + options.move-file
— pointing at a filesystem path the Civi server can read. So expose a
tiny multipart endpoint in the WebForm-mw Civi extension that copies
PHP's $_FILES['file']['tmp_name'] into the API call, then return the
new file id as JSON. PHP's $_FILES preserves binary natively.
Civi extension (requires admin deploy):
- CRM/WebformMw/Page/Upload.php : multipart POST handler. Validates
the upload, requires `access CiviCRM`, whitelists entity_table to
civicrm_contact|civicrm_activity, calls Attachment.create with
move-file pointing at the tmp upload, returns {id, name} JSON.
- xml/Menu/webform_mw.xml : registers civicrm/webform-mw/upload.
WebForm-mw side:
- lib/civicrm.ts : new civiMultipart() helper. POSTs multipart to an
arbitrary Civi path (not /civicrm/ajax/rest) with the same AuthX
headers. Returns the parsed JSON body.
- app/api/upload/route.ts : send the upload's bytes via civiMultipart
to civicrm/webform-mw/upload. Comment-block now records all four
upload paths we tried so a future reader doesn't repeat the cycle.
Deploy: admin syncs the updated civi-extension/webform-mw/ directory
and Disable/Re-enables the extension (or runs cv flush) so the new
menu route is registered.
The previous multipart attempt (f74fd07) was rejected by Civi with
"Mandatory key(s) missing: id or content or options.move-file" — the
/civicrm/ajax/rest endpoint on this install doesn't expose $_FILES to
v3 actions; only params['content'] is consulted. Our multipart `file`
part was ignored.
APIv3 Attachment.create has historically been the file-upload entry
point used by Civi's own form widgets and auto-decodes the `content`
field from base64, unlike v4 File.create which stores it verbatim.
Send the same shape we tried first (entity_table, entity_id, name,
mime_type, content=base64) but to v3 Attachment instead of v4 File.
The civi3Upload helper in lib/civicrm.ts is kept in place — it's not
useful for this endpoint but the multipart-POST shape may be needed
later for other Civi entities that do read $_FILES.
APIv4 File.create on this Civi install stores the `content` field
verbatim — no base64 decoding. The hex dump of a downloaded file
confirms it: bytes start with 69 56 42 4f ("iVBO...") which is the
base64 encoding of the PNG header (89 50 4e 47), not the header itself.
Every file uploaded via the form has been corrupt on disk since launch.
JSON can't carry binary safely (high bytes break UTF-8), so the fix is
to stop trying. APIv3 Attachment.create accepts a multipart `file` part
the standard way (read from $_FILES on the server side) which preserves
bytes exactly.
Changes:
- lib/civicrm.ts: new civi3Upload() helper. POSTs multipart/form-data
with `entity`, `action`, `json`, and `file` parts to /civicrm/ajax/rest
using the same AuthX headers as civi3(). Wraps the Uint8Array into an
ArrayBuffer slice so Blob's narrower BlobPart typing accepts it.
- app/api/upload/route.ts: replace the v4 File.create JSON call with
civi3Upload("Attachment", "create", ...). Attachment.create requires
an entity context, so anchor to the form-filler's contact id. Our
custom-field flow uses the returned file id directly (no entity_file
linkage needed for prefill/download), so the extra civicrm_entity_file
row is metadata-only.
Note: existing files in Civi (uploaded via the buggy path) are still
corrupt on disk. New uploads will be intact. To recover the old ones
the user would need to re-upload via the form, or run a one-off
backfill that reads the base64 text out of /civicrm.files/upload/ and
rewrites each file with its decoded bytes.
The {IN: [...]} operator object for `id` crashes Civi APIv3 on this
install — Civi's error renderer calls htmlentities() on the array
value and dies, producing the 500 with the html error page seen at
6c3e8dd's deploy.
Single-id calls succeed (confirmed in the user's API Explorer test
against id=150). So loop one call per file id and run them in
parallel via Promise.allSettled. Reports typically reference a
handful of files, so the round-trip overhead is small and individual
file failures don't poison the whole report.
Previous commit (8ace5f4) passed `return: ["id", "url"]` to APIv3
Attachment.get. APIv3 expects `return` as a comma-separated string
("id,url"); arrays are v4 syntax. Civi caught the type mismatch but
its error-rendering pathway then crashed on htmlentities() (which
was passed the offending array), producing a 500 with an HTML error
page rather than a clean JSON error.
Fix: pass return as "id,url" and add sequential:1 (canonical v3 client
shape — values come back as an array). The civi3 helper already
normalizes either response shape, but sequential matches what real
APIv3 clients send.
Logs to confirm after deploy:
- Success: no "[staff/report] Attachment.get (v3) failed" warning.
- File links resolve directly without bouncing off the
/civicrm/webform-mw/file extension route.
APIv4 Attachment isn't exposed on this Civi install (confirmed in the
June 2026 upload-spike notes), so the Attachment.get call we shipped at
63e73e7 silently returned nothing and we fell through to the bare
/civicrm/file URL — which crashes Civi on a null fcs JWT decode.
APIv3 Attachment.get IS exposed and returns the signed URL with fcs
baked in (verified against id=150 in the user's API Explorer):
"url": "https://.../civicrm/file?reset=1&id=150&fcs=<JWT>"
Changes:
- lib/civicrm.ts: add a civi3() helper that calls /civicrm/ajax/rest with
AuthX headers, normalizing v3's array-or-keyed-object values shape into
a plain array.
- app/api/staff/report/route.ts: replace the dead v4 Attachment.get with
civi3("Attachment", "get", { id: {IN: [...]}, return: ["id","url"] }).
Each file's url goes into the value payload as before, so the frontend
needs no change.
Fallback chain remains intact: if Attachment.get fails (auth, endpoint
unavailable, etc.) the frontend still uses the /civicrm/webform-mw/file
extension route from b65bc6d/41467bd.
Security-review follow-up to b65bc6d. The file-redirect route signs an
fcs JWT that Civi's /civicrm/file handler accepts as proof of access.
Any user with the base `access CiviCRM` permission could iterate file
IDs and have us laundering tokens past the entity-level ACLs that would
normally apply (e.g. a staff user without view permission on a given
contact could still pull files attached to that contact).
Tighten it:
- Resolve the file's linked entity_table + entity_id (was: entity_id only).
- Run the entity-type's native permission check before signing the JWT:
civicrm_activity -> CRM_Activity_BAO_Activity::checkPermission
civicrm_contact -> CRM_Contact_BAO_Contact_Permission::allow
Unknown entity types deny by default — adding a new type requires an
explicit edit here, so we don't accidentally widen the surface.
- Drop JWT lifetime from a week to 10 minutes. The token is minted at
click time (the user hits this route fresh on each file click), so
the long lifetime served no purpose and made each URL a longer-lived
bearer credential.
Files missing from civicrm_entity_file or pointing at unsupported entity
types now 403 via CRM_Utils_System::permissionDenied() instead of
producing a download URL.
Path 1 (APIv4 Attachment.get + select url) shipped but didn't fix the
Firebase\JWT decode crash — the deployed Civi version either omits `url`
from Attachment.get or returns it without the fcs param. Falling back to
the bare /civicrm/file?id=X URL hits the same JWT null crash.
Path 2: route file clicks through a tiny redirect endpoint in the Civi
extension instead. The extension runs PHP on Civi, has access to the
crypto.jwt service, and mints the same shape of token Civi's own file
URL builder uses ({exp, "civi.file": <id>}) before 302-redirecting to
the canonical /civicrm/file URL.
Civi extension changes:
- New CRM/WebformMw/Page/File.php — resolves eid from civicrm_entity_file
if not supplied, signs a 7-day JWT via Civi::service('crypto.jwt'),
redirects.
- xml/Menu/webform_mw.xml — registers civicrm/webform-mw/file. Requires
`access CiviCRM` (the user is already authenticated in the parent Civi
tab when they click the link).
Frontend (StaffReportView.tsx, FieldValue):
- When Attachment.get's url is missing, fall back to the new extension
route instead of bare /civicrm/file. Attachment.get's url remains the
fast path when present.
Deploy: admin needs to push the updated extension files to the Civi
server, then Disable/Enable webform-mw (or cv flush) so the new menu
route registers in civicrm_menu.
The Y1 Monthly Sales Target fields don't match the Y1_M<n>_<metric> regex
the monthly matrix collector uses to discover rows:
M1 -> Y1_Monthly_Sales_Targets (no _M1 suffix; trailing 's')
M2 -> Y1_Monthly_Sales_Targets_M2 (plural with _M2)
M3 -> Y1_Monthly_Sales_Target_M2 (Civi name says _M2 but the value
represents M3; pre-existing
schema error)
M4..M12 -> Y1_Monthly_Sales_Target_M<n>
Hardcode a period->civi-field-name map (Y1_MONTHLY_SALES_TARGET_FIELDS)
so the monthly matrix can pick these up alongside the regex-matched
Y1_M<n>_Actual_Sales / Y1_M<n>_Transactions rows. The M3->_M2
irregularity is called out inline so a future reader doesn't "fix" it
into a regression.
Civi serves uploaded files at /civicrm/file?id=X&eid=Y&fcs=<JWT>; the fcs
is a JWT signed with the site key. Without it, Civi's file handler crashes
on a null JWT decode (Firebase\JWT\JWT::decode argument null). We don't
have the site key on the Next.js side, so let Civi mint the URLs for us.
Backend (/api/staff/report):
- Add file_name selects for org-side file fields (Certificate of
Incorporation and friends) so org files have names alongside URLs.
- Collect every file id referenced by activity and org custom fields.
- Call APIv4 Attachment.get with select: ["id", "url"] to fetch signed
URLs in one round trip. Build a urlByFileId map.
- Org-side file values are now wrapped in { id, file_name, url } shape
matching the activity-side files (previously bare file ids that the
frontend couldn't render).
- Activity-side file values gain a url property from the map.
- If Attachment.get doesn't expose url on this Civi version, the call is
caught and we fall through to bare URLs without fcs (no regression).
Frontend (FieldValue):
- Prefer v.url when present, normalizing absolute and relative shapes
against CIVI_BASE_URL.
- Fall back to /civicrm/file?reset=1&id=X if url wasn't provided.
Five related refinements to the staff report:
1. Surface latest submitter. Pull the most recent non-empty
Survey_completed_by / Survey_completed_by_email values from the
Check_in_data__organizing_ history and render them just below the org
stats in the report header. Email is a mailto: link. Hidden when both
values are empty.
2. Y1 monthly matrix. Generalize the Y1 matrix collector to detect either
Y1_Q<n>_<metric> or Y1_M<n>_<metric> field-name patterns. Stage 5 now
renders the quarterly table (when present) followed by the monthly
table (when present); each table auto-labels its columns Q1..Qn or
M1..Mn from the data, and the caption reflects the cadence. Adding a
new Y1_M<n>_<metric> field in Civi extends the columns automatically.
3. Larger field value. The latest value in each CompactFieldRow is now
font-display text-xl text-leaf-800 (previously text-[13px] text-ink-soft).
Makes the current number the dominant element in each row.
4. Smaller right-aligned earlier-entries toggle. The "N earlier entries"
button moves out of the inline date line onto its own row beneath the
"as of <date>" caption, right-aligned, in a 10px link style.
5. Right-aligned expanded entries. When earlier entries are unhidden,
each row now shows date on the left and the value on the right, mirroring
the active value's right alignment. Values render in font-display text-base
text-ink-soft so they visually echo the latest value while being clearly
demoted in size and color. The list is constrained to max-w-[24rem] with
ml-auto so it sits under the active value column rather than spanning the
full row.
Captures the form-filler's name and email on every check-in. Both fields
are required; values write back to Check_in_data__organizing_.Survey_completed_by
and Survey_completed_by_email on the activity, giving us a per-submission
record of who filled out which check-in.
Implementation:
- config/form.ts: new submitterInfo section (rank -1) at the head of the
sections array. rank -1 keeps it out of the past/current/future stage
pathway computation.
- components/EngagementForm.tsx: filter the submitter section out of
sectionsToRender and render it directly with FieldRenderer inside a
bordered card above the stage list. The fields still flow through RHF
registration, onInvalid scroll-to-error, and the onSubmit visibility
filter the same as any other field.
The staff report auto-discovers these fields via CustomField.get since
they live in Check_in_data__organizing_, so the field history shows up
in the report with no extra wiring.
The iframe in the Engagement Report tab is sized to fit content, so it
has no internal scroll context. Clicking an anchor link inside it
changes the URL hash but the iframe content doesn't move and the user
has to scroll the outer CiviCRM page manually. Rather than coordinate
cross-frame scroll with the parent, just skip rendering the anchor
strip when framed. Standalone view is unchanged.
The framed report posts its content height to the parent so the Civi tab
can resize the iframe to fit. Two pieces interacted badly:
- The root layout sets html.h-full and body.min-h-full, so documentElement
and body heights track the iframe's viewport height.
- The parent template sets iframe.height = postedHeight + 24 every time
a height message arrives.
The combination produced an unbounded feedback loop: parent grows the
iframe by 24px, viewport grows, document height grows, ResizeObserver
fires, we post the new height, parent grows by another 24px. The outer
CiviCRM page scrollbar visibly shrank each cycle.
Fix on the report side (no extension change needed): when framed, override
html height to auto and body min-height to 0 so the document decouples
from the viewport. Observe body (the actual content), measure
body.scrollHeight, and skip posting when the value is unchanged. Original
styles are restored on unmount so route changes back to the standalone
view still work.
The Engagement Report tab page renders templates/CRM/WebformMw/Page/Tab.tpl
via Smarty. Without an explicit template-dir registration, Smarty cannot
locate the file and the tab fails to render. Use CRM_Core_Smarty's
prependTemplateDir() (cleaner than splicing template_dir by hand).
Mirrors the same fix the CiviCRM admin applied on the server so the repo
source no longer drifts from the working deployed state.
Adds a native window.confirm() inside onSubmit, after the uploads-in-flight
guard and before the submitting-state flip. Runs only after react-hook-form
validation passes, so users who hit Submit on an incomplete form still see
the existing field-level errors via onInvalid rather than a confusing
"are you sure?" prompt. Cancelling leaves the form state untouched.
The Engagement Report tab on Organization contact pages was loading the
contact-view summary recursively inside its own tab pane. Root cause:
Civi's menu router was not picking up the extension's xml/Menu file, so
`civicrm/contact/view/engagement-report` fell back to the parent
`civicrm/contact/view` route. Adding an explicit hook_civicrm_xmlMenu
implementation forces the menu file to register, after which the route
resolves to CRM_WebformMw_Page_Tab and the iframe renders as intended.
Deploy: replace the extension files on the Civi server, then in
Administer → System Settings → Extensions Disable + re-Enable webform-mw
(or run `cv flush` on the server) so the menu cache is rebuilt.
No fields on the survey should block submit — removes required:true
from Preliminary_Market_Assessment so the form is fully optional end
to end.
Surfaces Stage 0 sources-and-uses currency fields (Total_cost_of_project,
Member_equity_raised, Member_loans_raised, Member_preferred_shares_raised,
Bank_debt_raised, Grants_Donations_Raised, Other_sources_raised) starting
at Business Feasibility instead of Stabilize so they can be filled in
earlier in the lifecycle.
Cross-references the Civi-side cutover checklist from the deploy doc
so anyone deploying knows the CiviCRM prerequisites are tracked in a
single place.
App side:
- Per-route CSP: /staff/report now sets frame-ancestors 'self'
<CIVI_BASE_URL origin> and drops X-Frame-Options so the CiviCRM
extension can iframe it. All other routes keep frame-ancestors
'none' + X-Frame-Options: DENY via a path-negation source.
- Staff page recognises ?frame=1 and renders without SiteHeader/
SiteFooter so it fills the iframe cleanly.
- StaffReportView posts its scrollHeight to the parent window via
postMessage when framed; the Civi tab listens and auto-resizes
the iframe (no nested scrollbar). Anchor strip drops its sticky
positioning in frame mode since there's no internal scroll.
CiviCRM extension (civi-extension/webform-mw/, key webform-mw):
- info.xml + main hook file (webform_mw.php) implementing
hook_civicrm_tabset to add an 'Engagement Report' tab to
Organization contact-view pages.
- CRM/WebformMw/Page/Tab.php + Smarty template render an iframe
pointing at <WEBFORM_MW_APP_URL>/staff/report?org=<cid>&key=&frame=1,
with a postMessage listener that validates event.origin against
the configured app URL before resizing.
- Config via PHP constants in civicrm.settings.php (WEBFORM_MW_APP_URL,
WEBFORM_MW_STAFF_KEY) or matching env vars. Help banner shown when
unconfigured.
- README documents install, config, behaviour, security caveats.
UX iteration after first live look:
- Sticky anchor strip below the header with a chip per section (incl.
Submissions) so staff can jump around a long page.
- Compact one-line rows that show only the latest value; multi-history
fields get a muted 'N earlier entries' toggle that reveals the rest
inline. Same affordance for file fields.
- Empty fields collapse under a single 'N empty fields' toggle per
section instead of taking a row each.
- Stage 5: Y1_Q<n>_<metric> fields render as a read-only matrix table
(rows: metrics; columns: Q1..Q4) matching the form's matrix layout.
File proxy (/api/staff/file) deleted. APIv4 Attachment isn't exposed
on this Civi instance (per the June upload spike), which is why the
previous proxy returned broken images. Staff are already authenticated
to Civi when they arrive here, so file fields now render as outbound
links to CIVI_BASE_URL/civicrm/file?reset=1&id=<id> and the browser
uses the staff session. No more proxy auth, no more SSRF surface to
harden, no broken images.
CIVI_BASE_URL flows from the staff page (server component) into the
client as a prop. No secret material crosses the boundary.
Without this, even after setting STAFF_REPORT_KEY in the Amplify Secrets
tab the value never reaches the SSR Lambda — the build script only writes
the listed env vars into .env.production, which is what Next bundles.
- Allowlist inline MIME types (png/jpeg/gif/webp/pdf only); everything
else, including SVG and HTML, served as application/octet-stream
with content-disposition: attachment.
- X-Content-Type-Options: nosniff and a restrictive CSP on every response.
- Validate the upstream URL Civi returns: must match CIVI_BASE_URL origin
before we attach basic-auth creds and follow it. redirect: manual to
prevent off-host hops.
- Drop SVG from the client's inline-image list (server forces download).
Closes the file-upload gap. Files now actually land in CiviCRM (verified
empirically against the live Civi instance via spike scripts).
Spike findings (see scripts/spike-file-upload.mjs):
- APIv4 Attachment is NOT exposed on this Civi
- APIv4 File + EntityFile ARE exposed; File.create accepts inline
base64 `content` and returns a usable file id
- Custom file fields store the file id directly in the custom column,
so EntityFile linkage is unnecessary for this use case
- Round-trip via Contact.update + Contact.get .file_name join verified
on a real org contact
Pipeline:
Renderer (FileField) picks up onChange →
POST /api/upload (multipart) with file + cid + cs + fieldRef →
verifyChecksum, MIME allowlist + magic-byte sniff, 5 MB cap →
civi.File.create({ file_name, mime_type, content: base64 }) →
returns { id, file_name } →
renderer stores in RHF state via setValue
Form submit →
POST /api/submit (JSON) with the {id, file_name} value →
submit detects the file shape and writes the id as the value of
the activity/contact custom field
File changes:
app/api/upload/route.ts
Replaced the 501 stub with the real File.create call. Comment
documents that EntityFile linkage is intentionally skipped and that
orphan cleanup is owned by a CiviCRM scheduled job.
app/api/submit/route.ts
For type:"file" values shaped as {id, file_name}, write the id as
the custom field value (activity or contact, depending on the
civiField / civiContactField the field declares).
components/fields/FieldRenderer.tsx
Replaced the bare <input type=file> register() with FileField, an
upload-on-pick subcomponent. The native input is NOT register()'d:
its FileList value was the original bug. FileField owns its
uploading + error state and writes {id, file_name} via setValue on
success. Submit is blocked upstream while uploads are in flight.
components/StageSection.tsx, components/EngagementForm.tsx
Thread setValue, cid, cs, and an onUploadStateChange callback
through to FieldRenderer. EngagementForm tracks uploads-in-flight
count; onSubmit refuses to submit while the count is > 0.
config/form.ts
Promotes Certificate_of_Incorporation from readonly to a real
file field now that the pipeline works.
app/api/data/route.ts
Drops the readonly carveout that was only needed while the
certificate was readonly.
scripts/list-civi-entities.mjs (new)
APIv4 entity probe + APIv3 attachment-API probe. Used to determine
that File (not Attachment) was the right entity on this Civi.
scripts/spike-file-upload.mjs (new)
The actual end-to-end test that proved out the pipeline before
wiring. Safe to re-run on any Civi instance during future audits.
Not in this change:
- Orphan attachment cleanup (CiviCRM scheduled job, Civi admin scope)
- Per-field MIME allowlists (single global list for v1)
- S3 / presigned-URL path for >5 MB files (deferred; capped at 5 MB
today to stay under Amplify Lambda's 6 MB sync payload limit)
Lays groundwork for closing the file-upload gap discovered while wiring
the org-contact custom fields. Currently no file fields in the form
actually persist to CiviCRM -- the renderer FileList drops at the
onSubmit JSON.stringify, and there is no /api/upload route or
Attachment.create call anywhere.
This commit adds:
1. scripts/spike-attachment-upload.mjs
One-off spike to answer the open question that gates the rest of the
work: does APIv4 Attachment.create accept an unbound upload, or must
we attach to an entity at create time? If unbound works we can use
the planned two-step pattern (upload returns a file id; submit
references it). If not, activity-bound file fields need a different
flow because the activity does not exist yet at upload time.
The spike also exercises the Contact.update + .file_name read-back
path against the Certificate_of_Incorporation field on a real org
contact, then cleans up after itself.
Usage:
node --env-file=.env.local scripts/spike-attachment-upload.mjs \
--org-id=<id> [--keep]
2. app/api/upload/route.ts
Structural pieces that do not depend on the spike outcome:
- multipart parsing via Request.formData()
- 5 MB hard cap (under Amplify Lambda 6 MB sync payload limit)
- MIME allowlist (PDF, DOC/DOCX, XLS/XLSX, JPEG/PNG/GIF/WEBP)
- magic-byte sniff to cross-check the client-reported MIME
- filename sanitization (path traversal scrub, length cap)
- checksum verification, rate limiting, field-ref allowlist
- STUB-mode short-circuit for local dev without live Civi
- explicit 501 where the Civi Attachment.create wiring goes,
with a comment pointing at the spike that resolves it
Result: endpoint compiles, registers as a Next route, returns 501
with a clear message; build passes; nothing wired into the frontend
yet so the existing form is unaffected.
Phase 2 (renderer upload-on-pick), Phase 3 (submit reshape), Phase 4
(promote Certificate_of_Incorporation to editable) follow once the
spike output picks the Attachment.create variant.
Joel Brock