325615576aa70b2e43223cdec212f80dce47006b
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joel Brock
|
325615576a |
Upload: route through Civi extension multipart endpoint
Every JSON-based upload path on this Civi stores the `content` field
verbatim on disk — confirmed against both APIv4 File.create AND APIv3
Attachment.create (both came back as base64 text in hex dumps). The
multipart `file` part to /civicrm/ajax/rest is also a dead end: APIv3
Attachment.create on this install doesn't see $_FILES (rejected with
"Mandatory key(s) missing: id or content or options.move-file").
The one path Civi honors is APIv3 Attachment.create + options.move-file
— pointing at a filesystem path the Civi server can read. So expose a
tiny multipart endpoint in the WebForm-mw Civi extension that copies
PHP's $_FILES['file']['tmp_name'] into the API call, then return the
new file id as JSON. PHP's $_FILES preserves binary natively.
Civi extension (requires admin deploy):
- CRM/WebformMw/Page/Upload.php : multipart POST handler. Validates
the upload, requires `access CiviCRM`, whitelists entity_table to
civicrm_contact|civicrm_activity, calls Attachment.create with
move-file pointing at the tmp upload, returns {id, name} JSON.
- xml/Menu/webform_mw.xml : registers civicrm/webform-mw/upload.
WebForm-mw side:
- lib/civicrm.ts : new civiMultipart() helper. POSTs multipart to an
arbitrary Civi path (not /civicrm/ajax/rest) with the same AuthX
headers. Returns the parsed JSON body.
- app/api/upload/route.ts : send the upload's bytes via civiMultipart
to civicrm/webform-mw/upload. Comment-block now records all four
upload paths we tried so a future reader doesn't repeat the cycle.
Deploy: admin syncs the updated civi-extension/webform-mw/ directory
and Disable/Re-enables the extension (or runs cv flush) so the new
menu route is registered.
|
||
|
Joel Brock
|
e42e3b70ef |
Upload: APIv3 Attachment.create with base64 content (not multipart)
The previous multipart attempt (
|
||
|
Joel Brock
|
f74fd07ba5 |
Upload: switch to APIv3 Attachment.create multipart (decode bytes properly)
APIv4 File.create on this Civi install stores the `content` field
verbatim — no base64 decoding. The hex dump of a downloaded file
confirms it: bytes start with 69 56 42 4f ("iVBO...") which is the
base64 encoding of the PNG header (89 50 4e 47), not the header itself.
Every file uploaded via the form has been corrupt on disk since launch.
JSON can't carry binary safely (high bytes break UTF-8), so the fix is
to stop trying. APIv3 Attachment.create accepts a multipart `file` part
the standard way (read from $_FILES on the server side) which preserves
bytes exactly.
Changes:
- lib/civicrm.ts: new civi3Upload() helper. POSTs multipart/form-data
with `entity`, `action`, `json`, and `file` parts to /civicrm/ajax/rest
using the same AuthX headers as civi3(). Wraps the Uint8Array into an
ArrayBuffer slice so Blob's narrower BlobPart typing accepts it.
- app/api/upload/route.ts: replace the v4 File.create JSON call with
civi3Upload("Attachment", "create", ...). Attachment.create requires
an entity context, so anchor to the form-filler's contact id. Our
custom-field flow uses the returned file id directly (no entity_file
linkage needed for prefill/download), so the extra civicrm_entity_file
row is metadata-only.
Note: existing files in Civi (uploaded via the buggy path) are still
corrupt on disk. New uploads will be intact. To recover the old ones
the user would need to re-upload via the form, or run a one-off
backfill that reads the base64 text out of /civicrm.files/upload/ and
rewrites each file with its decoded bytes.
|
||
|
Joel Brock
|
2400931a04 |
File upload pipeline: wire end-to-end via APIv4 File.create
Closes the file-upload gap. Files now actually land in CiviCRM (verified
empirically against the live Civi instance via spike scripts).
Spike findings (see scripts/spike-file-upload.mjs):
- APIv4 Attachment is NOT exposed on this Civi
- APIv4 File + EntityFile ARE exposed; File.create accepts inline
base64 `content` and returns a usable file id
- Custom file fields store the file id directly in the custom column,
so EntityFile linkage is unnecessary for this use case
- Round-trip via Contact.update + Contact.get .file_name join verified
on a real org contact
Pipeline:
Renderer (FileField) picks up onChange →
POST /api/upload (multipart) with file + cid + cs + fieldRef →
verifyChecksum, MIME allowlist + magic-byte sniff, 5 MB cap →
civi.File.create({ file_name, mime_type, content: base64 }) →
returns { id, file_name } →
renderer stores in RHF state via setValue
Form submit →
POST /api/submit (JSON) with the {id, file_name} value →
submit detects the file shape and writes the id as the value of
the activity/contact custom field
File changes:
app/api/upload/route.ts
Replaced the 501 stub with the real File.create call. Comment
documents that EntityFile linkage is intentionally skipped and that
orphan cleanup is owned by a CiviCRM scheduled job.
app/api/submit/route.ts
For type:"file" values shaped as {id, file_name}, write the id as
the custom field value (activity or contact, depending on the
civiField / civiContactField the field declares).
components/fields/FieldRenderer.tsx
Replaced the bare <input type=file> register() with FileField, an
upload-on-pick subcomponent. The native input is NOT register()'d:
its FileList value was the original bug. FileField owns its
uploading + error state and writes {id, file_name} via setValue on
success. Submit is blocked upstream while uploads are in flight.
components/StageSection.tsx, components/EngagementForm.tsx
Thread setValue, cid, cs, and an onUploadStateChange callback
through to FieldRenderer. EngagementForm tracks uploads-in-flight
count; onSubmit refuses to submit while the count is > 0.
config/form.ts
Promotes Certificate_of_Incorporation from readonly to a real
file field now that the pipeline works.
app/api/data/route.ts
Drops the readonly carveout that was only needed while the
certificate was readonly.
scripts/list-civi-entities.mjs (new)
APIv4 entity probe + APIv3 attachment-API probe. Used to determine
that File (not Attachment) was the right entity on this Civi.
scripts/spike-file-upload.mjs (new)
The actual end-to-end test that proved out the pipeline before
wiring. Safe to re-run on any Civi instance during future audits.
Not in this change:
- Orphan attachment cleanup (CiviCRM scheduled job, Civi admin scope)
- Per-field MIME allowlists (single global list for v1)
- S3 / presigned-URL path for >5 MB files (deferred; capped at 5 MB
today to stay under Amplify Lambda's 6 MB sync payload limit)
|
||
|
Joel Brock
|
8159b87074 |
File upload pipeline: spike + endpoint skeleton (Phase 1, in progress)
Lays groundwork for closing the file-upload gap discovered while wiring
the org-contact custom fields. Currently no file fields in the form
actually persist to CiviCRM -- the renderer FileList drops at the
onSubmit JSON.stringify, and there is no /api/upload route or
Attachment.create call anywhere.
This commit adds:
1. scripts/spike-attachment-upload.mjs
One-off spike to answer the open question that gates the rest of the
work: does APIv4 Attachment.create accept an unbound upload, or must
we attach to an entity at create time? If unbound works we can use
the planned two-step pattern (upload returns a file id; submit
references it). If not, activity-bound file fields need a different
flow because the activity does not exist yet at upload time.
The spike also exercises the Contact.update + .file_name read-back
path against the Certificate_of_Incorporation field on a real org
contact, then cleans up after itself.
Usage:
node --env-file=.env.local scripts/spike-attachment-upload.mjs \
--org-id=<id> [--keep]
2. app/api/upload/route.ts
Structural pieces that do not depend on the spike outcome:
- multipart parsing via Request.formData()
- 5 MB hard cap (under Amplify Lambda 6 MB sync payload limit)
- MIME allowlist (PDF, DOC/DOCX, XLS/XLSX, JPEG/PNG/GIF/WEBP)
- magic-byte sniff to cross-check the client-reported MIME
- filename sanitization (path traversal scrub, length cap)
- checksum verification, rate limiting, field-ref allowlist
- STUB-mode short-circuit for local dev without live Civi
- explicit 501 where the Civi Attachment.create wiring goes,
with a comment pointing at the spike that resolves it
Result: endpoint compiles, registers as a Next route, returns 501
with a clear message; build passes; nothing wired into the frontend
yet so the existing form is unaffected.
Phase 2 (renderer upload-on-pick), Phase 3 (submit reshape), Phase 4
(promote Certificate_of_Incorporation to editable) follow once the
spike output picks the Attachment.create variant.
|