Upload: switch to APIv3 Attachment.create multipart (decode bytes properly)

APIv4 File.create on this Civi install stores the `content` field verbatim — no base64 decoding. The hex dump of a downloaded file confirms it: bytes start with 69 56 42 4f ("iVBO...") which is the base64 encoding of the PNG header (89 50 4e 47), not the header itself. Every file uploaded via the form has been corrupt on disk since launch. JSON can't carry binary safely (high bytes break UTF-8), so the fix is to stop trying. APIv3 Attachment.create accepts a multipart `file` part the standard way (read from $_FILES on the server side) which preserves bytes exactly. Changes: - lib/civicrm.ts: new civi3Upload() helper. POSTs multipart/form-data with `entity`, `action`, `json`, and `file` parts to /civicrm/ajax/rest using the same AuthX headers as civi3(). Wraps the Uint8Array into an ArrayBuffer slice so Blob's narrower BlobPart typing accepts it. - app/api/upload/route.ts: replace the v4 File.create JSON call with civi3Upload("Attachment", "create", ...). Attachment.create requires an entity context, so anchor to the form-filler's contact id. Our custom-field flow uses the returned file id directly (no entity_file linkage needed for prefill/download), so the extra civicrm_entity_file row is metadata-only. Note: existing files in Civi (uploaded via the buggy path) are still corrupt on disk. New uploads will be intact. To recover the old ones the user would need to re-upload via the form, or run a one-off backfill that reads the base64 text out of /civicrm.files/upload/ and rewrites each file with its decoded bytes.
2026-06-10 11:57:52 -07:00
parent e22c9226d8
commit f74fd07ba5
2 changed files with 127 additions and 29 deletions
@@ -25,7 +25,7 @@
 */

 import { NextResponse } from "next/server";
-import { civi, verifyChecksum } from "@/lib/civicrm";
+import { civi3Upload, verifyChecksum } from "@/lib/civicrm";
 import { allFields } from "@/config/form";
 import { rateLimit, clientIp } from "@/lib/rate-limit";

@@ -205,44 +205,52 @@ export async function POST(req: Request) {
    );
  }

-  // APIv4 File.create with inline base64 content. Spike (June 2026) on
-  // this Civi instance confirmed:
-  //   - APIv4 Attachment is NOT exposed
-  //   - APIv4 File + EntityFile ARE exposed
-  //   - File.create with file_name + mime_type + content (base64) returns
-  //     a usable file id
-  //   - Custom file fields store the file id directly in the custom
-  //     column, so EntityFile linkage is not needed for our use case
-  //   - Round-trip via Contact.update + Contact.get .file_name join works
+  // APIv3 Attachment.create with a multipart `file` part. The earlier
+  // approach used APIv4 File.create with `content: base64String` and that
+  // Civi on this install stores the literal base64 *text* on disk — every
+  // file came back corrupted (hex 6956... = "iVBO..." = base64 PNG header).
+  // APIv3 Attachment.create reads $_FILES['file'] from the multipart body
+  // and writes the bytes through unchanged.
  //
-  // We do not create EntityFile rows here. Civi's custom-field renderer
-  // joins through the custom column to civicrm_file directly, and the
-  // form-side prefill/read code in /api/data uses the same join.
+  // Attachment.create requires (entity_table, entity_id). Our custom-field
+  // flow stores the file id directly in the custom column (no
+  // civicrm_entity_file linkage needed for prefill/download), but the API
+  // still mandates an entity context for the upload call itself, so we
+  // anchor to the form-filler's contact id. The resulting civicrm_entity_file
+  // row is metadata-only — Civi's custom-field joins ignore it.
  //
-  // The returned id is what the frontend stores in RHF state and
-  // ultimately sends as the field value on /api/submit. /api/submit then
-  // writes that id to the activity custom field (for stage-N file fields)
-  // or to the org contact custom field (for Food_Co_op_Organizing.*).
+  // The returned id is what the frontend stores in RHF state and ultimately
+  // sends as the field value on /api/submit. /api/submit then writes that
+  // id to the activity custom field (stage-N) or to the org contact custom
+  // field (Food_Co_op_Organizing.*).
  //
  // Orphan files: if the user uploads and then abandons the form, the
-  // File row persists with no entity referencing it. Cleanup is handled
-  // by a CiviCRM scheduled job (configured separately by the Civi admin)
-  // that deletes File rows with no inbound references older than ~24h.
+  // civicrm_file + civicrm_entity_file rows persist with no business
+  // reference. Cleanup is handled by a CiviCRM scheduled job (configured
+  // separately by the Civi admin) that prunes orphans older than ~24h.
  let fileId: number;
  try {
-    const res = await civi<{ id: number }>("File", "create", {
-      values: {
-        file_name: safeName,
+    const res = await civi3Upload<{ id: string | number }>(
+      "Attachment",
+      "create",
+      {
+        entity_table: "civicrm_contact",
+        entity_id: Number(cid),
+        name: safeName,
        mime_type: clientMime,
-        content: Buffer.from(bytes).toString("base64"),
+        sequential: 1,
      },
-    });
-    const id = res.values?.[0]?.id;
-    if (!id) throw new Error("File.create returned no id");
-    fileId = Number(id);
+      { bytes, filename: safeName, mime: clientMime },
+    );
+    const idRaw = res.values?.[0]?.id;
+    const id = typeof idRaw === "string" ? Number(idRaw) : idRaw;
+    if (!id || !Number.isFinite(id)) {
+      throw new Error("Attachment.create returned no id");
+    }
+    fileId = id;
  } catch (err) {
    const msg = err instanceof Error ? err.message : String(err);
-    console.error("[upload] File.create failed:", msg);
+    console.error("[upload] Attachment.create failed:", msg);
    return NextResponse.json(
      { error: "Could not save the upload. Please try again." },
      { status: 502 },
@@ -154,6 +154,96 @@ export async function civi3<T = unknown>(
  return { values, count: json.count };
 }

+/**
+ * Multipart APIv3 call. Used for binary uploads — APIv4 File.create stores
+ * the `content` field verbatim (no base64 decoding), so files come back
+ * corrupted. APIv3 Attachment.create accepts the file via the standard
+ * multipart `file` part (read from $_FILES on the server) which preserves
+ * the bytes exactly.
+ *
+ * Caller provides `params` (non-binary metadata) and `file` (the binary +
+ * filename + mime). Authentication is the same AuthX headers as civi3().
+ */
+export async function civi3Upload<T = unknown>(
+  entity: string,
+  action: string,
+  params: Record<string, unknown>,
+  file: { bytes: Uint8Array; filename: string; mime: string },
+  opts: CiviApiOptions = {},
+): Promise<CiviApiResponse<T>> {
+  if (isStubMode()) {
+    console.warn(
+      `${STUB_LOG_PREFIX} v3-multipart ${entity}.${action} — env not set, returning empty values`,
+    );
+    return { values: [] };
+  }
+
+  const base = opts.baseUrl ?? process.env.CIVI_BASE_URL!;
+  const url = `${base}/civicrm/ajax/rest`;
+
+  const form = new FormData();
+  form.append("entity", entity);
+  form.append("action", action);
+  form.append("json", JSON.stringify(params));
+  // Wrap the Uint8Array in a fresh ArrayBuffer slice so Blob's typing
+  // (which only accepts ArrayBuffer, not the wider ArrayBufferLike) is
+  // happy. The slice is a no-op on real Uint8Array inputs.
+  const fileBuf = file.bytes.buffer.slice(
+    file.bytes.byteOffset,
+    file.bytes.byteOffset + file.bytes.byteLength,
+  ) as ArrayBuffer;
+  form.append(
+    "file",
+    new Blob([fileBuf], { type: file.mime }),
+    file.filename,
+  );
+
+  // Do NOT set Content-Type — fetch sets multipart/form-data with the
+  // correct boundary automatically when body is a FormData.
+  const headers: Record<string, string> = {
+    "X-Civi-Auth": `Bearer ${process.env.CIVI_API_KEY}`,
+    "X-Civi-Key": process.env.CIVI_SITE_KEY!,
+    "X-Requested-With": "XMLHttpRequest",
+  };
+  if (process.env.CIVI_HTTP_AUTH_USER && process.env.CIVI_HTTP_AUTH_PASS) {
+    const creds = Buffer.from(
+      `${process.env.CIVI_HTTP_AUTH_USER}:${process.env.CIVI_HTTP_AUTH_PASS}`,
+    ).toString("base64");
+    headers["Authorization"] = `Basic ${creds}`;
+  }
+
+  const res = await fetch(url, {
+    method: "POST",
+    headers,
+    body: form,
+    cache: "no-store",
+  });
+  if (!res.ok) {
+    const text = await res.text();
+    throw new Error(
+      `CiviCRM v3 ${entity}.${action} (multipart) failed (${res.status}): ${text}`,
+    );
+  }
+  const json = (await res.json()) as {
+    is_error?: number;
+    error_message?: string;
+    values?: T[] | Record<string, T>;
+    count?: number;
+  };
+  if (json.is_error) {
+    throw new Error(
+      `CiviCRM v3 ${entity}.${action} (multipart) error: ${json.error_message ?? "unknown"}`,
+    );
+  }
+  const raw = json.values;
+  const values: T[] = Array.isArray(raw)
+    ? raw
+    : raw && typeof raw === "object"
+    ? Object.values(raw)
+    : [];
+  return { values, count: json.count };
+}
+
 /**
 * Validate a contact checksum (cid + cs) against CiviCRM.
 *