diff --git a/src/content/docs/workers/static-assets/direct-upload.mdx b/src/content/docs/workers/static-assets/direct-upload.mdx new file mode 100644 index 00000000000000..ffb248d2d60f8d --- /dev/null +++ b/src/content/docs/workers/static-assets/direct-upload.mdx @@ -0,0 +1,383 @@ +--- +pcx_content_type: concept +title: Direct Uploads +sidebar: + order: 11 +head: [] +description: Upload assets through the Workers API. +--- + +import { + Badge, + Description, + FileTree, + InlineBadge, + Render, + TabItem, + Tabs, +} from "~/components"; + +:::note + +Directly uploading assets via APIs is an advanced approach that most users will not need. Instead, we encourage users to deploy your Worker with [Wrangler](/workers/static-assets/get-started/#1-create-a-new-worker-project-using-the-cli). + +::: + +Our API empowers users to upload and include static assets as part of a Worker. These static assets can be served for free, and additionally, users can also fetch assets through an optional [assets binding](/workers/static-assets/binding/) to power more advanced applications. This guide will describe the process for attaching assets to your Worker directly with the API. + +```mermaid +sequenceDiagram + participant User + participant Workers API + User<<->>Workers API: Submit manifest
POST /client/v4/accounts/:accountId/workers/scripts/:scriptName/assets-upload-session + User<<->>Workers API: Upload files
POST /client/v4/accounts/:accountId/workers/assets/upload?base64=true + User<<->>Workers API: Upload script version
PUT /client/v4/accounts/:accountId/workers/scripts/:scriptName +``` + +The asset upload flow can be distilled into three distinct phases: + +1. Registration of a manifest +2. Upload of the assets +3. Deployment of the Worker + +## Upload manifest + +The asset manifest is a ledger which keeps track of files we want to use in our Worker. This manifest is used to track assets associated with each Worker version, and eliminate the need to upload unchanged files prior to a new upload. + +The [manifest upload request](/api/operations/worker-script-update-create-assets-upload-session) describes each file which we intend to upload. Each file is its own key representing the file path and name, and is an object which contains metadata about the file. + +`hash` represents a 32 hexadecimal character hash of the file, while `size` is the size (in bytes) of the file. + +```bash +curl -X POST https://api.cloudflare.com/client/v4/accounts/{account_id}/workers/scripts/{script_name}/assets-upload-session \ +--header 'content-type: application/json' \ +--header 'Authorization: Bearer ' \ +--data '{ + "manifest": { + "/filea.html": { + "hash": "08f1dfda4574284ab3c21666d1", + "size": 12 + }, + "/fileb.html": { + "hash": "4f1c1af44620d531446ceef93f", + "size": 23 + }, + "/filec.html": { + "hash": "54995e302614e0523757a04ec1", + "size": 23 + } + } +}' +``` + +The resulting response will contain a JWT, which provides authentication during file upload. The JWT is valid for one hour. + +In addition to the JWT, the response instructs users how to optimally batch upload their files. These instructions are encoded in the `buckets` field. Each array in `buckets` contains a list of file hashes which should be uploaded together. Hashes of files that have been recently uploaded may not be returned in the API response; they do not need to be re-uploaded. + +```json +{ + "result": { + "jwt": "", + "buckets": [ + ["08f1dfda4574284ab3c21666d1", "4f1c1af44620d531446ceef93f"], + ["54995e302614e0523757a04ec1"] + ] + }, + "success": true, + "errors": null, + "messages": null +} +``` + +:::note + +If all assets have been previously uploaded, `buckets` will be empty, and `jwt` will contain a completion token. Uploading files is not necessary, and you can skip directly to [uploading a new script or version](/workers/static-assets/direct-upload/#createdeploy-new-version). + +::: + +### Limitations + +- Each file must be under 25 MiB +- The overall manifest must not contain more than 20,000 file entries + +## Upload Static Assets + +The [file upload API](/api/operations/worker-assets-upload) requires files be uploaded using `multipart/form-data`. The contents of each file must be base64 encoded, and the `base64` query parameter in the URL must be set to `true`. + +The `Authorization` header must be provided as a bearer token, using the JWT (upload token) from the aforementioned manifest upload call. + +Once every file in the manifest has been uploaded, a status code of 201 will be returned, with the `jwt` field present. This JWT is a final "completion" token which can be used to create a deployment of a Worker with this set of assets. This completion token is valid for 1 hour. + +## Create/Deploy New Version + +[Script](/api/operations/worker-script-upload-worker-module) and [version](/api/operations/worker-versions-upload-version) upload endpoints require specifying a metadata part in the form data. Here, we can provide the completion token from the previous (upload assets) step. + +```bash title="Example Worker Metadata Specifying Completion Token" +{ + "main_module": "main.js", + "assets": { + "jwt": "" + }, + "compatibility_date": "2021-09-14" +} +``` + +If this is a Worker which already has assets, and you wish to just re-use the existing set of assets, we do not have to specify the completion token again. Instead, we can pass the boolean `keep_assets` option. + +```bash title="Example Worker Metadata Specifying keep_assets" +{ + "main_module": "main.js", + "keep_assets": true, + "compatibility_date": "2021-09-14" +} +``` + +Asset [routing configuration](/workers/static-assets/routing/#routing-configuration) can be provided in the `assets` object, such as `html_handling` and `not_found_handling`. + +```bash title="Example Worker Metadata Specifying Asset Configuration" +{ + "main_module": "main.js", + "assets": { + "jwt": "", + "config" { + "html_handling": "auto-trailing-slash" + } + }, + "compatibility_date": "2021-09-14" +} +``` + +Optionally, an assets binding can be provided if you wish to fetch and serve assets from within your Worker code. + +```bash title="Example Worker Metadata Specifying Asset Binding" +{ + "main_module": "main.js", + "assets": { + ... + }, + "bindings": [ + ... + { + "name": "ASSETS", + "type": "assets" + } + ... + ] + "compatibility_date": "2021-09-14" +} +``` + +## Programmatic Example + + + +```ts +import * as fs from "fs"; +import * as path from "path"; +import * as crypto from "crypto"; +import { FormData, fetch } from "undici"; +import "node:process"; + +const accountId: string = ""; // Replace with your actual account ID +const filesDirectory: string = "assets"; // Adjust to your assets directory +const scriptName: string = "my-new-script"; // Replace with desired script name + +interface FileMetadata { + hash: string; + size: number; +} + +interface UploadSessionData { + uploadToken: string; + buckets: string[][]; + fileMetadata: Record; +} + +interface UploadResponse { + result: { + jwt: string; + buckets: string[][]; + }; + success: boolean; + errors: any; + messages: any; +} + +// Function to calculate the SHA-256 hash of a file and truncate to 32 characters +function calculateFileHash(filePath: string): { + fileHash: string; + fileSize: number; +} { + const hash = crypto.createHash("sha256"); + const fileBuffer = fs.readFileSync(filePath); + hash.update(fileBuffer); + const fileHash = hash.digest("hex").slice(0, 32); // Grab the first 32 characters + const fileSize = fileBuffer.length; + return { fileHash, fileSize }; +} + +// Function to gather file metadata for all files in the directory +function gatherFileMetadata(directory: string): Record { + const files = fs.readdirSync(directory); + const fileMetadata: Record = {}; + + files.forEach((file) => { + const filePath = path.join(directory, file); + const { fileHash, fileSize } = calculateFileHash(filePath); + fileMetadata["/" + file] = { + hash: fileHash, + size: fileSize, + }; + }); + + return fileMetadata; +} + +function findMatch( + fileHash: string, + fileMetadata: Record, +): string { + for (let prop in fileMetadata) { + const file = fileMetadata[prop] as FileMetadata; + if (file.hash === fileHash) { + return prop; + } + } + throw new Error("unknown fileHash"); +} + +// Function to upload a batch of files using the JWT from the first response +async function uploadFilesBatch( + jwt: string, + fileHashes: string[][], + fileMetadata: Record, +): Promise { + const form = new FormData(); + + for (const bucket of fileHashes) { + bucket.forEach((fileHash) => { + const fullPath = findMatch(fileHash, fileMetadata); + const relPath = filesDirectory + "/" + path.basename(fullPath); + const fileBuffer = fs.readFileSync(relPath); + const base64Data = fileBuffer.toString("base64"); // Convert file to Base64 + + form.append( + fileHash, + new File([base64Data], fileHash, { + type: "text/html", // Modify Content-Type header based on type of file + }), + fileHash, + ); + }); + + const response = await fetch( + `https://api.cloudflare.com/client/v4/accounts/${accountId}/workers/assets/upload?base64=true`, + { + method: "POST", + headers: { + Authorization: `Bearer ${jwt}`, + }, + body: form, + }, + ); + + const data = (await response.json()) as UploadResponse; + if (data && data.result.jwt) { + return data.result.jwt; + } + } + + throw new Error("Should have received completion token"); +} + +async function scriptUpload(completionToken: string): Promise { + const form = new FormData(); + + // Configure metadata + form.append( + "metadata", + JSON.stringify({ + main_module: "index.mjs", + compatibility_date: "2022-03-11", + assets: { + jwt: completionToken, // Provide the completion token from file uploads + }, + bindings: [{ name: "ASSETS", type: "assets" }], // Optional assets binding to fetch from user worker + }), + ); + + // Configure (optional) user worker + form.append( + "index.js", + new File( + [ + "export default {async fetch(request, env) { return new Response('Hello world from user worker!'); }}", + ], + "index.mjs", + { + type: "application/javascript+module", + }, + ), + ); + + const response = await fetch( + `https://api.cloudflare.com/client/v4/accounts/${accountId}/workers/scripts/${scriptName}`, + { + method: "PUT", + headers: { + Authorization: `Bearer ${process.env.CLOUDFLARE_API_TOKEN}`, + }, + body: form, + }, + ); + + if (response.status != 200) { + throw new Error("unexpected status code"); + } +} + +// Function to make the POST request to start the assets upload session +async function startUploadSession(): Promise { + const fileMetadata = gatherFileMetadata(filesDirectory); + + const requestBody = JSON.stringify({ + manifest: fileMetadata, + }); + + const response = await fetch( + `https://api.cloudflare.com/client/v4/accounts/${accountId}/workers/scripts/${scriptName}/assets-upload-session`, + { + method: "POST", + headers: { + Authorization: `Bearer ${process.env.CLOUDFLARE_API_TOKEN}`, + "Content-Type": "application/json", + }, + body: requestBody, + }, + ); + + const data = (await response.json()) as UploadResponse; + const jwt = data.result.jwt; + + return { + uploadToken: jwt, + buckets: data.result.buckets, + fileMetadata, + }; +} + +// Begin the upload session by uploading a new manifest +const { uploadToken, buckets, fileMetadata } = await startUploadSession(); + +// If all files are already uploaded, a completion token will be immediately returned. Otherwise, +// we should upload the missing files +let completionToken = uploadToken; +if (buckets.length > 0) { + completionToken = await uploadFilesBatch(uploadToken, buckets, fileMetadata); +} + +// Once we have uploaded all of our files, we can upload a new script, and assets, with completion token +await scriptUpload(completionToken); +``` + +