Web Server route add put-file and allow get-file support streaming

[linonetwo]

Me and some user would like to add file as external attachment on TidGi Mobile.

To sync it back to desktop, nodejs server needs to support PUT file. And of course, I can add it to plugin instead of in the core, since no one else tried to add this route, means only TidGi mobile users will use it.

[netlify]

✅ Deploy Preview for tiddlywiki-previews ready!

Name	Link
🔨 Latest commit	684d5a9
🔍 Latest deploy log	https://app.netlify.com/projects/tiddlywiki-previews/deploys/684703982d7e9900088a002b
😎 Deploy Preview	https://deploy-preview-9075--tiddlywiki-previews.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[github-actions]

Confirmed: linonetwo has already signed the Contributor License Agreement (see contributing.md)

[github-actions]

📊 Build Size Comparison: empty.html

Branch	Size
Base (master)	2528.4 KB
PR	2530.7 KB

Diff: ⬆️ Increase: +2.2 KB

[linonetwo]

If you think PR title is too simple, consider #7542

[pmario]

At the moment it's not possible to upload files to the file server for security reasons. I think this should be a core plugin. So users the include the server plugin will need to take care, that no "evil" files are uploaded.

Just my thoughts.

[Jermolene]

Thanks @linonetwo, looks good. Please could you add the <<.from-version "5.3.7">> marker and I'm happy to merge.

[linonetwo]

@Jermolene Added.

@pmario Don't worry, this is as not-secure as PUT tiddler, as long as user expose it on the internet. The intended usecase is on LAN, or under a random password like #7469

[Arlen22]

Shouldn't this be writing the response stream directly to the file system rather than receiving it as a buffer first?

Something like this? (Code untested, please test first)

exports.bodyFormat = "stream";
...
const stream = fs.createWriteStream();
stream.on("error", function(){
  response.writeHead(500).end();
});
stream.on("end", function(){
  if(!response.headersSent)
    response.writeHead(200).end();
});
request.pipe(stream);

@pmario Don't worry, this is as not-secure as PUT tiddler, as long as user expose it on the internet. The intended usecase is on LAN, or under a random password like #7469

I completely agree. Full takeover of the server computer can be effected using only a JavaScript tiddler that gets loaded as a module and executed on the server. All JavaScript is fully privileged in Node TiddlyWiki, and uploading static files is far less dangerous.

[linonetwo]

@Arlen22 You are right, the buffer way is simplier, and GET was using that, but we could switch to stream, because it will support larger files like videos. And since we are using stream, how about let GET static file also use stream? And since we are using stream, how about also add Range 206
Partial Content support for video streaming.
@Jermolene I've also update that, check it again please.

And seems I'm the first one that use the bodyFormat feature.

[linonetwo]

#9078

[Arlen22]

So are we just doing buffer in this PR and then stream in a different PR? Do both PRs get merged?

[linonetwo]

@Arlen22 This also use stream. This is PUT, another is GET.

[saqimtiaz]

For reference, here is a version written a few years ago for use with the FileUploads plugin:

/*\
title: $:/plugins/sq/node-files-PUT-support/server-route-upload.js
type: application/javascript
module-type: route

PUT /^\/files\/(.+)$/

Upload binary files to `files/` directory via HTTP PUT

\*/
(function() {

/*jslint node: true, browser: true */
/*global $tw: false */
"use strict";

const fs = require("fs");
const path = require("path");

exports.method = "PUT";
exports.platforms = ["node"];
exports.path = /^\/files\/(.+)$/;
exports.bodyFormat = "stream";

exports.handler = function(request, response, state) {
	const MAX_UPLOAD_SIZE = parseInt(process.env.MAX_UPLOAD_SIZE || "10000000"); // 10MB default

	let title = state.params[0];
	try {
		title = decodeURIComponent(title);
	} catch(e) {
		response.writeHead(400, {"Content-Type": "text/plain"});
		return response.end("Invalid URL encoding.");
	}

	const basePath = path.resolve($tw.boot.wikiTiddlersPath, "../files");
	const targetPath = path.normalize(path.join(basePath, title));

	// Prevent directory traversal
	if(!targetPath.startsWith(basePath)) {
		response.writeHead(400, {"Content-Type": "text/plain"});
		return response.end("Invalid file path.");
	}

	// Ensure the target directory exists
	try {
		$tw.utils.createDirectory(path.dirname(targetPath));
	} catch(e) {
		console.error("Failed to create directory:", e);
		response.writeHead(500, {"Content-Type": "text/plain"});
		return response.end("Server error");
	}

	// Prepare to write the file
	const outStream = fs.createWriteStream(targetPath);
	let totalBytes = 0;
	let limitExceeded = false;

	request.on("data", chunk => {
		totalBytes += chunk.length;
		if(totalBytes > MAX_UPLOAD_SIZE) {
			limitExceeded = true;
			response.writeHead(413, {"Content-Type": "text/plain"});
			response.end("File too large.");
			request.destroy();
			outStream.destroy();
		}
	});

	// Pipe the stream and handle result
	request.pipe(outStream);

	outStream.on("finish", () => {
		if(limitExceeded) return;
		console.log(`External file saved: ${title}`);
		response.setHeader("Content-Type", "application/json");
		response.end(JSON.stringify({
			status: "204",
			title: title
		}));
	});

	outStream.on("error", err => {
		console.error("Write stream error:", err);
		response.writeHead(500, {"Content-Type": "text/plain"});
		response.end("Server error");
	});
};

}());