[Meta] HTTP interfaces rework.

[azjezz]

This is a meta issue to track the rework that should be done for these interfaces and the problems they have;

[fredemmott]

Similarly to Env, we're likely to want to revisit this consistently as part of changes to entrypoint APIs as a whole - CLI, http, xbox etc.

This repository is likely to be a good place for experimenting with the http-specific parts of any proposal for that, prior to making them actually part of how entrypoints work

[azjezz]

Things i have encountered in the past few days, which should be fixed in the rework of HTTP interfaces.

1. an outgoing response should have a read+write body.
2. an incoming response should have a read only body, and support chunked responses.
3. an outgoing request should also support uploaded files, since an HTTP client could be able to upload files.

[azjezz]

as per the discussion in #39:

• headers, query parameters, and parsed body should all have type of dict<string, vec<string>> or vec<(string ,string)> to allow for duplicates, and be consistent with each other.

[fredemmott]

an outgoing response should have a read+write body.

Probably should usually have a read-write body, but streaming responses may have a write-only body by necessity

... though, with IO\pipe() + concurrent , perhaps true read-write isn't necessary. If it is, it needs to be SeekableReadWriteHandle, not just ReadWriteHandle

[azjezz]

SeekableReadWriteHandle

actually yes, it needs to also be seekable, which is what i did here: https://github.com/nuxed/nuxed/blob/develop/src/Http/Message/IResponse.hack

[azjezz]

this interfaces currently works perfectly for outgoing responses, but i'm having troubles using it for incoming responses ( with http client ), hence why i think it's better to split outgoing messages and incoming messages

[fredemmott]

uploaded files

There needs to be /a way/ to access them. I'm not sure if that belongs here, or in the frameworks

• multipart makes this complicated
• if here, it definitely needs to support lazy-loading
• even the list of file-names must be lazy-loaded
• fetching a full list of file names requires storing the files somewhere, as post data is only rewindable if it's been buffered, which may be undesirable

If this is part of the interfaces, that means that a default buffering method (e.g. file, ram) and strategy (e.g. limit to nGB/seconds) must be provided by all implementations.

The alternative would be just to provide the ReadHandle for the raw data, and let frameworks provide/let users choose between concrete implementations

[fredemmott]

That was about server requests; client requests are simpler. Probably should support either string content or ReadHandle content

[azjezz]

ReadHandle for the raw data

you mean raw data as in the whole request or post headers?

i think post headers is a good idea, but i would expect the handle to be rewindable

[fredemmott]

All non-header data.

The handle being rewindable implicitly requires local buffering, which can be undesirable when dealing with large files (e.g. video uploads)

[azjezz]

If this is part of the interfaces, that means that a default buffering method (e.g. file, ram) and strategy (e.g. limit to nGB/seconds) must be provided by all implementations.

or by HHVM, using the exact same way they are handled now, but instead of a global variable $_FILES, maybe HH\Server\get_request_files(): vec<RequestFileShape>.

this worked greatly so far in both PHP and HHVM, the server will upload the files to a temporary file each, and provide the path, name, and errors.

The handle being rewindable implicitly requires local buffering, which can be undesirable when dealing with large files (e.g. video uploads)

rewinding is also helpful in many cases.

E.g: a tool is built to deal with requests in some way where it reads the body, my framework implements the request interface with an additional method Request::toJson<T>(): Awaitable<T> , the user then calls $request->toJson<T>(); and later passes the request to the library they installed, which tries to read the body ( maybe doing some logging ), which fails.

[fredemmott]

A general alternative to using SeekableReadWriteHandle for server response bodies is to take (Awaitable<void> $producer, ReadHandle $reader)

There is then no need for seekable (edit: to implement middleware), but it's very verbose:

list($next_r, $next_w) = IO\pipe();
$produce_next = async { concurrent {
  await $producer;
  await async { while(true) {
    try {
      $chunk = await $read->readAllowPartialSuccessAsync();
      if ($chunk === '') {
        $next_w->close();
        break;
      }
      await $next_w->writeAllAsync($chunk);
  }};
 }};
 $response->setBody($next_producer, $next_r);

This allows true streaming with middleware and minimal/controllable buffering

[fredemmott]

If this is part of the interfaces, that means that a default buffering method (e.g. file, ram) and strategy (e.g. limit to nGB/seconds) must be provided by all implementations.

or by HHVM, using the exact same way they are handled now, but instead of a global variable $_FILES, maybe HH\Server\get_request_files(): vec<RequestFileShape>.

this worked greatly so far in both PHP and HHVM, the server will upload the files to a temporary file each, and provide the path, name, and errors.

Because of this implicit buffering/storage, we are unable to use $_FILES internally. It is currently provided as a convenience for external users, but it is not usable as a template for a canonical replacement.

[fredemmott]

I understand that rewinding is definitely useful - but the requirement for storage/buffering is also a cost which makes it completely unusable for some problems - both large uploads, and slow 'uploads' (e.g. push notifications via long-poll, or a hypothetical websocket implementation in hack).

Rewinding might be suitable for a higher level framework (or an additional, optional request/response type here), but it can't be a requirement for an interface that is meant to cover 100% of use cases.

While it definitely should optimize for the common cases, we still want to be able to support everything here, similarly to HSL IO.

[azjezz]

but it can't be a requirement for an interface that is meant to cover 100% of use cases.

I don't think it would cover 100% of uses cases regardless, there's always going to be edge cases, in which case, hhvm should be able to a way to access the request as-is ( full, including request line and headers ) in the current context as a read handle.