Skip to content

@tus/s3-store: failed part upload crashes entire server #722

Open
@mitjap

Description

@mitjap

When uploadPart or uploadIncompletePart throws (is rejected) entire server crashes. This happens often with Scaleway Object Storage as their service fails with error 500 with non-informative console message An error was encountered in a non-retryable streaming request. from AWS library. After listening for unhandledRejection event I got some more information.

Unhandled Rejection at: Promise {
  <rejected> S3ServiceException [InternalError]: We encountered an internal error. Please try again.
      at throwDefaultError (/app/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)
      at /app/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5
      at de_CommandError (/app/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4965:14)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async /app/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20
      at async /app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:483:18
      at async /app/node_modules/@smithy/middleware-retry/dist-cjs/index.js:321:38
      at async /app/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:315:18
      at async /app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:109:22
      at async /app/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:136:14 {
    '$fault': 'client',
    '$metadata': {
      httpStatusCode: 500,
      requestId: 'txg5c09a7e2fbe14ba89375-0067aa0421',
      extendedRequestId: 'txg5c09a7e2fbe14ba89375-0067aa0421',
      cfId: undefined
    },
    Code: 'InternalError',
    RequestId: 'txg5c09a7e2fbe14ba89375-0067aa0421',
    HostId: 'txg5c09a7e2fbe14ba89375-0067aa0421',
    Resource: '<s3-object-key>.bin'
  },

I believe the problem is that when a deferred promise is created it does not handle rejections. It is only inserted into a list.

promises.push(deferred)

At the end promises are aggregated with Promise.all and returned to the caller where an rejection handler is eventually added.
await Promise.all(promises)

return handler.send(req, res).catch(onError)

During this period if any of the promises in a list are rejected, the rejection is not handled.

Steps to reproduce

Use S3 server that sometimes throws errors (e.g. Scaleway Object Storage) or manually throw error. I modified uploadPart(metadata, readStream, partNumber) to randomly reject. Add provided snippet at the top of this function. This will result in server crash.

const errorProbability = Math.random()
if (errorProbability < 0.2) {
  return new Promise((resolve, reject) => {
    setTimeout(() => {
      reject(new Error('test'));
    }, 1000);
  })
} 

Expected behavior

Server should not crash. Failed part upload should end the current upload with HTTP 500 Internal Server Error status code.

Observation

Promises (rejections) returned by acquiredPermit?.release() and permit?.release() are also not handled and would cause server to crash.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions