Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exceptions thrown in a worker's setup function don't fail the test until it times out #29777

Closed
andreubotella opened this issue Jul 25, 2021 · 3 comments · Fixed by #29970
Closed
Labels

Comments

@andreubotella
Copy link
Member

andreubotella commented Jul 25, 2021

While working on implementing classic workers in Deno in order to run the worker WPT tests, I noticed that an exception thrown in a worker's setup function (in this case, in FileAPI/FileReaderSync.worker.html, since Deno doesn't yet support FileReaderSync), will not fail the test immediately. Instead, the test only shows up as failed with the proper error message after it would have timed out.

In the case of Deno, which is recognized as a ShellTestEnvironment, this is bad because WPT doesn't time out tests in shell environments, so the test will run forever, and it will never call the completion callback.

The issue here is that, once the completion message from the worker reaches the window environment, Tests.prototype.complete is only called if Tests.prototype.all_done returns true, which doesn't seem to be the case for exceptions thrown during a worker's setup.

@andreubotella
Copy link
Member Author

andreubotella commented Aug 9, 2021

As far as I can tell, this is caused because Tests.prototype.all_done includes the condition that the number of tests isn't zero. This condition is obviously not met when an exception is thrown in setup, but the reason why that exception is reported as an error immediately is because (single-realm) window tests seem to have a specific code path for early errors, which doesn't call tests.all_done(), whereas worker tests (and perhaps also multi-realm tests, though I haven't investigated this in depth) use the main code path, that doesn't seem to expect such early errors.

It seems like one of the cases where the this.tests.length > 0 condition is needed are service workers, since their registration is asynchronous and might happen after the load event. This condition then keeps the test running until fetch_tests_from_worker is called. An easy way to solve this would be to have fetch_tests_from_worker take a promise or an async function, which would insert a dummy pending remote that will complete when the promise resolves.

There are probably other cases where that condition is needed.

@andreubotella
Copy link
Member Author

andreubotella commented Aug 10, 2021

The same thing happens if you reject a remote promise_setup test, but in this case this isn't caused by the this.tests.length > 0 condition, so it's better to open a new issue for it (#29969).

@andreubotella
Copy link
Member Author

andreubotella commented Aug 11, 2021

I'll be documenting here the breakage I find from changing this in #29970:

  • The following tests must run a test after the window's load event fires, and relies on the test not completing because no tests have been run yet. It seems like abstracting things like tests.num_pending and tests.pending_remotes into a more general (promise-based?; see Allow unrestricted Promise usage in testharness.js rfcs#93) way to delay a test's completion would solve this.

    • /content-dpr/image-with-dpr-header.html
    • /content-security-policy/form-action/form-action-src-allowed-target-blank.sub.html
    • /content-security-policy/form-action/form-action-src-allowed-target-frame.sub.html
    • /content-security-policy/form-action/form-action-src-redirect-allowed-target-blank.sub.html
    • /content-security-policy/form-action/form-action-src-redirect-allowed-target-frame.sub.html
    • /content-security-policy/generic/generic-0_8_1.sub.html
    • /css/css-images/image-orientation/image-orientation-from-image-computed-style.html
    • /css/css-images/image-orientation/image-orientation-none-computed-style.html
    • /css/css-overflow/overflow-padding.html
    • /css/css-scrollbars/scrollbar-width-001.html
    • /css/css-scrollbars/scrollbar-width-002.html
    • /css/css-scrollbars/scrollbar-width-003.html
    • /css/css-scrollbars/scrollbar-width-004.html
    • /css/css-sizing/aspect-ratio/quirks-mode-001.html
    • /css/css-sizing/aspect-ratio/quirks-mode-002.html
    • /css/css-sizing/aspect-ratio/quirks-mode-003.html
    • /css/css-sizing/aspect-ratio/replaced-element-028.html
    • /css/css-tables/html5-table-formatting-3.html
    • /css/cssom/ttwf-cssom-doc-ext-load-tree-order.html
    • /css/cssom-view/devicePixelRatio-undisplayed-iframe.tentative.html
    • /dom/events/document-level-touchmove-event-listener-passive-by-default.tentative.html
    • /dom/events/document-level-wheel-event-listener-passive-by-default.tentative.html
    • /dom/events/scrolling/input-text-scroll-event-when-using-arrow-keys.html
    • /encoding/legacy-mb-japanese/euc-jp/eucjp-decode-cseucpkdfmtjapanese.html
    • /encoding/legacy-mb-japanese/euc-jp/eucjp-decode-errors.html
    • /encoding/legacy-mb-japanese/euc-jp/eucjp-decode-x-euc-jp.html
    • /encoding/legacy-mb-japanese/euc-jp/eucjp-decode.html
    • /encoding/legacy-mb-japanese/iso-2022-jp/iso2022jp-decode-csiso2022jp.html
    • /encoding/legacy-mb-japanese/iso-2022-jp/iso2022jp-decode-errors.html
    • /encoding/legacy-mb-japanese/iso-2022-jp/iso2022jp-decode.html
    • /encoding/legacy-mb-japanese/shift_jis/sjis-decode-csshiftjis.html
    • /encoding/legacy-mb-japanese/shift_jis/sjis-decode-errors.html
    • /encoding/legacy-mb-japanese/shift_jis/sjis-decode-ms932.html
    • /encoding/legacy-mb-japanese/shift_jis/sjis-decode-ms_kanji.html
    • /encoding/legacy-mb-japanese/shift_jis/sjis-decode-shift-jis.html
    • /encoding/legacy-mb-japanese/shift_jis/sjis-decode-sjis.html
    • /encoding/legacy-mb-japanese/shift_jis/sjis-decode-windows-31j.html
    • /encoding/legacy-mb-japanese/shift_jis/sjis-decode-x-sjis.html
    • /encoding/legacy-mb-japanese/shift_jis/sjis-decode.html
    • /encoding/legacy-mb-korean/euc-kr/euckr-decode-cseuckr.html
    • /encoding/legacy-mb-korean/euc-kr/euckr-decode-csksc56011987.html
    • /encoding/legacy-mb-korean/euc-kr/euckr-decode-errors.html
    • /encoding/legacy-mb-korean/euc-kr/euckr-decode-iso-ir-149.html
    • /encoding/legacy-mb-korean/euc-kr/euckr-decode-korean.html
    • /encoding/legacy-mb-korean/euc-kr/euckr-decode-ks_c_5601-1987.html
    • /encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc5601.html
    • /encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html
    • /encoding/legacy-mb-korean/euc-kr/euckr-decode-windows-949.html
    • /encoding/legacy-mb-korean/euc-kr/euckr-decode.html
    • /encoding/legacy-mb-tchinese/big5/big5-decode-big5-hkscs.html
    • /encoding/legacy-mb-tchinese/big5/big5-decode-cn-big5.html
    • /encoding/legacy-mb-tchinese/big5/big5-decode-csbig5.html
    • /encoding/legacy-mb-tchinese/big5/big5-decode-errors.html
    • /encoding/legacy-mb-tchinese/big5/big5-decode-extra.html
    • /encoding/legacy-mb-tchinese/big5/big5-decode-x-x-big5.html
    • /encoding/legacy-mb-tchinese/big5/big5-decode.html
    • /fetch/redirect-navigate/preserve-fragment.html
    • /font-access/font_access-chooser-selection.tentative.manual.https.html
    • /html/browsers/history/the-location-interface/location-protocol-setter-non-broken-weird.html
    • /html/browsers/history/the-location-interface/location-protocol-setter-non-broken.html
  • These tests don't seem to obviously depend on the load event, but still are registering tests after it.

    • /content-security-policy/navigation/javascript-url-navigation-inherits-csp.html (creates a test after a timeout)
    • /content-security-policy/script-src/worker-data-set-timeout.sub.html (creates a test after a fetch completes)
    • /css/css-position/sticky/position-sticky-scrolled-remove-sibling.html (creates a test inside requestAnimationFrame)
    • /css/cssom-view/getBoundingClientRect-empty-inline.html (creates a test after document.fonts.ready resolves)
    • /css/cssom-view/scrollIntoView-smooth.html (creates a test inside requestAnimationFrame)
    • /fetch/metadata/font.https.sub.html (creates a test after document.fonts.ready resolves)
    • /fetch/metadata/sharedworker.https.sub.html (creates tests after a shared worker receives error and message events)
    • /fetch/metadata/xslt.https.sub.html (creates tests after the window receives a message event)
    • /html/browsers/the-window-object/apis-for-creating-and-navigating-browsing-contexts-by-name/open-features-negative-innerwidth-innerheight.html (creates tests after the window receives a message event)
    • /html/browsers/the-window-object/apis-for-creating-and-navigating-browsing-contexts-by-name/open-features-negative-screenx-screeny.html (creates tests after the window receives a message event)
    • /html/browsers/the-window-object/apis-for-creating-and-navigating-browsing-contexts-by-name/open-features-negative-top-left.html (creates tests after the window receives a message event)
    • /html/browsers/the-window-object/apis-for-creating-and-navigating-browsing-contexts-by-name/open-features-negative-width-height.html (creates tests after the window receives a message event)
    • /html/browsers/the-window-object/apis-for-creating-and-navigating-browsing-contexts-by-name/open-features-non-integer-height.html (creates tests after the window receives a message event)
    • /html/browsers/the-window-object/apis-for-creating-and-navigating-browsing-contexts-by-name/open-features-non-integer-innerheight.html (creates tests after the window receives a message event)
    • /html/browsers/the-window-object/apis-for-creating-and-navigating-browsing-contexts-by-name/open-features-non-integer-innerwidth.html (creates tests after the window receives a message event)
    • /html/browsers/the-window-object/apis-for-creating-and-navigating-browsing-contexts-by-name/open-features-non-integer-left.html (creates tests after the window receives a message event)
    • /html/browsers/the-window-object/apis-for-creating-and-navigating-browsing-contexts-by-name/open-features-non-integer-screenx.html (creates tests after the window receives a message event)
    • /html/browsers/the-window-object/apis-for-creating-and-navigating-browsing-contexts-by-name/open-features-non-integer-screeny.html (creates tests after the window receives a message event)
    • /html/browsers/the-window-object/apis-for-creating-and-navigating-browsing-contexts-by-name/open-features-non-integer-top.html (creates tests after the window receives a message event)
    • /html/browsers/the-window-object/apis-for-creating-and-navigating-browsing-contexts-by-name/open-features-non-integer-width.html (creates tests after the window receives a message event)
  • These are service worker tests which use their own HTML boilerplate, and since SW registration is async, they end up registering the RemoteContext after the load event fires. Most of these tests can be trivially changed to pass the async function which registers the SW to fetch_tests_from_workers. This change is already implemented in my PR for the autogenerated HTML boilerplate of SW tests, and so those tests aren't listed below.

    • /content-security-policy/securitypolicyviolation/inside-service-worker.https.html
  • These tests clearly fail as a result of these changes, but it's not clear to me why.

    • /content-security-policy/navigate-to/meta-refresh-allowed.html
    • /content-security-policy/navigate-to/meta-refresh-redirected-allowed.html

Edit: Given the many tests that break, it's probably best to go back to the drawing board and try something else.


Edit 2: I've since repurposed PR #29970 to have a different fix to this problem which doesn't seem to break any tests.

andreubotella pushed a commit that referenced this issue Aug 11, 2021
…ontexts

`Tests.prototype.all_done` used to treat the presence of at least one
test as a necessary condition for all tests to be done. This would be
false in cases where an exception is thrown inside the `setup` function,
but the code path that runs in that case doesn't use the `all_done()`
function.

However, if an exception is thrown inside the `setup` function in a
worker test, `tests.all_done()` is called in that code path, which
results in the completion message from the worker being ignored. This
change fixes this by changing the condition to require the presence of
either tests or remote contexts.

For service worker tests, however, that is not enough, since service
worker registration is asynchronous. This change lets the
`fetch_tests_from_worker` function take a promise or an async function
that returns the worker, and it guarantees that the test won't be
considered complete until the promise is resolved. The HTML boilerplate
for service worker tests is also changed in turn.

Closes #29777.
andreubotella pushed a commit that referenced this issue Aug 11, 2021
…remotes

`Tests.prototype.all_done` used to treat the presence of at least one
test as a necessary condition for all tests to be done. This would be
false in cases where an exception is thrown inside the `setup` function,
but the code path that runs in that case doesn't use the `all_done()`
function.

However, if an exception is thrown inside the `setup` function in a
worker test, `tests.all_done()` is called in that code path, which
results in the completion message from the worker being ignored. This
change fixes this by changing the condition to require the presence of
either tests or remote contexts.

For service worker tests, however, that is not enough, since service
worker registration is asynchronous. This change lets the
`fetch_tests_from_worker` function take a promise or an async function
that returns the worker, and it guarantees that the test won't be
considered complete until the promise is resolved. The HTML boilerplate
for service worker tests is also changed in turn.

Closes #29777.
andreubotella pushed a commit that referenced this issue Aug 31, 2021
…remotes

`Tests.prototype.all_done` used to treat the presence of at least one
test as a necessary condition for all tests to be done. This would be
false in cases where an exception is thrown inside the `setup` function,
but the code path that runs in that case doesn't use the `all_done()`
function.

However, if an exception is thrown inside the `setup` function in a
worker test, `tests.all_done()` is called in that code path, which
results in the completion message from the worker being ignored. This
change fixes this by changing the condition to require the presence of
either tests or remote contexts.

For service worker tests, however, that is not enough, since service
worker registration is asynchronous. This change lets the
`fetch_tests_from_worker` function take a promise or an async function
that returns the worker, and it guarantees that the test won't be
considered complete until the promise is resolved. The HTML boilerplate
for service worker tests is also changed in turn.

Closes #29777.
andreubotella pushed a commit that referenced this issue Sep 10, 2021
…remotes

`Tests.prototype.all_done` used to treat the presence of at least one
test as a necessary condition for all tests to be done. This would be
false in cases where an exception is thrown inside the `setup` function,
but the code path that runs in that case doesn't use the `all_done()`
function.

However, if an exception is thrown inside the `setup` function in a
worker test, `tests.all_done()` is called in that code path, which
results in the completion message from the worker being ignored. This
change fixes this by changing the condition to require the presence of
either tests or remote contexts.

Closes #29777.
jgraham pushed a commit that referenced this issue Sep 13, 2021
…remotes

`Tests.prototype.all_done` used to treat the presence of at least one
test as a necessary condition for all tests to be done. This would be
false in cases where an exception is thrown inside the `setup` function,
but the code path that runs in that case doesn't use the `all_done()`
function.

However, if an exception is thrown inside the `setup` function in a
worker test, `tests.all_done()` is called in that code path, which
results in the completion message from the worker being ignored. This
change fixes this by changing the condition to require the presence of
either tests or remote contexts.

Closes #29777.
andreubotella pushed a commit to andreubotella/deno that referenced this issue Sep 25, 2021
Classic workers were implemented in denoland#11338, which also enabled the WPT
tests in the `workers` directory. However, the rest of WPT worker tests
were not enabled because a number of them were hanging due to
web-platform-tests/wpt#29777. Now that that WPT issue is fixed, the bulk
of worker tests can be enabled.

There are still a few tests that hang, and so haven't been enabled. In
particular:

- The following tests seem to hang because a promise fails to resolve.
  We can detect such cases in non-worker tests because the process will
  exit without calling the WPT completion callback, but in worker tests
  the worker message ops will keep the event loop running. This will be
  fixed when we add timeouts to WPT tests (denoland#9460).

  - `/fetch/api/basic/error-after-response.any.worker.html`
  - `/html/webappapis/microtask-queuing/queue-microtask-exceptions.any.worker.html`
  - `/webmessaging/message-channels/worker-post-after-close.any.worker.html`
  - `/webmessaging/message-channels/worker.any.worker.html`
  - `/websockets/Create-on-worker-shutdown.any.worker.html`

- The following tests apparently hang because a promise rejection is
  never handled, which will kill the process in the main thread but not
  in workers (denoland#12221).

  - `/streams/readable-streams/async-iterator.any.worker.html`
  - `/workers/interfaces/WorkerUtils/importScripts/report-error-setTimeout-cross-origin.sub.any.worker.html`
  - `/workers/interfaces/WorkerUtils/importScripts/report-error-setTimeout-redirect-to-cross-origin.sub.any.worker.html`
  - `/workers/interfaces/WorkerUtils/importScripts/report-error-setTimeout-same-origin.sub.any.worker.html`
andreubotella pushed a commit to andreubotella/deno that referenced this issue Oct 7, 2021
Classic workers were implemented in denoland#11338, which also enabled the WPT
tests in the `workers` directory. However, the rest of WPT worker tests
were not enabled because a number of them were hanging due to
web-platform-tests/wpt#29777. Now that that WPT issue is fixed, the bulk
of worker tests can be enabled.

There are still a few tests that hang, and so haven't been enabled. In
particular:

- The following tests seem to hang because a promise fails to resolve.
  We can detect such cases in non-worker tests because the process will
  exit without calling the WPT completion callback, but in worker tests
  the worker message ops will keep the event loop running. This will be
  fixed when we add timeouts to WPT tests (denoland#9460).

  - `/fetch/api/basic/error-after-response.any.worker.html`
  - `/html/webappapis/microtask-queuing/queue-microtask-exceptions.any.worker.html`
  - `/webmessaging/message-channels/worker-post-after-close.any.worker.html`
  - `/webmessaging/message-channels/worker.any.worker.html`
  - `/websockets/Create-on-worker-shutdown.any.worker.html`

- The following tests apparently hang because a promise rejection is
  never handled, which will kill the process in the main thread but not
  in workers (denoland#12221).

  - `/streams/readable-streams/async-iterator.any.worker.html`
  - `/workers/interfaces/WorkerUtils/importScripts/report-error-setTimeout-cross-origin.sub.any.worker.html`
  - `/workers/interfaces/WorkerUtils/importScripts/report-error-setTimeout-redirect-to-cross-origin.sub.any.worker.html`
  - `/workers/interfaces/WorkerUtils/importScripts/report-error-setTimeout-same-origin.sub.any.worker.html`
bartlomieju pushed a commit to denoland/deno that referenced this issue Oct 8, 2021
Classic workers were implemented in #11338, which also enabled the WPT
tests in the `workers` directory. However, the rest of WPT worker tests
were not enabled because a number of them were hanging due to
web-platform-tests/wpt#29777. Now that that WPT issue is fixed, the bulk
of worker tests can be enabled.

There are still a few tests that hang, and so haven't been enabled. In
particular:

- The following tests seem to hang because a promise fails to resolve.
  We can detect such cases in non-worker tests because the process will
  exit without calling the WPT completion callback, but in worker tests
  the worker message ops will keep the event loop running. This will be
  fixed when we add timeouts to WPT tests (#9460).

  - `/fetch/api/basic/error-after-response.any.worker.html`
  - `/html/webappapis/microtask-queuing/queue-microtask-exceptions.any.worker.html`
  - `/webmessaging/message-channels/worker-post-after-close.any.worker.html`
  - `/webmessaging/message-channels/worker.any.worker.html`
  - `/websockets/Create-on-worker-shutdown.any.worker.html`

- The following tests apparently hang because a promise rejection is
  never handled, which will kill the process in the main thread but not
  in workers (#12221).

  - `/streams/readable-streams/async-iterator.any.worker.html`
  - `/workers/interfaces/WorkerUtils/importScripts/report-error-setTimeout-cross-origin.sub.any.worker.html`
  - `/workers/interfaces/WorkerUtils/importScripts/report-error-setTimeout-redirect-to-cross-origin.sub.any.worker.html`
  - `/workers/interfaces/WorkerUtils/importScripts/report-error-setTimeout-same-origin.sub.any.worker.html`
bartlomieju pushed a commit to bartlomieju/deno that referenced this issue Oct 10, 2021
Classic workers were implemented in denoland#11338, which also enabled the WPT
tests in the `workers` directory. However, the rest of WPT worker tests
were not enabled because a number of them were hanging due to
web-platform-tests/wpt#29777. Now that that WPT issue is fixed, the bulk
of worker tests can be enabled.

There are still a few tests that hang, and so haven't been enabled. In
particular:

- The following tests seem to hang because a promise fails to resolve.
  We can detect such cases in non-worker tests because the process will
  exit without calling the WPT completion callback, but in worker tests
  the worker message ops will keep the event loop running. This will be
  fixed when we add timeouts to WPT tests (denoland#9460).

  - `/fetch/api/basic/error-after-response.any.worker.html`
  - `/html/webappapis/microtask-queuing/queue-microtask-exceptions.any.worker.html`
  - `/webmessaging/message-channels/worker-post-after-close.any.worker.html`
  - `/webmessaging/message-channels/worker.any.worker.html`
  - `/websockets/Create-on-worker-shutdown.any.worker.html`

- The following tests apparently hang because a promise rejection is
  never handled, which will kill the process in the main thread but not
  in workers (denoland#12221).

  - `/streams/readable-streams/async-iterator.any.worker.html`
  - `/workers/interfaces/WorkerUtils/importScripts/report-error-setTimeout-cross-origin.sub.any.worker.html`
  - `/workers/interfaces/WorkerUtils/importScripts/report-error-setTimeout-redirect-to-cross-origin.sub.any.worker.html`
  - `/workers/interfaces/WorkerUtils/importScripts/report-error-setTimeout-same-origin.sub.any.worker.html`
Gabisampaio pushed a commit to Gabisampaio/wpt that referenced this issue Nov 18, 2021
…remotes

`Tests.prototype.all_done` used to treat the presence of at least one
test as a necessary condition for all tests to be done. This would be
false in cases where an exception is thrown inside the `setup` function,
but the code path that runs in that case doesn't use the `all_done()`
function.

However, if an exception is thrown inside the `setup` function in a
worker test, `tests.all_done()` is called in that code path, which
results in the completion message from the worker being ignored. This
change fixes this by changing the condition to require the presence of
either tests or remote contexts.

Closes web-platform-tests#29777.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant