refactor: switch Builder Stream Templates to SchemaForm (#16468)

## What
Adds support for dynamic streams (i.e. `Stream Templates` in the UI) to the schema-form Builder feature branch.

## How
Similar to the other PRs in the stack, switches logic from the old `formValues` to the new `manifest` field in Builder state which SchemaForm writes to.

This PR also involves shifting `generatedStreams`, an ephemeral form state, to the top level of `BuilderState` so that it is no longer part of `formValues`, and updating things accordingly.

---

To achieve the generated streams view, I had to have a nested SchemaForm since `generatedStreams` are not actually part of the manifest (they are ephemeral). This required resolving the refs in the schema so that I could point it to the `DeclarativeStream` definition in the schema. If I didn't resolve first, then there would be unresolved refs in that definition which SchemaForm couldn't resolve since it wasn't given the full schema.

Since resolving all refs in the schema is somewhat slow, I chose to put that into a webworker and show a loading icon in the meantime so that it could just run in the background. Once it has been resolved once, the result is cached and therefore doesn't need to be resolved again.

---

The generated stream view used to put everything inside of a disabled fieldset, but this wasn't a great approach since it also disabled things like `Advanced` collapsibles, so users wouldn't be able to see what was inside of those. It also wasn't obvious that the controls themselves were disabled since they weren't styled any differently - they were just uninteractible, which made it feel somewhat broken.

I took a different approach here, where I added a `disableFormControls` prop to `SchemaForm` and then passed this through as a `disabled` prop on each form control/button/checkbox. This way, we actually use the form controls' own `disabled` styles which is a lot clearer and still allows things like `Advanced` collapsibles to be interacted with.

## Testing
To test, I recommend following the guide at https://docs.airbyte.com/platform/next/connector-development/connector-builder-ui/stream-templates - some of it might be slightly outdated from the changes made in this PR but it still should be pretty consistent.

## Recommended reading order
1. `oss/airbyte-webapp/src/components/connectorBuilder/Builder/DynamicStreamConfigView.tsx` main changes for the dynamic stream view
2. `oss/airbyte-webapp/src/components/connectorBuilder/Builder/GeneratedStreamView.tsx` and `oss/airbyte-webapp/src/core/services/connectorBuilder/SchemaResolverWorker.ts` - changes to achieve rendering the generated stream view with a nested SchemaForm and the web-worker needed to resolve the schema
3. `oss/airbyte-webapp/src/components/connectorBuilder/Builder/StreamConfigView.tsx`
4. `oss/airbyte-webapp/src/components/forms/SchemaForm/*` - add ability to disable all controls

Author: lmossman

airbyte-webapp/src/components/connectorBuilder/Builder/Builder.tsx

@@ -19,6 +19,7 @@ import styles from "./Builder.module.scss";
 import { BuilderSidebar } from "./BuilderSidebar";
 import { ComponentsView } from "./ComponentsView";
 import { DynamicStreamConfigView } from "./DynamicStreamConfigView";
+import { GeneratedStreamView } from "./GeneratedStreamView";
 import { GlobalConfigView } from "./GlobalConfigView";
 import { InputForm, newInputInEditing } from "./InputsForm";
 import { InputsView } from "./InputsView";
@@ -36,16 +37,17 @@ function getView(selectedView: BuilderState["view"], scrollToTop: () => void) {
     case "components":
       return <ComponentsView />;
     case "dynamic_stream":
-      return <DynamicStreamConfigView key={selectedView.index} streamId={selectedView} scrollToTop={scrollToTop} />;
+      return <DynamicStreamConfigView key={selectedView.index} streamId={selectedView} />;
     case "stream":
-    case "generated_stream":
       return (
         <StreamConfigView
           streamId={selectedView}
           key={`${selectedView.type}-${selectedView.index}`}
           scrollToTop={scrollToTop}
         />
       );
+    case "generated_stream":
+      return <GeneratedStreamView streamId={selectedView} scrollToTop={scrollToTop} />;
     default:
       assertNever(selectedView);
   }

airbyte-webapp/src/components/connectorBuilder/Builder/BuilderSidebar.module.scss

@@ -109,3 +109,7 @@ $buttonWidth: 26px;
   height: $buttonWidth !important;
   border-radius: 50%;
 }
+
+.streamTemplateList {
+  min-height: 70px;
+}

airbyte-webapp/src/components/connectorBuilder/Builder/BuilderSidebar.tsx

@@ -115,7 +115,7 @@ const DynamicStreamViewButton: React.FC<DynamicStreamViewButtonProps> = ({ name,
         }}
       >
         <FlexContainer className={styles.streamViewButtonContent} alignItems="center">
-          {generatedStreams && (
+          {generatedStreams && generatedStreams.length > 0 && (
             <Icon type={isOpen ? "chevronDown" : "chevronRight"} onClick={() => setIsOpen((isOpen) => !isOpen)} />
           )}
           {name && name.trim() ? (
@@ -127,7 +127,7 @@ const DynamicStreamViewButton: React.FC<DynamicStreamViewButtonProps> = ({ name,
           )}
         </FlexContainer>
       </ViewSelectButton>
-      {generatedStreams && isOpen && (
+      {generatedStreams && generatedStreams.length > 0 && isOpen && (
         <FlexContainer direction="column" gap="none" className={styles.generatedStreamViewContainer}>
           {generatedStreams.map((stream, index) => {
             const streamId: StreamId = { type: "generated_stream", index, dynamicStreamName: name ?? "" };
@@ -326,7 +326,12 @@ export const BuilderSidebar: React.FC<BuilderSidebarProps> = () => {
       </FlexContainer>
 
       {areDynamicStreamsEnabled && (
-        <FlexContainer direction="column" alignItems="stretch" gap="sm" className={styles.streamListContainer}>
+        <FlexContainer
+          direction="column"
+          alignItems="stretch"
+          gap="sm"
+          className={classNames(styles.streamListContainer, styles.streamTemplateList)}
+        >
           <FlexContainer className={styles.streamsHeader} alignItems="center" justifyContent="space-between">
             <FlexContainer alignItems="center" gap="none">
               <Text className={styles.streamsHeading} size="xs" bold>

airbyte-webapp/src/components/connectorBuilder/Builder/DynamicStreamConfigView.tsx

@@ -2,64 +2,66 @@ import React, { useCallback } from "react";
 import { useFormContext } from "react-hook-form";
 import { FormattedMessage, useIntl } from "react-intl";
 
+import { SchemaFormControl } from "components/forms/SchemaForm/Controls/SchemaFormControl";
 import { Button } from "components/ui/Button";
+import { Card } from "components/ui/Card";
 import { FlexContainer } from "components/ui/Flex";
 import { Heading } from "components/ui/Heading";
 import { InfoTooltip } from "components/ui/Tooltip";
 
+import { DynamicDeclarativeStream } from "core/api/types/ConnectorManifest";
 import { Action, Namespace, useAnalyticsService } from "core/services/analytics";
-import { links } from "core/utils/links";
 import { useConfirmationModalService } from "hooks/services/ConfirmationModal";
 import { BuilderView } from "services/connectorBuilder/ConnectorBuilderStateService";
 
-import { BuilderCard } from "./BuilderCard";
 import { BuilderConfigView } from "./BuilderConfigView";
-import { BuilderField } from "./BuilderField";
 import styles from "./DynamicStreamConfigView.module.scss";
-import { getDescriptionByManifest, getLabelByManifest } from "./manifestHelpers";
 import { StreamConfigView } from "./StreamConfigView";
-import { manifestRecordSelectorToBuilder } from "../convertManifestToBuilderForm";
-import { builderRecordSelectorToManifest, DynamicStreamPathFn, StreamId } from "../types";
-import { useBuilderWatch } from "../useBuilderWatch";
+import { getStreamFieldPath, StreamId } from "../types";
 
 interface DynamicStreamConfigViewProps {
   streamId: StreamId;
-  scrollToTop: () => void;
 }
 
-export const DynamicStreamConfigView: React.FC<DynamicStreamConfigViewProps> = ({ streamId, scrollToTop }) => {
+export const DynamicStreamConfigView: React.FC<DynamicStreamConfigViewProps> = ({ streamId }) => {
   const { openConfirmationModal, closeConfirmationModal } = useConfirmationModalService();
   const analyticsService = useAnalyticsService();
-  const { setValue } = useFormContext();
+  const { setValue, getValues } = useFormContext();
   const { formatMessage } = useIntl();
-  const baseUrl = useBuilderWatch("formValues.global.urlBase");
-  const dynamicStreamFieldPath: DynamicStreamPathFn = useCallback(
-    <T extends string>(fieldPath: T) => `formValues.dynamicStreams.${streamId.index}.${fieldPath}` as const,
-    [streamId.index]
-  );
 
-  const dynamicStreams = useBuilderWatch("formValues.dynamicStreams");
+  const dynamicStreamFieldPath = useCallback(
+    (fieldPath?: string) => getStreamFieldPath(streamId, fieldPath),
+    [streamId]
+  );
 
-  const handleDelete = () => {
+  const handleDelete = useCallback(() => {
     openConfirmationModal({
       text: "connectorBuilder.deleteDynamicStreamModal.text",
       title: "connectorBuilder.deleteDynamicStreamModal.title",
       submitButtonText: "connectorBuilder.deleteDynamicStreamModal.submitButton",
       onSubmit: () => {
+        const dynamicStreams: DynamicDeclarativeStream[] = getValues("manifest.dynamic_streams");
         const updatedStreams = dynamicStreams.filter((_, index) => index !== streamId.index);
         const streamToSelect = streamId.index >= updatedStreams.length ? updatedStreams.length - 1 : streamId.index;
         const viewToSelect: BuilderView =
           updatedStreams.length === 0 ? { type: "global" } : { type: "dynamic_stream", index: streamToSelect };
-        setValue("formValues.dynamicStreams", updatedStreams);
+        setValue("manifest.dynamic_streams", updatedStreams);
         setValue("view", viewToSelect);
         closeConfirmationModal();
         analyticsService.track(Namespace.CONNECTOR_BUILDER, Action.DYNAMIC_STREAM_DELETE, {
           actionDescription: "Dynamic stream deleted",
-          dynamic_stream_name: dynamicStreams[streamId.index].dynamicStreamName,
+          dynamic_stream_name: dynamicStreams[streamId.index].name,
         });
       },
     });
-  };
+  }, [analyticsService, closeConfirmationModal, getValues, openConfirmationModal, setValue, streamId]);
+
+  const templateHeaderRef = React.useRef<HTMLDivElement>(null);
+  const scrollToTopOfTemplate = useCallback(() => {
+    if (templateHeaderRef.current) {
+      templateHeaderRef.current.scrollIntoView({ behavior: "auto" });
+    }
+  }, []);
 
   if (streamId.type !== "dynamic_stream") {
     return null;
@@ -69,10 +71,11 @@ export const DynamicStreamConfigView: React.FC<DynamicStreamConfigViewProps> = (
     <BuilderConfigView className={styles.fullHeight}>
       <FlexContainer direction="column" gap="2xl" className={styles.fullHeight}>
         <FlexContainer justifyContent="space-between" alignItems="center">
-          <BuilderField
-            type="string"
-            path={dynamicStreamFieldPath("dynamicStreamName")}
-            containerClassName={styles.streamNameInput}
+          <SchemaFormControl
+            path={dynamicStreamFieldPath("name")}
+            titleOverride={null}
+            className={styles.streamNameInput}
+            placeholder={formatMessage({ id: "connectorBuilder.streamTemplateName.placeholder" })}
           />
           <Button variant="danger" onClick={handleDelete}>
             <FormattedMessage id="connectorBuilder.deleteDynamicStreamModal.title" />
@@ -82,67 +85,44 @@ export const DynamicStreamConfigView: React.FC<DynamicStreamConfigViewProps> = (
         <FlexContainer direction="column" gap="md">
           <FlexContainer gap="none">
             <Heading as="h2" size="sm">
-              <FormattedMessage id="connectorBuilder.dynamicStream.retriever.header" />
+              <FormattedMessage id="connectorBuilder.dynamicStream.resolver.header" />
             </Heading>
             <InfoTooltip placement="top">
-              <FormattedMessage id="connectorBuilder.dynamicStream.retriever.tooltip" />
+              <FormattedMessage id="connectorBuilder.dynamicStream.resolver.tooltip" />
             </InfoTooltip>
           </FlexContainer>
 
-          <BuilderCard>
-            <BuilderField
-              type="jinja"
-              path={dynamicStreamFieldPath("componentsResolver.retriever.requester.path")}
-              manifestPath="HttpRequester.properties.path"
-              preview={baseUrl ? (value) => `${baseUrl}${value}` : undefined}
-            />
-          </BuilderCard>
-          <BuilderCard
-            docLink={links.connectorBuilderRecordSelector}
-            label={getLabelByManifest("RecordSelector")}
-            tooltip={getDescriptionByManifest("RecordSelector")}
-            inputsConfig={{
-              toggleable: false,
-              path: dynamicStreamFieldPath("componentsResolver.retriever.record_selector"),
-              defaultValue: {
-                fieldPath: [],
-                normalizeToSchema: false,
-              },
-              yamlConfig: {
-                builderToManifest: builderRecordSelectorToManifest,
-                manifestToBuilder: manifestRecordSelectorToBuilder,
-              },
-            }}
-          >
-            <BuilderField
-              type="array"
-              path={dynamicStreamFieldPath("componentsResolver.retriever.record_selector.extractor.field_path")}
-              manifestPath="DpathExtractor.properties.field_path"
-              optional
+          <Card>
+            <SchemaFormControl
+              path={dynamicStreamFieldPath("components_resolver")}
+              nonAdvancedFields={NON_ADVANCED_RESOLVER_FIELDS}
             />
-            <BuilderField
-              type="jinja"
-              path={dynamicStreamFieldPath("componentsResolver.retriever.record_selector.record_filter.condition")}
-              label={getLabelByManifest("RecordFilter")}
-              manifestPath="RecordFilter.properties.condition"
-              pattern={formatMessage({ id: "connectorBuilder.condition.pattern" })}
-              optional
-            />
-          </BuilderCard>
+          </Card>
         </FlexContainer>
 
         <FlexContainer direction="column" gap="none" className={styles.fullHeight}>
-          <FlexContainer gap="none">
+          <FlexContainer gap="none" ref={templateHeaderRef}>
             <Heading as="h2" size="sm">
               <FormattedMessage id="connectorBuilder.dynamicStream.template.header" />
             </Heading>
             <InfoTooltip placement="top">
               <FormattedMessage id="connectorBuilder.dynamicStream.template.tooltip" />
             </InfoTooltip>
           </FlexContainer>
-          <StreamConfigView streamId={streamId} scrollToTop={scrollToTop} />
+          <StreamConfigView streamId={streamId} scrollToTop={scrollToTopOfTemplate} />
         </FlexContainer>
       </FlexContainer>
     </BuilderConfigView>
   );
 };
+
+const NON_ADVANCED_RESOLVER_FIELDS = [
+  "components_mapping",
+  "retriever.requester.url_base",
+  "retriever.requester.path",
+  "retriever.requester.url",
+  "retriever.requester.http_method",
+  "retriever.requester.authenticator",
+  "retriever.record_selector.extractor",
+  "stream_config",
+];

airbyte-webapp/src/components/connectorBuilder/Builder/GeneratedStreamView.module.scss

@@ -0,0 +1,4 @@
+.loadingContainer {
+  width: 100%;
+  height: 100%;
+}

airbyte-webapp/src/components/connectorBuilder/Builder/GeneratedStreamView.tsx

@@ -0,0 +1,81 @@
+import { Suspense, useCallback } from "react";
+
+import { SchemaForm } from "components/forms/SchemaForm/SchemaForm";
+import { LogoAnimation } from "components/LoadingPage/LogoAnimation";
+import { FlexContainer } from "components/ui/Flex";
+
+import { IncomingSchema, OutgoingSchema } from "core/services/connectorBuilder/SchemaResolverWorker";
+import SchemaResolverWorker from "core/services/connectorBuilder/SchemaResolverWorker?worker";
+
+import styles from "./GeneratedStreamView.module.scss";
+import { StreamConfigView } from "./StreamConfigView";
+import declarativeComponentSchema from "../../../../build/declarative_component_schema.yaml";
+import { StreamId, getStreamFieldPath } from "../types";
+
+const worker = new SchemaResolverWorker();
+
+let cachedResolvedSchema: object | null = null;
+let resolvedSchemaPromise: Promise<object> | null = null;
+
+// Use a web-worker to resolve the schema, since it is a large file
+// and we don't want to block the main thread.
+const getResolvedSchema = (): object => {
+  if (cachedResolvedSchema) {
+    return cachedResolvedSchema;
+  }
+
+  if (!resolvedSchemaPromise) {
+    resolvedSchemaPromise = new Promise((resolve) => {
+      // Setup the worker message handler
+      worker.onmessage = (event: MessageEvent<OutgoingSchema>) => {
+        const { resolvedSchema } = event.data;
+        cachedResolvedSchema = resolvedSchema;
+        resolve(resolvedSchema);
+      };
+
+      // Post the message to the worker
+      worker.postMessage({ schema: declarativeComponentSchema } as IncomingSchema);
+
+      // Suspend rendering while we wait
+      throw resolvedSchemaPromise;
+    });
+  }
+
+  throw resolvedSchemaPromise;
+};
+
+export const GeneratedStreamView: React.FC<{ streamId: StreamId; scrollToTop: () => void }> = ({
+  streamId,
+  scrollToTop,
+}) => {
+  return (
+    <Suspense
+      fallback={
+        <FlexContainer className={styles.loadingContainer} justifyContent="center" alignItems="center">
+          <LogoAnimation />
+        </FlexContainer>
+      }
+    >
+      <GeneratedStreamForm streamId={streamId} scrollToTop={scrollToTop} />
+    </Suspense>
+  );
+};
+
+const GeneratedStreamForm = ({ streamId, scrollToTop }: { streamId: StreamId; scrollToTop: () => void }) => {
+  // Resolve the schema so that we can use a sub-schema in it for SchemaForm below
+  // eslint-disable-next-line @typescript-eslint/no-explicit-any
+  const resolvedSchema = getResolvedSchema() as Record<string, any>;
+  const streamFieldPath = useCallback((fieldPath?: string) => getStreamFieldPath(streamId, fieldPath), [streamId]);
+
+  return (
+    <SchemaForm
+      key={streamFieldPath()}
+      schema={resolvedSchema.definitions.DeclarativeStream}
+      nestedUnderPath={streamFieldPath()}
+      disableFormControls
+      disableValidation
+    >
+      <StreamConfigView streamId={streamId} scrollToTop={scrollToTop} />
+    </SchemaForm>
+  );
+};