Skip to content

Commit 7253409

Browse files
authored
[examples] consolidate docs_beta_snippets and docs_snippets (#28771)
## Summary & Motivation Closes DOC-898. Consolidates the `examples/docs_beta_snippets/` and `examples/docs_snippets` directory for improved maintainability and organization. ## How I Tested These Changes ## Changelog NOCHANGELOG
1 parent 79ec9f1 commit 7253409

File tree

510 files changed

+899
-1102
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

510 files changed

+899
-1102
lines changed

.buildkite/dagster-buildkite/dagster_buildkite/steps/packages.py

-8
Original file line numberDiff line numberDiff line change
@@ -303,14 +303,6 @@ def k8s_extra_cmds(version: AvailablePythonVersion, _) -> List[str]:
303303
# snippets in all python versions since we are testing the core code exercised by the
304304
# snippets against all supported python versions.
305305
unsupported_python_versions=AvailablePythonVersion.get_all_except_default(),
306-
),
307-
PackageSpec(
308-
"examples/docs_beta_snippets",
309-
# The docs_snippets test suite also installs a ton of packages in the same environment,
310-
# which is liable to cause dependency collisions. It's not necessary to test all these
311-
# snippets in all python versions since we are testing the core code exercised by the
312-
# snippets against all supported python versions.
313-
unsupported_python_versions=AvailablePythonVersion.get_all_except_default(),
314306
pytest_tox_factors=["all", "integrations", "docs_snapshot_test"],
315307
always_run_if=has_dg_or_components_changes,
316308
),

.buildkite/dagster-buildkite/dagster_buildkite/utils.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -324,7 +324,7 @@ def has_dg_or_components_changes():
324324
return any(
325325
"dagster-dg" in str(path)
326326
or "dagster-components" in str(path)
327-
or "docs_beta_snippets" in str(path)
327+
or "docs_snippets" in str(path)
328328
for path in ChangedFiles.all
329329
)
330330

.github/workflows/build-docs.yml

-2
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@ on:
66
- 'release-*'
77
paths:
88
- docs/**
9-
- examples/docs_beta_snippets/**
109
- examples/docs_snippets/**
1110
- examples/docs_projects/**
1211
- examples/getting_started_etl_tutorial/**
@@ -20,7 +19,6 @@ on:
2019
pull_request:
2120
paths:
2221
- docs/**
23-
- examples/docs_beta_snippets/**
2422
- examples/docs_snippets/**
2523
- examples/docs_projects/**
2624
- examples/getting_started_etl_tutorial/**

docs/CONTRIBUTING.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -250,7 +250,7 @@ Some CLI invocations may be brief enough that we don't want to include them in a
250250
<CliInvocationExample contents="uv add 'dagster-components[sling]'" />
251251
```
252252

253-
For more information on testing the CLI commands used in docs, see [the README in docs tests](../../examples/docs_beta_snippets/docs_beta_snippets_tests/snippet_checks/README.md).
253+
For more information on testing the CLI commands used in docs, see [the README in docs tests](../../examples/docs_snippets/docs_snippets_tests/snippet_checks/README.md).
254254

255255
### Code reference links
256256

docs/Makefile

+5-5
Original file line numberDiff line numberDiff line change
@@ -30,16 +30,16 @@ sphinx_objects_inv:
3030
cp sphinx/_build/json/objects.inv static/.
3131

3232
regenerate_cli_snippets:
33-
cd ../examples/docs_beta_snippets && tox -e docs_snapshot_update && echo "\nSnippets regenerated!"
33+
cd ../examples/docs_snippets && tox -e docs_snapshot_update && echo "\nSnippets regenerated!"
3434

3535
regenerate_cli_snippets_and_screenshots:
36-
cd ../examples/docs_beta_snippets && EXTRA_PARAMS='--update-screenshots' tox -e docs_snapshot_update && echo "\nSnippets regenerated!"
36+
cd ../examples/docs_snippets && EXTRA_PARAMS='--update-screenshots' tox -e docs_snapshot_update && echo "\nSnippets regenerated!"
3737

3838
regenerate_cli_snippets_and_test:
39-
cd ../examples/docs_beta_snippets && tox -e docs_snapshot_update && tox -e docs_snapshot_test && echo "\nSnippets regenerated and tested!"
39+
cd ../examples/docs_snippets && tox -e docs_snapshot_update && tox -e docs_snapshot_test && echo "\nSnippets regenerated and tested!"
4040

4141
regenerate_cli_snippets_and_test_debug:
42-
cd ../examples/docs_beta_snippets && rm -rf /tmp/cli_snippets && mkdir -p /tmp/cli_snippets && (DAGSTER_CLI_SNIPPET_WORKING_DIR=/tmp/cli_snippets tox -e docs_snapshot_update && DAGSTER_CLI_SNIPPET_WORKING_DIR=/tmp/cli_snippets tox -e docs_snapshot_test && echo "\nSnippets regenerated and tested!") || (echo "Snippet regeneration failed. Please check the logs for more information. Working directory: /tmp/cli_snippets" && exit 1)
42+
cd ../examples/docs_snippets && rm -rf /tmp/cli_snippets && mkdir -p /tmp/cli_snippets && (DAGSTER_CLI_SNIPPET_WORKING_DIR=/tmp/cli_snippets tox -e docs_snapshot_update && DAGSTER_CLI_SNIPPET_WORKING_DIR=/tmp/cli_snippets tox -e docs_snapshot_test && echo "\nSnippets regenerated and tested!") || (echo "Snippet regeneration failed. Please check the logs for more information. Working directory: /tmp/cli_snippets" && exit 1)
4343

4444
test_cli_snippets_simulate_bk:
45-
docker run --platform linux/amd64 --rm -it -v "$(DAGSTER_GIT_REPO_DIR):/dagster" --entrypoint /bin/sh dagster/buildkite-test:py3.9-2025-01-31T181043 -c 'cd /dagster/examples/docs_beta_snippets && tox -e docs_snapshot_test -- --pdb'
45+
docker run --platform linux/amd64 --rm -it -v "$(DAGSTER_GIT_REPO_DIR):/dagster" --entrypoint /bin/sh dagster/buildkite-test:py3.9-2025-01-31T181043 -c 'cd /dagster/examples/docs_snippets && tox -e docs_snapshot_test -- --pdb'

docs/README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ This is the home of the Dagster documentation. The documentation site is built u
88

99
- `./src` contains custom components, styles, themes, and layouts.
1010
- `./docs/` contains documentation Markdown files.
11-
- `/examples/docs_snippets/docs_snippets/` contains code examples for the documentation. Some code examples also live in `/examples/` and `/examples/docs_beta_snippets/docs_beta_snippets/`.
11+
- `/examples/docs_snippets/docs_snippets/` contains code examples for the documentation. Some code examples also live in `/examples/` and `/examples/docs_snippets/docs_snippets/`.
1212

1313
The docs are organized into the following sections:
1414

docs/docs/dagster-plus/deployment/deployment-types/hybrid/kubernetes/setup.md

+3-3
Original file line numberDiff line numberDiff line change
@@ -433,19 +433,19 @@ The units for CPU and memory resources are described [in this document](https://
433433

434434
The default behavior in Dagster+ is to create one pod for a run. Each asset targeted by that run is executed in subprocess within the pod. Use a job tag to request resources for this pod, which in turn makes those resources available to the targeted assets.
435435

436-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/hybrid/agents/kubernetes/resource_request_job.py" language="python" title="Request resources for a job" />
436+
<CodeExample path="docs_snippets/docs_snippets/dagster-plus/deployment/hybrid/agents/kubernetes/resource_request_job.py" language="python" title="Request resources for a job" />
437437

438438
Another option is to launch a pod for each asset by telling Dagster to use the Kubernetes job executor. In this case, you can specify resources for each individual asset.
439439

440-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/hybrid/agents/kubernetes/resource_request_asset.py" language="python" title="Request resources for an asset" />
440+
<CodeExample path="docs_snippets/docs_snippets/dagster-plus/deployment/hybrid/agents/kubernetes/resource_request_asset.py" language="python" title="Request resources for an asset" />
441441

442442
</TabItem>
443443

444444
<TabItem value="pipes" label="Resources if using kubernetes pipes">
445445

446446
Dagster can launch and manage existing Docker images as Kubernetes jobs using the [Dagster kubernetes pipes integration](/integrations/libraries/kubernetes). To request resources for these jobs by supplying the appropriate Kubernetes pod spec.
447447

448-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/hybrid/agents/kubernetes/resource_request_pipes.py" language="python" title="Request resources for a k8s pipes asset" />
448+
<CodeExample path="docs_snippets/docs_snippets/dagster-plus/deployment/hybrid/agents/kubernetes/resource_request_pipes.py" language="python" title="Request resources for a k8s pipes asset" />
449449

450450

451451

docs/docs/dagster-plus/deployment/deployment-types/hybrid/local.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ Your local agent will need a token to authenticate with your Dagster+ account. T
4141

4242
1. Create a directory to act as your Dagster home. This guide uses `~/dagster_home`, but the directory can be located wherever you want.
4343
2. In the new directory, create a `dagster.yaml` file with the following:
44-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/hybrid/agents/local_dagster.yaml" language="yaml" title="dagster.yaml" />
44+
<CodeExample path="docs_snippets/docs_snippets/dagster-plus/deployment/hybrid/agents/local_dagster.yaml" language="yaml" title="dagster.yaml" />
4545
3. In the file, fill in the following:
4646
- `agent_token.env` - The name of the environment variable storing the agent token you created in Step 1.
4747
- `deployment` - The name of the deployment associated with this instance of the agent. In the preceding example, `prod` was used as the deployment.

docs/docs/dagster-plus/deployment/deployment-types/serverless/runtime-environment.md

+10-10
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ By default, Dagster+ Serverless will package your code as PEX files and deploys
1717

1818
You can add dependencies by including the corresponding Python libraries in your Dagster project's `setup.py` file. These should follow [PEP 508](https://peps.python.org/pep-0508/).
1919

20-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/serverless/runtime-environment/example_setup.py" language="Python" title="Example setup.py" />
20+
<CodeExample path="docs_snippets/docs_snippets/dagster-plus/deployment/serverless/runtime-environment/example_setup.py" language="Python" title="Example setup.py" />
2121

2222
You can also use a tarball to install a dependency, such as if `pip` is unable to resolve a package using `dependency_links`. For example, `soda` and `soda-snowflake` provide tarballs that you can include in the `install_requires` section:
2323

@@ -48,18 +48,18 @@ Python versions 3.9 through 3.12 are all supported for Serverless deployments. Y
4848
<Tabs groupId="method">
4949
<TabItem value="GitHub" label="GitHub">
5050
In your `.github/workflows/deploy.yml` file, update the `PYTHON_VERSION` environment variable with your desired Python version:
51-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/serverless/runtime-environment/github_python_version.yaml" language="yaml" title="Updating the Python version in deploy.yml" />
51+
<CodeExample path="docs_snippets/docs_snippets/dagster-plus/deployment/serverless/runtime-environment/github_python_version.yaml" language="yaml" title="Updating the Python version in deploy.yml" />
5252

5353
</TabItem>
5454
<TabItem value="GitLab" label="GitLab">
5555
1. Open your `.gitlab-ci.yml` file. If your `.gitlab-ci.yml` contains an `include` with a link to a Dagster provided CI/CD template:
56-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/serverless/runtime-environment/gitlab_template.yaml" language="yaml" />
56+
<CodeExample path="docs_snippets/docs_snippets/dagster-plus/deployment/serverless/runtime-environment/gitlab_template.yaml" language="yaml" />
5757

5858
Follow the link and replace the contents of your `.gitlab-ci.yml` with the YAML document at the link address. Otherwise, continue to the next step.
5959

6060
3. Update the `PYTHON_VERSION` environment variable with your desired Python version
6161

62-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/serverless/runtime-environment/gitlab_python_version.yaml" language="yaml" title="Updating the Python version in .gitlab-ci.yml" />
62+
<CodeExample path="docs_snippets/docs_snippets/dagster-plus/deployment/serverless/runtime-environment/gitlab_python_version.yaml" language="yaml" title="Updating the Python version in .gitlab-ci.yml" />
6363

6464
</TabItem>
6565
<TabItem value="CLI" label="CLI">
@@ -101,7 +101,7 @@ Setting a custom base image isn't supported for GitLab CI/CD workflows out of th
101101
<Tabs groupId="method">
102102
<TabItem value="GitHub" label="GitHub">
103103
In your `.github/workflows/deploy.yml` file, add the `SERVERLESS_BASE_IMAGE_TAG` environment variable and set it to the tag printed out in the previous step:
104-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/serverless/runtime-environment/github_base_image.yaml" language="yaml" title="Setting a custom base image in deploy.yml" />
104+
<CodeExample path="docs_snippets/docs_snippets/dagster-plus/deployment/serverless/runtime-environment/github_base_image.yaml" language="yaml" title="Setting a custom base image in deploy.yml" />
105105
106106
</TabItem>
107107
@@ -132,7 +132,7 @@ To add data files to your deployment, use the [Data Files Support](https://setup
132132
```
133133
134134
If you want to include the data folder, modify your `setup.py` to add the `package_data` line:
135-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/serverless/runtime-environment/data_files_setup.py" language="Python" title="Loading data files in setup.py" />
135+
<CodeExample path="docs_snippets/docs_snippets/dagster-plus/deployment/serverless/runtime-environment/data_files_setup.py" language="Python" title="Loading data files in setup.py" />
136136
137137
## Disable PEX deploys
138138
@@ -141,18 +141,18 @@ You have the option to disable PEX-based deploys and deploy using a Docker image
141141
<Tabs groupId="method">
142142
<TabItem value="GitHub" label="GitHub">
143143
In your `.github/workflows/deploy.yml` file, update the `ENABLE_FAST_DEPLOYS` environment variable to `false`:
144-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/serverless/runtime-environment/github_disable_pex.yaml" language="yaml" title="Disable PEX deploys in deploy.yml" />
144+
<CodeExample path="docs_snippets/docs_snippets/dagster-plus/deployment/serverless/runtime-environment/github_disable_pex.yaml" language="yaml" title="Disable PEX deploys in deploy.yml" />
145145
146146
</TabItem>
147147
<TabItem value="GitLab" label="GitLab">
148148
1. Open your `.gitlab-ci.yml` file. If your `.gitlab-ci.yml` contains an `include` with a link to a Dagster provided CI/CD template:
149-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/serverless/runtime-environment/gitlab_template.yaml" language="yaml" />
149+
<CodeExample path="docs_snippets/docs_snippets/dagster-plus/deployment/serverless/runtime-environment/gitlab_template.yaml" language="yaml" />
150150
151151
Follow the link and replace the contents of your `.gitlab-ci.yml` with the YAML document at the link address. Otherwise, continue to the next step.
152152
153153
3. Update the `DISABLE_FAST_DEPLOYS` variable to `true`
154154
155-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/serverless/runtime-environment/gitlab_disable_pex.yaml" language="yaml" title="Disable PEX deploys in .gitlab-ci.yml" />
155+
<CodeExample path="docs_snippets/docs_snippets/dagster-plus/deployment/serverless/runtime-environment/gitlab_disable_pex.yaml" language="yaml" title="Disable PEX deploys in .gitlab-ci.yml" />
156156
157157
</TabItem>
158158
<TabItem value="CLI" label="CLI">
@@ -186,7 +186,7 @@ Setting a custom base image isn't supported for GitLab CI/CD workflows out of th
186186
<Tabs groupId="method">
187187
<TabItem value="GitHub" label="GitHub">
188188
In your `.github/workflows/deploy.yml` file, add the `SERVERLESS_BASE_IMAGE_TAG` environment variable and set it to the tag printed out in the previous step:
189-
<CodeExample path="docs_beta_snippets/docs_beta_snippets/dagster-plus/deployment/serverless/runtime-environment/github_no_pex_custom_base_image.yaml" language="yaml" title="Setting a custom base image in `deploy.yml`" />
189+
<CodeExample path="docs_snippets/docs_snippets/dagster-plus/deployment/serverless/runtime-environment/github_no_pex_custom_base_image.yaml" language="yaml" title="Setting a custom base image in `deploy.yml`" />
190190

191191
</TabItem>
192192
<TabItem value="CLI" label="CLI">

0 commit comments

Comments
 (0)