schema changes for posit connect cloud #2693

mslynch · 2025-07-01T17:49:34Z

Intent

Resolves #2677.

The changes to the schema in v4 are:

publishing schema:
- Add .server_type field, defaulting to connect
- Rename draft field .connect.access to .connect.access_control
- Add .connect_cloud field
  - Contains the following fields:
    - .vanity_name
    - .access_control
      - public_access: boolean
      - organization_access: disabled | viewer | editor
  - Contains a subset of of the fields previously allowed for Connect. The following fields are not settable for PCC:
    - .has_parameters
    - .python.package_file
    - .python.package_manager
    - .python.requires_python
    - .r.package_file
    - .r.package_manager
    - .r.requires_r
    - .jupyter.hide_all_input
    - .jupyter.hide_tagged_input
    - .quarto.version
    - .quarto.engines
    - .connect
Publishing record schema:
- Add connect_cloud enum value to .server_type field
- Add .account_name field, which represents the Posit Connect Cloud account the content was deployed to

This PR also removes some dead code from initialize.

Type of Change

- Bug Fix
- New Feature
- Breaking Change
- Documentation
- Refactor
- Tooling

Approach

This PR also includes changes for backwards compatibility with old schemas. When loading a TOML file, instead of loading it directly into a struct, we load it into a map, extract $schema, and validate against that schema. Then depending on the schema version, we apply changes to make the file data compatible with the current schema version.

User Impact

New publishes and published records will be created in the V4 format. V3 files are still supported.

Automated Tests

Added tests for conversion from v3 schema to v4.

Directions for Reviewers

Publish something new - it should create files in the v4 format.
Publish something old with v3 files - it should work, but a new deployment record should be created in the v4 format.

Checklist

I have updated CHANGELOG.md to cover notable changes.

Co-authored-by: Copilot <[email protected]>

Copilot

Pull Request Overview

This PR upgrades the publishing and record schemas from v3 to v4 to support Posit Connect Cloud, adds a required server_type field, and introduces the new connect_cloud configuration with vanity URLs and access control. It also refactors TOML decoding to use strict mapstructure rules, adds schema‐version upgrade logic, and updates tests/examples to v4.

Bump $schema URLs to v4 and require server_type on all configs and records
Add connect_cloud fields (vanity_name & access_control) and update Go types with mapstructure tags
Implement UpgradePublishingSchema/UpgradePublishingRecordSchema and switch validators to support multiple schema versions

Reviewed Changes

Copilot reviewed 50 out of 50 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
internal/schema/schema.go	Switched `ConfigSchemaURL`/`DeploymentSchemaURL` to v4 and made the validator accept legacy & v4 URLs
internal/schema/conversion.go	Introduced schema‐upgrade functions to migrate v3 data to v4
internal/util/toml.go	Added `DecodeTOMLMap` helper using mapstructure for strict decoding
internal/config/types.go	Extended `Config` struct with `ServerType` & `ConnectCloud` fields and mapstructure tags
internal/services/api/put_configuration.go	Changed `NewValidator` call to take `[]string` of schema URLs

Comments suppressed due to low confidence (2)

internal/schema/conversion.go:10

There are currently no unit tests for UpgradePublishingSchema/UpgradePublishingRecordSchema; consider adding tests to verify that v3 data is correctly migrated to v4.

func UpgradePublishingSchema(data map[string]interface{}) {

internal/services/api/http_service.go:32

[nitpick] The tag 'AccessControl Log' differs from the 'AccessControl' tag used in tests; consider unifying the log-request name for consistency.

		handler = middleware.LogRequest("AccessControl Log", log, handler)

internal/util/toml.go

internal/services/api/put_configuration.go

rodriin · 2025-07-15T18:52:12Z

@sagerb @dotNomad I have verified the E2E tests are passing successfully. It would be great if you could help review this PR since I think it is ready for review. Thank you!

dotNomad

Saw a couple things as I started to review this. Commenting at the end of my day. Apologies for the long review on this. It is quite large, and the schema changes do not make this for an easy review.

dotNomad · 2025-07-15T23:33:06Z

internal/schema/schemas/posit-publishing-schema-v4.json

+              "type": "string",
+              "default": "pip",
+              "description": "Package manager that will install the dependencies. If package-manager is none, dependencies are assumed to be pre-installed on the server. The default is 'pip'.",
+              "examples": ["pip", "conda", "pipenv", "poetry", "none"]


This stood out to me. My understanding is we only support uv and pip. Looks like these were added with the update to v4.

dotNomad · 2025-07-15T23:45:54Z

internal/schema/schemas/draft/posit-publishing-record-schema-v4.json

+    "id": {
+      "type": "string",
+      "description": "Unique ID of this deployment.",
+      "examples": ["de2e7bdb-b085-401e-a65c-443e40009749"]
+    },


I pushed up the drafts to the CDN to test this out and I'm seeing "id" is a required property which doesn't appear to occur with v3.

dotNomad · 2025-07-15T23:53:04Z

internal/schema/schemas/posit-publishing-schema-v4.json

+      "type": "object",
+      "description": "Python language and dependencies.",


I think we want to keep the additionalProperties: false on these sections. I'm not 100% sure though so let me know if I'm missing something there.

dotNomad · 2025-07-15T23:54:20Z

internal/schema/schemas/posit-publishing-schema-v4.json

+              "type": "string",
+              "default": "",
+              "description": "Python interpreter version, in PEP 440 format, required to run the content. If not specified it will be detected from the one in use.",
+              "examples": [">3.8", "<3.9", "==3.5"]


Looks like this got a adjusted. Not sure which is more correct.

Suggested change

"examples": [">3.8", "<3.9", "==3.5"]

"examples": [">3.8", "<3.9", "=3.5"]

dotNomad

Took another look through some of this. When pointing at the draft schema for the configuration file I'm seeing an error "Unknown Title"

Even Better TOML doesn't seem to show the error, but something is having trouble validating that.

It looks like we should update the docs/configuration.md file to include new information about the server type and other Connect Cloud fields.

Conversion on the deployment record to v4 worked great, but I had a similar issue where when I pointed to schemas/draft/posit-publishing-record-schema-v4.json (which are basically the same, but not exactly?) then the Publisher couldn't validate it and I could no longer select it as a deployment. My worry is that if that is happening with the draft it will happen with the non-draft v4 record schema.

It looks like the configuration saved to the record uses v4 which is nice even though the Configuration is v3. What are our thoughts on how to migrate configurations? Should we consider that or are we accepting that v3 configurations work with v4 deployment records?

I've been looking through this for a few hours and I'm having trouble verifying everything here and keeping all of the changes in my head.

Would it be possible to split this up a bit into easier to review PRs?

Some ideas for where to split it:

the removal of dead code from initialize
potentially the TOML backwards compatibility could be its own PR since that changes how schemas are validated
the schema change to use $defs more
- We could try a draft version with just Connect changes and verify those changes then another PR with the Posit Connect Cloud def with those schema properties

I really want to verify the new changes to confirm things are working as expected, but its difficult with the amount of them. Let me know your thoughts on a potential split up to make it easier to verify.

If we want to keep everything in the same PR we can start verifying functionality with the internal/schema/schemas/draft/...-v4.json files as I was doing above.

mslynch and others added 28 commits June 24, 2025 09:59

add support for retrieving stored Posit Connect Cloud credentials

8232132

populate server type on credentials from file

9ec3147

fixes for file-based cred storage

f1704f4

refactor request handling

5007084

test coverage for adding PCC creds

3a89427

validation for file credentials

d202ab0

omitempty on file credential fields

a49fa9e

update changelog

d39553c

call ServerTypeFromURL with normalizedUrl

84005a9

Co-authored-by: Copilot <[email protected]>

remove shinyapps.io and Posit Cloud server types

4b7ec7a

schema changes for posit connect cloud

fab8361

Merge branch 'main' into pcc-creds

4a31a14

changelog correction

60ad41b

include changes from draft schema and add draft v4 publishing schema

276a9d7

fix many tests

2801e6d

remove dead code

824dd21

Merge branch 'main' into schema-v4

780db48

fix tests

d90bf3b

remove more dead code

8702d1f

refactor

f525e53

schema upgrades

f6ecf4f

supply default value for server_type in publishing config

3a9c9bc

leave access connect control for draft schema

df0c44f

rename access to access_control

814eb87

Merge branch 'pcc-creds' into schema-v4

8ab2b45

add account name to publishing recrod

207a284

linting

59b3280

add test coverage

1c8dfcf

mslynch changed the base branch from main to pcc-creds July 7, 2025 19:58

add typescript types

50d4bcb

mslynch added 3 commits July 7, 2025 16:12

add ConnectCloud to config struct

f20a938

update changelog

d91b02f

revert typescript changes

23577aa

mslynch requested a review from Copilot July 8, 2025 13:44

Copilot AI reviewed Jul 8, 2025

View reviewed changes

internal/util/toml.go Show resolved Hide resolved

internal/services/api/put_configuration.go Show resolved Hide resolved

add test coverage

37a6b99

Base automatically changed from pcc-creds to main July 8, 2025 19:23

mslynch added 2 commits July 8, 2025 14:52

Merge branch 'main' into schema-v4

de73ed1

revert accidental changes

523bb64

mslynch requested a review from rodriin July 8, 2025 20:17

mslynch marked this pull request as ready for review July 8, 2025 20:17

mslynch requested review from dotNomad, sagerb and marcosnav as code owners July 8, 2025 20:17

mslynch and others added 5 commits July 14, 2025 11:02

Merge branch 'main' into schema-v4

38df7bd

fix casing on Config struct; default serverType on UI side

47d65c5

serverType, not server_type in tests

a8ff6b2

Fixing broken unit tests

2f87613

Fixing the E2E tests

fc6ee24

dotNomad reviewed Jul 15, 2025

View reviewed changes

dotNomad reviewed Jul 16, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

schema changes for posit connect cloud #2693

schema changes for posit connect cloud #2693

mslynch commented Jul 1, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

rodriin commented Jul 15, 2025

Uh oh!

dotNomad left a comment

Uh oh!

dotNomad Jul 15, 2025

Uh oh!

dotNomad Jul 15, 2025

Uh oh!

dotNomad Jul 15, 2025

Uh oh!

dotNomad Jul 15, 2025

Uh oh!

dotNomad left a comment

Uh oh!

Uh oh!

		"type": "object",
		"description": "Python language and dependencies.",

	"examples": [">3.8", "<3.9", "==3.5"]
	"examples": [">3.8", "<3.9", "=3.5"]

schema changes for posit connect cloud #2693

Are you sure you want to change the base?

schema changes for posit connect cloud #2693

Conversation

mslynch commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Intent

Type of Change

Approach

User Impact

Automated Tests

Directions for Reviewers

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

rodriin commented Jul 15, 2025

Uh oh!

dotNomad left a comment

Choose a reason for hiding this comment

Uh oh!

dotNomad Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

dotNomad Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

dotNomad Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

dotNomad Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

dotNomad left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mslynch commented Jul 1, 2025 •

edited

Loading