Skip to content

Commit 17f1072

Browse files
EliCDaviskalebpace
andauthored
Documentation, Guides, Tests, and Struct Node Output Refactor (#108)
* Now capturing error and timing metrics, new Divide and DivideToArray nodes * start of showing node execution times and error messages * capturing timings of nodes * timing updates * struct node output port refactor * Execution times on webpage update as you make changes * standardized node naming * fixed resizing issue * Documentation, new nodes spline: position, positions for array, tangent, tangents for array * import manifest into editor * Editor section of unity integration * flake update * flake update * fix: duplicate material names in obj * fix: npmDepsHash skip cache mismatch error (#111) We add `-Lo` to print verbose build logs so that errors [here](https://github.com/EliCDavis/polyform/actions/runs/16831757233/job/47680959627#step:5:214) will display the actual hash mismatch error [log](https://github.com/EliCDavis/polyform/actions/runs/16832252332/job/47682633319#step:5:214) so we know which to update. Adds two small [Nix Flake App](https://nix.dev/manual/nix/2.28/command-ref/new-cli/nix3-run.html#apps) scripts to both document how to update sri files or clear Github Action cache, as well as make them easy to re-run. Tried to find a way to make SRI updates runnable without nix, but since both utilties are fairly bespoke programs to fetch/install, it seemed easier to just keep it in a derivation to be run with WSL or a linux container. * Doc update --------- Co-authored-by: Kaleb Pace <[email protected]>
1 parent 06fdf37 commit 17f1072

File tree

194 files changed

+3671
-2608
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

194 files changed

+3671
-2608
lines changed

.github/workflows/build.yml

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,14 @@
11
name: Build
22

33
on:
4-
push:
5-
branches: [ "main" ]
6-
pull_request:
7-
branches: [ "main" ]
8-
# Allows manual triggering
9-
workflow_dispatch:
4+
push:
5+
branches:
6+
- main
7+
pull_request:
8+
branches:
9+
- main
10+
# Allows manual triggering
11+
workflow_dispatch:
1012

1113
jobs:
1214
check:
@@ -26,7 +28,7 @@ jobs:
2628
- uses: DeterminateSystems/flakehub-cache-action@main
2729

2830
- name: Restore Nix-cached Release Artifacts
29-
run: nix build .#release -o ./artifacts
31+
run: nix build .#release -Lo ./artifacts
3032

3133
pages:
3234
runs-on: ubuntu-latest
@@ -36,4 +38,4 @@ jobs:
3638
- uses: DeterminateSystems/flakehub-cache-action@main
3739

3840
- name: Restore Nix-cached Pages Artifacts
39-
run: nix build .#pages -o ./dist
41+
run: nix build .#pages -Lo ./dist

.github/workflows/flake-update.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ name: update-flake-lock
22
on:
33
workflow_dispatch: # allows manual triggering
44
schedule:
5-
- cron: "0 0 * * 0" # runs weekly on Sunday at 00:00
5+
- cron: "0 0 1 * *" # runs weekly on Sunday at 00:00
66

77
jobs:
88
lockfile:

.github/workflows/release.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ jobs:
2828
- uses: DeterminateSystems/flakehub-cache-action@main
2929

3030
- name: Restore Nix-cached Release Artifacts
31-
run: nix build .#release -o ./artifacts
31+
run: nix build .#release -Lo ./artifacts
3232

3333
- name: Publish GitHub Release
3434
run: |
@@ -45,7 +45,7 @@ jobs:
4545
- uses: DeterminateSystems/flakehub-cache-action@main
4646

4747
- name: Restore Nix-cached Pages Artifacts
48-
run: nix build .#pages -o ./dist
48+
run: nix build .#pages -Lo ./dist
4949

5050
- name: Setup Pages
5151
uses: actions/configure-pages@v4

README.md

Lines changed: 25 additions & 197 deletions
Large diffs are not rendered by default.

docs/README.md

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,13 @@
22

33
## Guides
44

5-
* [Creating a Custom Node](./guides/CreatingNodes.md)
5+
* [Polyform Concepts](./guides/Concepts/README.md)
6+
* Adding New Nodes
7+
* [Creating a Custom Node](./guides/CreatingNodes/README.md)
8+
* [Manifests](./guides/CreatingManifests/README.md)
9+
* [Best Practices](./guides/NodeBestPractices/README.md)
10+
* [Unity Integration](./guides/UnityIntegration/README.md)
11+
12+
## Resources
13+
14+
[Recomended Reading](./resources/README.md)

docs/guides/Concepts/README.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
[Table of Contents](../../README.md)
2+
3+
# Concepts
4+
5+
## Variables
6+
7+
Variables are high-level values that a graph exposes for external control. They act as inputs to the procedural generation process. Changing them alters the graph’s behavior without modifying the graph’s structure.
8+
9+
Variables can represent anything from a single number or color to more complex data like positions, textures, or mesh data.
10+
11+
## Profiles
12+
13+
Profiles are saved sets of variable values. They make it easy to store and reuse specific configurations for a graph.
14+
15+
## Manifest
16+
17+
A manifest is the structured output of running a graph with a given set of variables.
18+
19+
It describes what the graph produced, generally including references to generated geometry, textures, or metadata.
Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
Previous: [Creating a Custom Node](../CreatingNodes/README.md) | [Table of Contents](../../README.md) | Next: [Best Practices](../NodeBestPractices/README.md)
2+
3+
# Creating Manifest Nodes
4+
5+
Polyform operates on [Directed Acyclic Graphs (DAGs)](https://en.wikipedia.org/wiki/Directed_acyclic_graph), where each node performs a specific task and connects to other nodes through inputs and outputs.
6+
7+
Manifest nodes are special types of nodes that **define the final output of a graph**, typically writing one or more files to the filesystem.
8+
9+
## What is a Manifest?
10+
11+
A **manifest** is a structure that encapsulates **a set of named entries**, each representing an output artifact such as an image, mesh, or any other serializable content. It also optionally defines a **"main" entry**, which can help tell viewers the first entry to download.
12+
13+
For example, if we have a manifest for a textured GLTF, you might have 2 entries: the GLTF itself, and a seperate image file the GLTF references. In this scenario, we'd set the GLTF file to be the main entry that a viewer would initially download.
14+
15+
Here's the core structure used for manifests:
16+
17+
```go
18+
type Entry struct {
19+
Metadata map[string]any `json:"metadata"` // Descriptive metadata about the artifact
20+
Artifact Artifact `json:"-"` // The actual data to be written
21+
}
22+
23+
type Manifest struct {
24+
Main string `json:"main"` // Optional: name of the main entry
25+
Entries map[string]Entry `json:"entries"` // All entries in the manifest
26+
}
27+
```
28+
29+
## Artifacts
30+
31+
The `Artifact` interface defines what it means to be a serializable output. Any type that implements this interface can be written as part of a manifest.
32+
33+
```go
34+
type Artifact interface {
35+
Write(io.Writer) error // Defines how the artifact is written to disk or elsewhere
36+
Mime() string // Describes the type of content (e.g., "image/png", "application/json")
37+
}
38+
```
39+
40+
Here's a minimal example for a text file:
41+
42+
```go
43+
type TextArtifact struct {
44+
Content string
45+
}
46+
47+
func (ta TextArtifact) Write(w io.Writer) error {
48+
_, err := io.WriteString(w, ta.Content)
49+
return err
50+
}
51+
52+
func (ta TextArtifact) Mime() string {
53+
return "text/plain"
54+
}
55+
```
56+
57+
Then you can include this in your manifest entry:
58+
59+
```go
60+
entry := manifest.Entry{
61+
Metadata: map[string]any{"type": "text"},
62+
Artifact: TextArtifact{Content: "Hello World!"},
63+
}
64+
```
65+
66+
## Creating a Manifest Node
67+
68+
A manifest node in Polyform is simply a node whose **output type is `manifest.Manifest`**. When Polyform runs a graph and reaches this node, it knows how to gather the manifest entries and process them as final outputs.
69+
70+
Here’s an example of what a simple manifest node might look like:
71+
72+
```go
73+
type MyManifestNode struct {
74+
Image nodes.Output[image.Image] `description:"The image to export"`
75+
Path nodes.Output[string] `description:"Path to export to"`
76+
}
77+
78+
func (mmn MyManifestNode) Output(out *nodes.StructOutput[manifest.Manifest]) {
79+
img := nodes.TryGetOutputValue(out, mmn.Image, nil)
80+
if img == nil {
81+
return
82+
}
83+
84+
entry := manifest.Entry{
85+
Metadata: map[string]any{
86+
"description": "Exported PNG image",
87+
},
88+
Artifact: &manifest.ImageArtifact{
89+
Image: img,
90+
MimeType: "image/png",
91+
},
92+
}
93+
94+
entryPath := nodes.TryGetOutputValue(out, mmn.Path, "export.png")
95+
out.Set(manifest.Manifest{
96+
Main: entryPath,
97+
Entries: map[string]manifest.Entry{
98+
entryPath: entry,
99+
},
100+
})
101+
}
102+
```
103+
104+
> **Note**: `manifest.ImageArtifact` in this example is a hypothetical implementation of the `Artifact` interface that knows how to write an image as PNG.
Lines changed: 31 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,17 @@
1+
[Table of Contents](../../README.md) | Next: [Manifests](../CreatingManifests/README.md)
2+
13
# Creating a Custom Node
24

35
In Polyform, a node generally represents a single operation in a procedural graph.
46

5-
Inputs to nodes are other nodes' outputs, and outputs are computed dynamically as the graph executes.
7+
* Inputs to nodes are other nodes' outputs.
8+
* Outputs are computed on demand as the graph executes.
69

710
## Defining a Node
811

9-
Nodes generally start as a struct, where the fields of that struct represent it's connections to other node's output.
12+
Nodes generally start as a struct, where the fields of that struct represent its connections to other node's output.
1013

11-
For the sake of presenting information to the user, as well as automatically generating documentation, you can define descriptions for node as well as each of it's inputs.
14+
For the sake of presenting information to the user, as well as automatically generating documentation, you can define descriptions for the node as well as each of its inputs.
1215

1316
```go
1417
type MathNode struct {
@@ -23,42 +26,45 @@ func (MathNode) Description() string {
2326

2427
### Defining Outputs
2528

26-
Any method on a struct that returns a `nodes.StructOutput[T]` is automatically treated as an output port by the system. `StructOutput[T]` acts as a wrapper around the value you want to output for supporting things like error handling.
29+
Any method on a struct that accepts a `*nodes.StructOutput[T]` is automatically treated as an output port by the system. `StructOutput[T]` acts as a wrapper around the value you want to output, supporting things like timing operations and error handling.
2730

2831
```go
29-
func (cn MathNode) Add() nodes.StructOutput[float64] {
32+
func (cn MathNode) Add(out *nodes.StructOutput[float64]) {
3033
a := cn.A.Value()
3134
b := cn.B.Value()
32-
return nodes.NewStructOutput(a + b)
35+
out.Set(a + b)
3336
}
3437
```
3538

36-
However, when implementing the function, a node should never assume any of its input ports are set. Calling `Value()` on a `nil` input port will cause the runtime to panic and the graph execution to halt. The utility `nodes.TryGetOutputValue` has been introduced that will attempt to take the value of an input port if it exists, and returns a fallback value when the input port is `nil`. Updating our code to be more safe results in our `Add` function looking like:
39+
However, when implementing the function, a node should never assume any of its input ports are set. Calling `Value()` on a `nil` input port will cause the runtime to panic and the graph execution to halt. The utility `nodes.TryGetOutputValue` has been introduced that will attempt to take the value of an input port if it exists, and returns a fallback value when the input port is `nil`. `TryGetOutputValue` also keeps up with how much time it takes for the input to execute, subtracting it from the `MathNode`'s execution time, allowing for proper reporting of performance on a per node basis. Updating our code to be more safe results in our `Add` function looking like:
3740

3841
```go
39-
func (mn MathNode) Add() nodes.StructOutput[float64] {
40-
a := nodes.TryGetOutputValue(mn.A, 0)
41-
b := nodes.TryGetOutputValue(mn.B, 0)
42-
return nodes.NewStructOutput(a + b)
42+
func (mn MathNode) Add(out *nodes.StructOutput[float64]) {
43+
a := nodes.TryGetOutputValue(out, mn.A, 0)
44+
b := nodes.TryGetOutputValue(out, mn.B, 0)
45+
out.Set(a + b)
4346
}
4447
```
4548

46-
Sometimes, the current input into a node is in effect "invalid" and no real computation can be done. In these scenarios a _sensible_ default value needs to be returned. Unfortunately, what "_sensible_" means is dependent upon the kind of operations being performed by the node, but a good rule of thumb is returning the ["zero" value](https://go.dev/ref/spec#The_zero_value) of the datatype.
49+
Sometimes, the current input into a node is effectively "invalid" and no real computation can be done. In these scenarios a _sensible_ default value needs to be returned. Unfortunately, what "_sensible_" means is dependent upon the kind of operations being performed by the node, but a good rule of thumb is returning the ["zero" value](https://go.dev/ref/spec#The_zero_value) of the datatype.
4750

4851
Before we return our sensible value, we can capture an error to alert the graph system that something has gone wrong.
4952

5053
```go
51-
func (mn MathNode) Divide() nodes.StructOutput[float64] {
52-
a := nodes.TryGetOutputValue(mn.A, 0)
53-
b := nodes.TryGetOutputValue(mn.B, 0)
54+
func (mn MathNode) Divide(out *nodes.StructOutput[float64]) {
55+
a := nodes.TryGetOutputValue(out, mn.A, 0)
56+
b := nodes.TryGetOutputValue(out, mn.B, 0)
5457

5558
if b == 0 {
56-
out := nodes.NewStructOutput[float64](0.)
59+
// By default, the output is the zero value already, so this line
60+
// effectively acts as a no-op, and is kept for demonstration purposes
61+
// only.
62+
out.Set(0)
5763
out.CaptureError(errors.New("can't divide by 0"))
58-
return out
64+
return
5965
}
6066

61-
return nodes.NewStructOutput(a / b)
67+
out.Set(a / b)
6268
}
6369
```
6470

@@ -78,22 +84,20 @@ func (MathNode) DivideDescription() string {
7884

7985
### Array Inputs
8086

81-
If the operation you're performing can take any number of inputs, you can define your input as type `[]nodes.Output[T]`. Doing so allows users to wire up multiple nodes into the same input slot.
87+
If the operation you're performing can take any number of inputs, you can define your input as type `[]nodes.Output[T]`. Doing so allows users to wire up multiple nodes into the same input slot. You can then use `nodes.GetOutputValues` to call resolve all inputs, creating timings while doing so.
8288

8389
```go
8490
type SumNode struct {
8591
Values []nodes.Output[float64] `description:"The nodes to sum"`
8692
}
8793

88-
func (sn SumNode) Sum() nodes.StructOutput[float64] {
94+
func (sn SumNode) Sum(out *nodes.StructOutput[float64]) {
8995
var total float64
90-
for _, v := range sn.Values {
91-
if v == nil {
92-
continue
93-
}
94-
total += v.Value()
96+
values := nodes.GetOutputValues(out, sn.Values)
97+
for _, v := range values {
98+
total += v
9599
}
96-
return nodes.NewStructOutput(total)
100+
out.Set(total)
97101
}
98102
```
99103

@@ -103,7 +107,7 @@ Now that you've defined your node, you need to take steps to include it in a bui
103107

104108
### Registering the Package
105109

106-
The standard way to register your nodes with the graph system is to define a `init` function in your package. You can read more about it's [specifics of execution here](https://go.dev/doc/effective_go#init).
110+
The standard way to register your nodes with the graph system is to define a `init` function in your package. You can read more about its [specifics of execution here](https://go.dev/doc/effective_go#init).
107111

108112
Inside the init function, you create a `TypeFactory` which collects all the types your package wants to register. Then you pass that factory to `generator.RegisterTypes(factory)` to integrate your custom nodes into Polyform’s node registry.
109113

0 commit comments

Comments
 (0)