Skip to content

CONSIDER - YAML library replacement #280

Open
@lostsnow

Description

@lostsnow
Member

Currently the yaml library go-yaml/yaml no longer appears to be under active development, which means that some issues may not be solved upstream:

So there are a couple of options right now

  • Keep using yaml.v2.
  • Use yaml.v3 and follow the v3 format (this option might as well keep the yaml.v2 option in use now).
  • Switch to another yaml library (may have to follow their format too), such as:

Activity

Volte6

Volte6 commented on Mar 16, 2025

@Volte6
Member

goccy looks promising.

I think we need to evaluate what effects this will have and what would need to be changed/updated.

Jasrags

Jasrags commented on Apr 23, 2025

@Jasrags
Contributor

I’d suggest we use yaml.v3 if it’s compatible with the functionality we need. I've been using it in a few projects without issue. While upstream maintenance is a concern, v3 currently seems stable enough for typical use cases.

One way to future-proof ourselves would be to define an interface around our YAML functionality. This would allow us to isolate any future changes or library swaps behind a single implementation, making it easy to adapt if we decide to switch libraries later on.

Here’s a simple example of what that interface might look like:

type YAML interface {
    LoadFromFile(path string, out interface{}) error
    SaveToFile(path string, in interface{}) error
}
lostsnow

lostsnow commented on Apr 25, 2025

@lostsnow
MemberAuthor

There is an additional problem, the formatting is not consistent from version to version or library to library, so the yaml files under _datafiles change a lot.

Jasrags

Jasrags commented on Apr 25, 2025

@Jasrags
Contributor

Given we have access to the current and new library it would not be too much effort to write a go program to read in and then rewrite the yaml files.

lostsnow

lostsnow commented on Apr 25, 2025

@lostsnow
MemberAuthor

go-yaml/yaml#865

Yes, rewrite the Marshal() function is a way

self-assigned this
on Apr 28, 2025
Jasrags

Jasrags commented on Apr 28, 2025

@Jasrags
Contributor

I'm going to start digging into this to see the level of effort to switch and expected changes.

Volte6

Volte6 commented on Apr 28, 2025

@Volte6
Member
Jasrags

Jasrags commented on Apr 28, 2025

@Jasrags
Contributor

Understood, will still dig into overall evaluation on what we should do going forward.

Jasrags

Jasrags commented on Apr 28, 2025

@Jasrags
Contributor

Here are some benchmarks

Sample Struct (NOTE: sigs_k8s_io_yaml uses the json tag for yaml parsing)

type SampleData struct {
	Name         string            `yaml:"name" json:"name"`
	Count        int               `yaml:"count" json:"count"`
	Tags         []string          `yaml:"tags" json:"tags"`
	Metadata     map[string]string `yaml:"metadata" json:"metadata"`
	Active       bool              `yaml:"active" json:"active"`
	FloatValue   float64           `yaml:"float_value" json:"float_value"`
	Int64Value   int64             `yaml:"int64_value" json:"int64_value"`
	UintValue    uint              `yaml:"uint_value" json:"uint_value"`
	ByteValue    byte              `yaml:"byte_value" json:"byte_value"`
	RuneValue    rune              `yaml:"rune_value" json:"rune_value"`
	EmbeddedData SampleEmbdedData  `yaml:"embedded_data" json:"embedded_data"`
}

type SampleEmbdedData struct {
	ID   string `yaml:"id" json:"id"`
	Name string `yaml:"name" json:"name"`
}

BenchmarkMarshal

goos: darwin
goarch: amd64
pkg: yamleval
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkMarshal/yaml_v2-12         	   61995	     18220 ns/op	   24693 B/op	      98 allocs/op
BenchmarkMarshal/yaml_v3-12         	   57420	     20769 ns/op	   37436 B/op	      99 allocs/op
BenchmarkMarshal/sigs_k8s_io_yaml-12         	   20882	     57766 ns/op	   81010 B/op	     330 allocs/op
BenchmarkMarshal/goccy_go-yaml-12            	   26550	     45611 ns/op	   29100 B/op	     767 allocs/op

BenchmarkUnmarshal

goos: darwin
goarch: amd64
pkg: yamleval
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkUnmarshal/yaml_v2-12         	   49386	     22921 ns/op	   11806 B/op	     185 allocs/op
BenchmarkUnmarshal/yaml_v3-12         	   39450	     30325 ns/op	   15480 B/op	     218 allocs/op
BenchmarkUnmarshal/sigs_k8s_io_yaml-12         	   28678	     40945 ns/op	   18688 B/op	     320 allocs/op
BenchmarkUnmarshal/goccy_go-yaml-12            	   19525	     61845 ns/op	   39045 B/op	     773 allocs/op

Output files

test_goccy_go-yaml.yaml

name: Example
count: 42
tags:
- alpha
- beta
metadata:
  env: production
  version: "1.0"
active: true
float_value: 3.14
int64_value: 1234567890123456789
uint_value: 42
byte_value: 255
rune_value: 65
embedded_data:
  id: "123"
  name: Embedded Example

test_sigs_k8s_io_yaml.yaml

active: true
byte_value: 255
count: 42
embedded_data:
  id: "123"
  name: Embedded Example
float_value: 3.14
int64_value: 1234567890123456789
metadata:
  env: production
  version: "1.0"
name: Example
rune_value: 65
tags:
- alpha
- beta
uint_value: 42

test_yaml_v2.yaml

name: Example
count: 42
tags:
- alpha
- beta
metadata:
  env: production
  version: "1.0"
active: true
float_value: 3.14
int64_value: 1234567890123456789
uint_value: 42
byte_value: 255
rune_value: 65
embedded_data:
  id: "123"
  name: Embedded Example

test_yaml_v3.yaml

name: Example
count: 42
tags:
    - alpha
    - beta
metadata:
    env: production
    version: "1.0"
active: true
float_value: 3.14
int64_value: 1234567890123456789
uint_value: 42
byte_value: 255
rune_value: 65
embedded_data:
    id: "123"
    name: Embedded Example

test_yaml_output.zip

Jasrags

Jasrags commented on Apr 28, 2025

@Jasrags
Contributor

Overall, I think a good next step would be to add the following interface and implementation of the v2 engine and move usage to that.

// YAMLEngine defines a common interface for YAML marshal/unmarshal engines.
type YAMLEngine interface {
	Name() string
	Marshal(v any) ([]byte, error)
	Unmarshal(data []byte, v any) error
}

// YAMLV2Engine adapts gopkg.in/yaml.v2 to YAMLEngine.
type YAMLV2Engine struct{}

func (e YAMLV2Engine) Name() string {
	return "yaml_v2"
}

func (e YAMLV2Engine) Marshal(v any) ([]byte, error) {
	return yamlv2.Marshal(v)
}

func (e YAMLV2Engine) Unmarshal(data []byte, v any) error {
	return yamlv2.Unmarshal(data, v)
}
Volte6

Volte6 commented on Apr 28, 2025

@Volte6
Member

We need to understand how this will affect current yaml usage, especially around struct tags:

Supported/Used tags:

  • 'omitempty'
  • '-'
  • 'flow'

Custom Tags:

  • 'env'

Custom Marshal/Unmarshal

Also, how it will affect the requirements around how we layout the yaml - will the current yaml remain valid? Will it need to be mass-updated in some way? For example, are tabbing changes required? Do rules for 'empty' field detection change?

Jasrags

Jasrags commented on Apr 28, 2025

@Jasrags
Contributor

I'll add those examples into the tests and add the results along with diffs, overall it seems like minor spacing changes on list items.

Jasrags

Jasrags commented on Apr 28, 2025

@Jasrags
Contributor

diff --color data/test_yaml_v2.yaml data/test_goccy_go-yaml.yaml

no diff

diff --color data/test_yaml_v2.yaml data/test_sigs_k8s_io_yaml.yaml

1,8d0
< name: Example
< count: 42
< tags:
< - alpha
< - beta
< metadata:
<   env: production
<   version: "1.0"
10,12d1
< float_value: 3.14
< int64_value: 1234567890123456789
< uint_value: 42
14c3
< rune_value: 65
---
> count: 42
17a7
> float_value: 3.14
18a9,18
> int64_value: 1234567890123456789
> metadata:
>   env: production
>   version: "1.0"
> name: Example
> rune_value: 65
> tags:
> - alpha
> - beta
> uint_value: 42

diff --color data/test_yaml_v2.yaml data/test_yaml_v3.yaml

4,5c4,5
< - alpha
< - beta
---
>     - alpha
>     - beta
7,8c7,8
<   env: production
<   version: "1.0"
---
>     env: production
>     version: "1.0"
16,17c16,17
<   id: "123"
<   name: Embedded Example
---
>     id: "123"
>     name: Embedded Example

4 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    Participants

    @Jasrags@Volte6@lostsnow

    Issue actions

      CONSIDER - YAML library replacement · Issue #280 · GoMudEngine/GoMud