v2.1.0

Summary

The v2.1.0 release of f1 is closer to a complete rewrite, than a feature release. Includes massive
performance, security, usability and maintainability improvements.

Transforming f1 into a modern, high-quality and efficient package.

Highlights

• Massive performance improvements, by redesigning the work scheduling and removing per-iteration
  memory allocations.
  • Performance no longer degrades with time.
  • Significantly more iterations per second, with significantly less resources used:
    • users low(2) concurrency: Up to 33 times more iterations per second (390k -> 12.8 mil) with 1.5x less user cpu time
      and 23x less system CPU time.
    • users high(100) concurrency: Up to 28 times more iterations per second (173k -> 5mil) with 34x
      less system CPU time and better CPU utilisation.
    • constant high(1000) concurrency: can sustain 10,000 IPS with 1000 workers, without dropped
      iterations(previously 20% dropped) while using 300x less CPU time.
• Rework output formatting and support structured logging, including JSON logs with LOG_FORMAT=json,
  log level support with LOG_LEVEL, and providing custom loggers to f1. Migrate logger to
  log/slog with logrus compatibility layer.
• Remove archived, abandoned, single-use and unnecessary dependencies to reduce the package footprint
  and mitigate supply chain attacks.
• Secure the CI pipeline with strict permissions, SHA version locking and CodeQL.
• Test suite now takes seconds(11s) instead of minutes(1m 5s) with better coverage.
• Features:
  • Require go 1.22
  • Support Bash v2 completions
  • Support CPU & Memory profiling with filename completion
  • Support custom start-time for staged tests
• Fixes:
  • exit after a timeout on stuck iterations
  • don't panic on unknown scenario name
  • change t.Log level to Info from Error
  • Fix rounding on iteration-distribution calculations

Detailed

Breaking

• remove tracer
• Remove fluentd integration
• remove --verbose-fail flag

Deprecations

• deprecate GetMetrics()
• deprecate --verbose-fail, t.Logger() and t.NewT

Performance

• allocations: change testing.T iteration value
• allocations: remove per iteration allocations
• allocations/cpu: remove performance impact of progress reporting
• allocations/cpu: don't start a goroutine for each iteration
• cpu: rewrite iteration management to improve throughput
• cpu: rewrite work scheduling to improve performance

Features

• Require go 1.22
• Support Bash v2 completions
• Support CPU & Memory profiling with filename completion
• Support custom start-time for staged tests

Fixes

• exit after a timeout on stuck iterations
• don't panic on unknown scenario name
• change t.Log level to Info from Error
• Fix rounding on iteration-distribution calculations

Usability

• Remove panic logs and replace with errors

Testing

• rewrite prometheus test code to allow parallel tests
• allow running tests in parallel
• fix randomly failing TestRandomDistribution test
• run tests in parallel
• add example cmd to be used for benchmarking
• more readable run_test failures

CI

• Update GH actions
• Use gorelease to check for breaking API changes
• Enable CodeQL
• Ignore cache when running tests

Dependencies

• Remove archived https://github.com/pkg/errors dependency and use stdlib error handling

• Remove archived https://github.com/giantswarm/retry-go test dependency:

  • replace retry lib with avast/retry-go
  • remove avast/retry-go dependency

• Remove github.com/asaskevich/govalidator dependency

• Remove freeport dependency

• Remove obsolete go-atamanand replace with stdlib

• Remove obsolete ggtimer dependecy and replace with stdlib

• Remove go-gaussian and replace with custom and faster implementation

• Remove durafmt dependency and replace with stdlib usage

• Remove google/uuid dependency and replace with stdlib

• Remove logrus_fluent dependency

• Update dependencies to latest versions:

  • update dependencies & remove vendor folder
  • bump dependencies
  • bump github.com/prometheus/common from 0.54.0 to 0.55.0
  • bump github.com/spf13/cobra from 1.8.0 to 1.8.1
  • bump github.com/prometheus/common from 0.53.0 to 0.54.0
  • bump github.com/prometheus/client_golang from 1.19.0 to 1.19.1
  • bump github.com/prometheus/common from 0.52.3 to 0.53.0
  • bump github.com/prometheus/common from 0.52.2 to 0.52.3
  • bump github.com/prometheus/common from 0.51.1 to 0.52.2
  • bump github.com/prometheus/client_model from 0.6.0 to 0.6.1
  • bump github.com/guptarohit/asciigraph from 0.6.0 to 0.7.1
  • bump github.com/prometheus/common from 0.50.0 to 0.51.1
  • bump github.com/guptarohit/asciigraph from 0.5.6 to 0.6.0

Linting

Run golangci-lint as part of CI and enable most linters:

• use golangci-lint & fix formatting
• enable more linters
• enable gochecknoglobals linter
• enable more linters
• bump golangci-lint to 1.57.1
• update golangci-lint to 1.57.2
• bump golangci-lint to 1.58.0
• security: check golangci-lint signatures on installing
• bump golangci-lint to 1.58.2
• enable copyloopvar linter
• bump golangci-lint to 1.59.0
• bump golangci-lint to 1.59.1
• enable testpackage linter

Documentation:

• Cleanup README.md
• Document ENV variables

Cleanup

• remove unused snyk file
• consistent ENV setting handling
• testing: simplify goleak check
• simplify template rendering
• metrics: remove magic strings & panic
• redo gauss calculations
• inject printer to allow output stream redirect
• use constants for flag names
• allow using non-global metrics instance
• remove uuid dependency & more readable test output
• use atomic types instead of function on primitives
• use full go version
• cleanup logrus usage
• handle interrupts on the command level & introduce command context
• use the printer output stream instead of os.Stdout
• add toolchain to go.mod
• security: add top level token permissions to CI jobs
• tracer: remove variadic arguments to avoid slice allocations
• increase iteration counter to uint64
• metrics: remove unused Teardown summary vec
• cleanup unused CancellableTimer code
• remove unneeded int conversion
• simplify metric recording & progress
• rework cli output rendering
• cleanup test output
• min and max times in totals progress reporting
• rewrite raterun to make it more obvious
• fix rate per second reporting

What's Changed

• Bump github.com/prometheus/client_golang from 1.17.0 to 1.18.0 by @dependabot in #148
• Bump github.com/google/uuid from 1.4.0 to 1.6.0 by @dependabot in #151
• Bump go.uber.org/goleak from 1.2.0 to 1.3.0 by @dependabot in #143
• chore: use golangci-lint & fix formatting by @nvloff-f3 in #158
• chore: enable more linters by @nvloff-f3 in #159
• chore: remove pkg/errors usage by @nvloff-f3 in #160
• chore: replace retry lib with avast/retry-go by @nvloff-f3 in #161
• chore: update actions & add breaking change guard by @nvloff-f3 in #162
• chore: remove govalidator lib usage by @nvloff-f3 in #163
• chore: update dependencies & remove vendor folder by @nvloff-f3 in #164
• chore: remove unused snyk file by @nvloff-f3 in #167
• feat: rework root cmd, profiling & completions by @nvloff-f3 in #166
• feat: extract env settings by @nvloff-f3 in #168
• chore: rewrite prometheus test code by @nvloff-f3 in #169
• chore: rework templating & remove go-ataman by @nvloff-f3 in #181
• chore: ignore cache when running tests by @nvloff-f3 in #184
• chore(metrics): remove magic strings & panic by @nvloff-f3 in #187
• chore: enable gochecknoglobals linter by @nvloff-f3 in #186
• chore: simplify goleak check by @nvloff-f3 in #180
• chore: enable more linters by @nvloff-f3 in #188
• feat: bump go to 1.22 & golangci-lint to 1.57.1 by @nvloff-f3 in #189
• chore(deps): bump github.com/prometheus/common from 0.50.0 to 0.51.1 by @dependabot in #191
• chore(deps): bump github.com/guptarohit/asciigraph from 0.5.6 to 0.6.0 by @dependabot in #190
• chore: remove ggtimer dependency by @nvloff-f3 in #193
• chore: update golangci-lint to 1.57.2 by @nvloff-f3 in #192
• chore: redo gauss calculations by @nvloff-f3 in #195
• chore(deps): bump github.com/guptarohit/asciigraph from 0.6.0 to 0.7.1 by @dependabot in #194
• chore: inject printer to allow output stream redirect by @nvloff-f3 in #196
• chore: enable codeql by @nvloff-f3 in #165
• chore: remove durafmt by @nvloff-f3 in #197
• chore: constant flag names by @nvloff-f3 in #198
• chore(deps): bump github.com/prometheus/client_model from 0.6.0 to 0.6.1 by @dependabot in #201
• chore(deps): bump github.com/prometheus/common from 0.51.1 to 0.52.2 by @dependabot in #202
• chore: allow using non-global metrics instance by @nvloff-f3 in #200
• chore: remove uuid dependency & more readable test output by @nvloff-f3 in #203
• chore: use atomic types instead of function on primitives by @nvloff-f3 in #204
• chore: allow running tests in parallel by @nvloff-f3 in #205
• fix: handle interrupts on the command level by @nvloff-f3 in #206
• fix: use the printer output stream instead of os.Stdout by @nvloff-f3 in #207
• chore: fix TestRandomDistribution test by @nvloff-f3 in #208
• chore: add toolchain to go.mod by @nvloff-f3 in #209
• chore: add top level token permissions to CI jobs by @nvloff-f3 in #210
• chore: run tests in parallel by @nvloff-f3 in #211
• chore: use full go version by @nvloff-f3 in #212
• OpenSSF Scorecard by @nvloff-f3 in #199
• feat: logger changes by @nvloff-f3 in #213
• chore: fix scorecard upload by @nvloff-f3 in #215
• chore(deps): bump github.com/prometheus/common from 0.52.2 to 0.52.3 by @dependabot in #214
• chore: add cmd to be used for benchmarking by @nvloff-f3 in #216
• fix(tracer): remove variadic arguments by @nvloff-f3 in #217
• feat(perf): change testing.T iteration value by @nvloff-f3 in #218
• feat: increase iteration counter to uint64 by @nvloff-f3 in #219
• BREAKING: don't expose internal metrics by @nvloff-f3 in #220
• fix(metrics): remove unused Teardown summary vec by @nvloff-f3 in #223
• chore: cleanup unused CancellableTimer code by @nvloff-f3 in #224
• chore: remove unneeded int conversion by @nvloff-f3 in #225
• chore(tests): more readable run_test failures by @nvloff-f3 in #226
• chore(deps): bump github.com/avast/retry-go/v4 from 4.5.1 to 4.6.0 by @dependabot in #222
• chore(deps): bump github.com/prometheus/common from 0.52.3 to 0.53.0 by @dependabot in #221
• Revert breaking metrics change by @nvloff-f3 in #227
• chore(metrics): simplify metric recording & progress by @nvloff-f3 in #228
• feat: remove per iteration allocations by @nvloff-f3 in #229
• fix: don't panic on unknown scenario name by @nvloff-f3 in #230
• Revert "chore(profiling): don't run GC on heap profile" by @nvloff-f3 in #231
• chore: bump golangci-lint to 1.58.0 by @nvloff-f3 in #232
• feat: support custom start-time for staged tests by @sshutovskyi-f3 in #233
• chore: check golangci-lint signatures on installing by @nvloff-f3 in #234
• chore: update scorecard action by @nvloff-f3 in #235
• refactor(result): rework cli output rendering by @nvloff-f3 in #236
• chore(deps): bump github.com/prometheus/client_golang from 1.19.0 to 1.19.1 by @dependabot in #237
• feat: remove performance impact of progress reporting by @nvloff-f3 in #238
• feat: don't start a goroutine for each iteration by @nvloff-f3 in #239
• chore: bump golangci-lint to 1.58.2 by @nvloff-f3 in #240
• chore: remove scorecard action by @nvloff-f3 in #241
• chore: enable copyloopvar linter by @nvloff-f3 in #242
• chore: cleanup test output by @nvloff-f3 in #243
• feat: rewrite iteration management to improve throughput by @nvloff-f3 in #244
• chore: bump golangci-lint to 1.59.0 by @nvloff-f3 in #247
• fix: min and max times in totals progress reporting by @nvloff-f3 in #249
• chore: remove avast/retry-go dependency by @nvloff-f3 in #248
• fix(progress): fix rate per second reporting by @nvloff-f3 in #251
• refactor(progress): rewrite raterun to make it more obvious by @nvloff-f3 in #250
• feat: rewrite work scheduling to improve performance by @nvloff-f3 in #252
• chore(deps): bump github.com/prometheus/common from 0.53.0 to 0.54.0 by @dependabot in #253
• Fix rounding on iteration-distribution calculations by @jrpeon-form3 in #141
• chore: bump golangci-lint to 1.59.1 by @nvloff-f3 in #255
• feat: remove tracer by @nvloff-f3 in #254
• chore: enable testpackage linter by @nvloff-f3 in #256
• feat: remove fluentd integration by @nvloff-f3 in #257
• chore(deps): bump github.com/spf13/cobra from 1.8.0 to 1.8.1 by @dependabot in #259
• chore(deps): bump github.com/prometheus/common from 0.54.0 to 0.55.0 by @dependabot in #261
• feat: rewrite output by @nvloff-f3 in #260
• feat: deprecate GetMetrics() by @nvloff-f3 in #262
• feat: change t.Log level to Info from Error by @nvloff-f3 in #263
• chore: Cleanup README.md by @nvloff-f3 in #264
• chore: bump dependencies by @nvloff-f3 in #266
• feat: support user provided loggers by @nvloff-f3 in #267
• fix: exit after a timeout on stuck iterations by @nvloff-f3 in #265

New Contributors

• @nvloff-f3 made their first contribution in #158
• @sshutovskyi-f3 made their first contribution in #233
• @jrpeon-form3 made their first contribution in #141

Full Changelog: v2.0.22...v2.1.0