v1.6.5

Release 1.6.5

What's New

Bugfixes and dependency updates.

Component Updates and Bug Fixes

• github.com/openziti/channel/v4: v4.2.8 -> v4.2.13

  • Issue #194 - Add GetUnderlays and GetUnderlayCountsByType to Channel

• github.com/openziti/foundation/v2: v2.0.66 -> v2.0.69

  • Issue #443 - Allow injecting custom method into go-routine pools, to allow identifying them in stack dumps

• github.com/openziti/identity: v1.0.105 -> v1.0.108

• github.com/openziti/metrics: v1.4.1 -> v1.4.2

• github.com/openziti/runzmd: v1.0.73 -> v1.0.76

• github.com/openziti/storage: v0.4.17 -> v0.4.20

• github.com/openziti/transport/v2: v2.0.177 -> v2.0.180

• github.com/openziti/xweb/v2: v2.3.3 -> v2.3.4

• github.com/openziti/ziti: v1.6.3 -> v1.6.5

  • Issue #3149 - add dial/bind type column to sp list

v1.6.3

Release 1.6.3

What's New

• Router Network Interface Discovery

Router Interface Discovery

Routers can now discover their network interfaces and publish this information to the
controller.

This feature will be used in the future to allow controller side configuration of
router link listeners and edge listeners.

Update Router Configuration

There is new router configuration to manage this:

interfaceDiscovery:
  # This feature is enabled by default, but can be disabled by setting this to true
  disabled: false

  # How often to poll for interface changes. Defaults to 1 minute
  checkInterval: 1m

  # How often to report the current set of interfaces, when nothing has changed.
  # This is a failsafe reporting mechanism, in the very unlikely event that an 
  # earlier change report was lost or disregarded due to distributed controller
  # eventual consistency
  minReportInterval: 24h

Update REST APIs

Network interfaces, where reported, can now be viewed on the following endpoints.

• routers
• edge-routers
• identities (if the router is an ER/T, with an associated identities)

At some point in the future, we expect to allow SDKs to also optionally report their
network interfaces as well. Those will be available via the identities REST API.

Example:

$ ziti fabric list routers 'name="edge-router-1"' -j | jq
{
  "data": [
    {
      "_links": {
        "self": {
          "href": "./routers/oLvcT6VepI"
        },
        "terminators": {
          "href": "./routers/oLvcT6VepI/terminators"
        }
      },
      "createdAt": "2025-03-24T04:35:59.077Z",
      "id": "oLvcT6VepI",
      "tags": {},
      "updatedAt": "2025-06-11T14:29:22.083Z",
      "connected": false,
      "cost": 0,
      "disabled": false,
      "fingerprint": "b7b03c55be77df0ec57e49d8fe2610e0b99fa61c",
      "interfaces": [
        {
          "addresses": [
            "192.168.3.29/24",
            "aaaa::aaaa:aaaa:aaaa:aaaa/64"
          ],
          "hardwareAddress": "aa:aa:aa:aa:aa:aa",
          "index": 4,
          "isBroadcast": true,
          "isLoopback": false,
          "isMulticast": true,
          "isRunning": true,
          "isUp": true,
          "mtu": 1500,
          "name": "wifi0"
        },
        {
          "addresses": null,
          "hardwareAddress": "aa:aa:aa:aa:aa:aa",
          "index": 2,
          "isBroadcast": true,
          "isLoopback": false,
          "isMulticast": true,
          "isRunning": false,
          "isUp": true,
          "mtu": 1500,
          "name": "eth1"
        },
        {
          "addresses": [
            "192.168.1.2/24",
            "aaaa::aaaa:aaa:aaaa:aaaa/64"
          ],
          "hardwareAddress": "aa:aa:aa:aa:aa:aa",
          "index": 16,
          "isBroadcast": true,
          "isLoopback": false,
          "isMulticast": true,
          "isRunning": true,
          "isUp": true,
          "mtu": 1500,
          "name": "eth0"
        },
        {
          "addresses": [
            "127.0.0.1/8",
            "::1/128"
          ],
          "hardwareAddress": "",
          "index": 1,
          "isBroadcast": false,
          "isLoopback": true,
          "isMulticast": false,
          "isRunning": true,
          "isUp": true,
          "mtu": 65536,
          "name": "lo"
        }
      ],
      "listenerAddresses": null,
      "name": "edge-router-1",
      "noTraversal": false
    }
  ],
  "meta": {
    "filterableFields": [
      "id",
      "isSystem",
      "name",
      "fingerprint",
      "cost",
      "createdAt",
      "updatedAt",
      "tags",
      "noTraversal",
      "disabled",
      "connected"
    ],
    "pagination": {
      "limit": 10,
      "offset": 0,
      "totalCount": 1
    }
  }
}

Note that addresses have been sanitized.

Component Updates and Bug Fixes

• github.com/openziti/agent: v1.0.27 -> v1.0.29

• github.com/openziti/channel/v4: v4.2.0 -> v4.2.8

• github.com/openziti/edge-api: v0.26.45 -> v0.26.46

  • Issue #155 - Add network interface list to routers and identities

• github.com/openziti/foundation/v2: v2.0.63 -> v2.0.66

• github.com/openziti/identity: v1.0.101 -> v1.0.105

• github.com/openziti/runzmd: v1.0.72 -> v1.0.73

• github.com/openziti/sdk-golang: v1.1.1 -> v1.1.2

  • Issue #742 - Additional CtrlId and GetDestinationType for inspect support
  • Issue #739 - go-jose v2.6.3 CVE-2025-27144 resolution
  • Issue #735 - Ensure Authenticate can't be called in parallel

• github.com/openziti/secretstream: v0.1.34 -> v0.1.36

• github.com/openziti/storage: v0.4.11 -> v0.4.17

  • Issue #106 - panic in TypedBucket.GetList

• github.com/openziti/transport/v2: v2.0.171 -> v2.0.177

• github.com/openziti/xweb/v2: v2.3.2 -> v2.3.3

• github.com/openziti/ziti: v1.6.2 -> v1.6.3

  • Issue #3124 - ids used by circuits and ingress/egress can conflict in an HA setup
  • Issue #3117 - authenticators LastAuthResolvedToRoot not set, createdAt/lastUpdateAt zero zulu
  • Issue #3111 - Add API for xgress router factories allowing router env injection
  • Issue #3119 - Using the same heartbeatmsg instance across channels causes data race
  • Issue #3115 - Fix racy link state access in router link registry
  • Issue #3113 - Close links when link groups no longer indicate that a link should be allowed
  • Issue #3082 - Add network interfaces to controller data model
  • Issue #3083 - Add optional network interface discovery to routers
  • Issue #2862 - Large scale data-flow test
  • Issue #3102 - Implement remote control for ziti-traffic-test/loop4
  • Issue #3098 - Implement circuit validation API and CLI

v1.6.2

Release 1.6.2

What's New

• System Certificate Authentication Improper Chain Detection
• Authentication Events
• Multi-underlay channel group secret
• Flag to disable posture check functionality

System Certificate Authentication Improper Chain Detection

Previous versions of SDKs and controllers issued and stored client certificate chains with differing levels of fidelity.
Depending on when an identity was enrolled, it may or may not have a proper client certificate chain. In single
controller environments, the controller will automatically include its known intermediates to help create valid
x509 certificate chains back to the network's Root CA.

In HA environments, each controller does not know of all the intermediates in the system by default. In order to support
dynamically scaling controllers, clients must store and provide a proper client certificate chain (a leaf certificate
followed by all necessary intermediate CA certs). Identities enrolled with SDKs that did not store the chain
or controllers that did not provide the chain will encounter authentication issues in multi-controller environments.

To help determine which identities have issues, the system has been augmented to allow detection of problematic
certificate authenticators. Firstly, authenticators have flags on them that are set during authentication to detect
the current behavior of a specific client that owns and is invoking the certificate authenticator.

Detecting Improper Client Cert Chains As A Client SDK

The current API Sessions endpoint (https://<host>/edge/client/v1/current-api-session) now returns a value named
improperClientCertChain as a boolean value. If it is not present or false no action should be taken. If true
it means that the current API Session was authenticated with an internal PKI issued certificate where the client
did not provide a full chain to the root CA during authentication; indicating a problem with the certificate storage
mechanism in the application or due to the controller version used during enrollment/extension not providing a chain.
The SDK should proactively opt to begin certificate extension on its own to obtain a proper chain. Authentication
succeeded in this case because the controller relied upon a deprecated certificate pool that happen to include the necessary
intermediate CAs.

Detecting Clients Without Proper Chains

After a client has authenticated with the system via a network-issued certificate at least one time, a number of fields
are set depending on what the client provided. These values can be reviewed via the Edge Management API for
Authenticators (edge/management/v1/authenticators). Where an authenticator has isIssuedByNetwork set to true and
lastAuthResolvedToRoot set to false, indicates that the related identity/client is not providing a chain when it
should.

Additionally, if authenticator events are enabled and being processed, events will have a field
improper_client_cert_chain set to true (see Authentication Events below)

Fixing Clients Chains

Once an authenticator has been identified as problematic, an administrator should verify the client is using the newest
possible versions of its SDK and either re-enroll it or request the identity to extend the next time it
authenticates. Both scenarios result in a new certificate and chain being provided to the client.

• Re-Enrollment removes the current authenticator, making authentication impossible until enrollment is completed.
  Once completed, a new authenticator is created, and the new certificate generated from the process can be used to
  authenticate. Humans or other external automation processes have to deliver and consume the new enrollment JWT.
• Extension leaves the authenticator in place and authentication can still occur. When the extension process is
  completed, the authenticator is updated and will use the new certificate generated from the process. SDKs will
  handle the process by raising SDK specific events to drive the process.

While related and similar, using one over the other depends on the situation. Re-Enrollment is best for when
an client or its host are unrecoverable; either through damage, decommissioning, or suspected to be compromised.
Extension is useful when the client and host are in a known good status and one simply wants to issue new
certificates to said client. Extension is also "hands off" from the client as long as the client is using
SDKs that support enrollment request/key rolling.

Useful Authenticator Values

Authenticators are an entity within OpenZiti and can be queried via the CLI (ziti edge list authenticators) or
Edge Management API (GET https://<host>/edge/management/v1/authenticators). Below are properties that are
useful for determining clients with improper chains.

• isIssuedByNetwork (boolean) indicates the authenticator is a certificate that was issued by the network
• isExtendRequested (boolean) indicated the authenticator will report to the client to extend its certificate on next
  authentication.
• isKeyRollRequested (boolean) indicates if key rolling is requested with an outstanding isExtendRequested
• extendRequestedAt (string, date time) indicates when an outstanding extension was requested, otherwise null
  or not provided
• lastAuthResolvedToRoot (boolean) indicates if the last time the client authenticated, it provided a certificate
  chain that resolved to the root CA. Only valid if isIssuedByNetwork is true.

Re-Enrollment

Re-enrollment can be accomplished via the CLI or the Edge Management API. Re-enrollment removes the current authenticator,
stopping the client from authenticating. All of the underlying configuration for the related identity is preserved.
The newly created enrollment will have a new enrollment JWT that will be consumed during the enrollment process and
will result in a new client certificate and chain.

CLI:

ziti edge update authenticator cert <id> --re-enroll [--duration <duration>]

Edge Management API:

POST /edge/management/v1/authenticators/:id/re-enroll

{
    expiresAt: "2025-05-21T13:47:26Z"
}

Both of the above provide an enrollment id and or enrollment JWT token as output. These can be consumed in various
OpenZiti applications or CLI commands.

CLI Enrollment

 ziti edge enroll -h
enroll an identity

Usage:
  ziti edge enroll path/to/jwt [flags]

Flags:
      --ca string         Additional trusted certificates
  -c, --cert string       The certificate to present when establishing a connection.
  -h, --help              help for enroll
  -n, --idname string     Names the identity. Ignored if not 3rd party auto enrollment
  -j, --jwt string        Enrollment token (JWT file). Required
  -k, --key string        The key to use with the certificate. Optionally specify the engine to use. supported engines: [parsec]
  -a, --keyAlg RSA|EC     Crypto algorithm to use when generating private key (default RSA)
  -o, --out string        Output configuration file.
  -p, --password string   Password for updb enrollment, prompted if not provided and necessary
      --rm                Remove the JWT on success
  -u, --username string   Username for updb enrollment, prompted if not provided and necessary
  -v, --verbose           Enable verbose logging

OpenZiti Tunneler Enrollment

ziti-edge-tunnel add --jwt "$(< ./in-file.jwt)" --identity myIdentityName

Request Extension

Extension leaves the authenticator in place, but notifies SDKs that they should generate a CSR to request a new
certificate. Once the process is complete, the client will have a new certificate and chain that must be used
for later authentication requests. This can be initiated through the CLI or Edge Management API.

CLI:

ziti edge update authenticator cert <id> --request-extend [--request-key-roll]

Edge Management API:

POST /edge/management/v1/authenticators/:id/request-extend

{
    rollKeys: true,
}

These commands will cause the authenticator to be updated to have isExtendRequested, extendRequestedAt, and optionally isKeyRollRequested` updated on the authenticator. These values will be used to signal the client to
extend their certificate on the next successful authentication attempt.

Authentication Events

A new event names space authentication has been added for events with two types success and fail. These can be
enabled via the controller configuration files.

Event Documentation:

// An AuthenticationEvent is emitted when an authentication attempt is made
//
// Types of authentication events
//   - fail - authentication failed
//   - success - authentication succeeded
//
// Types of authentication methods
//   - updb - username password from the internal database
//   - cert - a certificate, either first party or 3rd party
//   - ext-jwt - an external JWT from an IDP

Example Controller Configuration:

events:
  jsonLogger:
    subscriptions:
      - type: authentication
#          - success
#          - fail
    handler:
      type: file
      format: json
      path: ${TMPDIR}/ziti-events.log

Example Output:

{
  "namespace": "authentication",
  "event_src_id": "ctrl_client",
  "timestamp": "2025-05-21T10:24:32.6532054-04:00",
  "event_type": "fail",
  "type": "password",
  "authenticator_id": "B0JrMDPGxd",
  "external_jwt_signer_id": "",
  "identity_id": "BGJabPP0K",
  "auth_policy_id": "default",
  "remote_address": "127.0.0.1:51587",
  "success": false,
  "reason": "could not authenticate, password does not match",
  "improper_client_cert_chain": false
}

Multi-underlay channel group secret

For additional security the experimental multi-underlay channel code now requires that
clients provide a shared secret. This ensures that channels are get the expected
underlays without requiring much larger group ids. On the client side this will requ...

v1.6.1

Release 1.6.1

What's New

• Bug fixes and library updates
• Ability to request that SDKs extend and optionally roll their key
• Address translations can now be specified in host.v1 service configuration

Ability to request that SDKs extend and optionally roll their key

It is now possible for administrators to flag specific certificate authenticators as needed to extend their current
certificate early and/or optionally roll the keypair that underpins the certificate. This capability only works for
certificates issued by the OpenZiti network. If '3rd party CAs' are in use, those certificate authenticators will not
work with this system.

SDKs must support this capability for it to have any effect, and the application utilizing the SDK must respond to the
certificate extension events to store certificate credentials.

This capability is located in the Management API at /edge/management/v1/authenticators/{id>/request-extend.
Its payload is currently and optional boolean value for rollKeys that can be set to true/false and defaults to
false if not provided.

This can also be issued via the CLI:

> ziti edge update authenticator cert -h
Request a specific certificate authenticator to --requestExtend or --requestKeyRoll, --requestKeyRoll implies --requestExtend

Usage:
  ziti edge update authenticator cert <authenticatorId> [--requestExtend] [--requestKeyRoll] [flags]

Flags:
  -h, --help             help for cert
  -e, --requestExtend    Specify the certificate authenticator should be flagged for extension
  -r, --requestKeyRoll   Specify the certificate authenticator should be flagged for key rolling, implies --requestExtend

Requesting an extension flags new fields on a certificate authenticator in the values isExtendRequest and
isKeyRollRequested. These values are set to false after the client performs a certificate extension. The CLI
has been updated to report these values on certificate authenticators via ziti edge list authenticators.

These values are also present on the /edge/client/v1/current-api-session endpoint when a client has use certificate
authentication to initiate an API Session using a certificate authenticator.

Additionally, a log of key rolling activity per authenticator will be available in a future release.

host.v1 Address Translation

The host.v1 service configuration type now includes a forwardAddressTranslations field that specifies
how a hosting tunneler should translate destination IPs from the client when connecting to the underlay
application.

Component Updates and Bug Fixes

• github.com/openziti/edge-api: v0.26.42 -> v0.26.43
• github.com/openziti/ziti: v1.6.0 -> v1.6.1
  • Issue #2996 - Add ability to signal SDKs to extend cert authenticator
  • Issue #2963 - support intercept.v1 --> host.v1 address translation

v1.6.0

Release 1.6.0

What's New

• Bug fixes and library updates
• Change to cluster add peer
• Experimental multi-underlay SDK support
• Experimental SDK flow-control support

Cluster Add Peer Change

The ziti agent cluster add command no longer supports the --id argument
for providing the peer id. The add operation will now always connect to the
peer, verify the certs and get the peer advertise address and id from the
peer directly. This will ensure that the peer is reachable and valid before
it is added to the cluster.

Multi-underlay SDK support

For SDKs which support it, the edge router now supports a separate control channel
connection along side the data connection. If the SDK doesn't request separate
channels, the edge router will continue to work with a single connection. This
feature is still experimental and may have bugs, may be changed or be removed.

SDK Flow-control

For SDKs which support it, the edge router now supports delegating flow control
to the SDK. This feature is still experimental and may have bugs, may be changed
or be removed.

Component Updates and Bug Fixes

• github.com/openziti/channel/v4: v3.0.39 -> v4.0.6

  • Issue #182 - MultiListener can deadlock
  • Issue #180 - Add GetUserData to Channel interface
  • Issue #176 - Multi-channel need a mechanism to notify the txer that the underlay has closed
  • Issue #172 - Support multi-underlay channels

• github.com/openziti/identity: v1.0.100 -> v1.0.101

  • Issue #64 - Support a way to check if a cert/serverCert can be saved

• github.com/openziti/metrics: v1.3.0 -> v1.4.1

  • Issue #53 - Add reporter useful for emitting metrics to stdout

• github.com/openziti/sdk-golang: v0.25.1 -> v1.1.0

  • Issue #702 - [Go SDK] Support xgress flow control from the SDK
  • Issue #722 - Move xgress impl to SDK
  • Issue #717 - ER connection race condition can leak connections
  • Issue #689 - Concurrent map iteration and modification in getEdgeRouterConn causes panic
  • Issue #701 - Support multi-underlay channels for edge router connections

• github.com/openziti/transport/v2: v2.0.167 -> v2.0.168

• github.com/openziti/xweb/v2: v2.3.0 -> v2.3.1

• github.com/openziti/ziti: v1.5.4 -> v1.6.0

  • Issue #3005 - Always check that a controller is reachable and valid before adding it to an HA controller cluster
  • Issue #2986 - [Router] Support xgress flow control from the SDK
  • Issue #2999 - OIDC JWT backed sessions cannot verify extended certs
  • Issue #2997 - Add Authenticator Id to OIDC JWTs/return for current-api-session
  • Issue #2904 - Support client certificate authorities in TLS handshake
  • Issue #2973 - CLI: add a subcommand to retrieve network JWT
  • Issue #2984 - Extend enrollments does not return a full chain
  • Issue #2930 - Support multi-underlay channels for the edge SDK
  • Issue #2978 - Create loop4 sim for testing circuit contention and scale
  • Issue #2981 - Remove PayloadBufferForwarder API from xgress retransmitter
  • Issue #2906 - Controller not removed from DB controller store when removed from controller
  • Issue #2922 - Validate node address before adding to cluster
  • Issue #2932 - Fix router data model 'create public key' related errors
  • Issue #2919 - Make xgress pluggable, so it can be used from the SDK
  • Issue #2955 - Extract xgress inspection types
  • Issue #2954 - Encapsulate xgress metrics
  • Issue #2952 - Remove global payload ingester
  • Issue #2951 - Remove global xgress retransmitter
  • Issue #2950 - Move router specific xgress code to a new xgress_router package
  • Issue #2920 - Make xgress acker configurable

v1.5.4

Release 1.5.4

What's new

• Bug fixes

Component Updates and Bug Fixes

• github.com/openziti/ziti: v1.5.3 -> v1.5.4
  • Issue #2947 - Panic on router started up if edge/tunnel bindings not configured
  • Issue #2948 - Allow ER/T to run without edge listener

v1.5.3

Release 1.5.3

What's New

• This release updates the Go version from 1.23 to 1.24

v1.1.18

Release 1.1.18

What's New

• Backport of exponential terminator cost crediting

Component Updates and Bug Fixes

• github.com/openziti/ziti: v1.1.17 -> v1.1.18
  • Issue #2851 - Change terminator failure cost crediting to be exponential based on time since last failure

v1.5.2

Release 1.5.2

What's New

• This release reverts a change refactoring some flow-control apis, as the change caused a panic

v1.5.1

Release 1.5.1

What's New

• Bug fixes and features

• github.com/openziti/sdk-golang: v0.25.0 -> v0.25.1

  • Issue #699 - SDK UPDB enrollment

• github.com/openziti/ziti: v1.5.0 -> v1.5.1

  • Issue #2931 - help ext-jwt-signer auth by logging incoming jwt audience
  • Issue #2934 - API Session Certs in HA not connect to ERs in all scenarios
  • Issue #2926 - Implement minimal Xgress SDK