Infrastructure Provider Development

[sudomateo]

Hello all! I'm developing an Oxide infrastructure provider at https://github.com/oxidecomputer/omni-infra-provider-oxide and I started this discussion to discuss my experience and ask questions. I'll place my issues and questions as separate comments to keep things a bit organized.

I have a draft infrastructure provider in development at oxidecomputer/omni-infra-provider-oxide#2. It's meant to be a dynamic infrastructure provider rather than static. I was able to successfully start the infrastructure provider and connect it to Omni. I was not able to, however, create a Kubernetes cluster that successfully configures the machines that were created.

[sudomateo]

Troubleshooting the Infrastructure Provider

I created a cluster in Omni using the oxide machine class and I saw the logs from the infrastructure provider and the instances were successfully created in Oxide. However, the instances never joined Omni, presumably because the instances were booted with a base Talos Linux image (schematic ID 743d53d3c9cc1942e0a3fc7167565665ea25823e6261d82bf022e9a9e50ed84d) and weren't told how to join Omni.

I'm not sure if it's the responsibility of the infrastructure provider to configure the Talos Linux instance to join Omni at boot time, via some pre-generated Talos Linux schematic with the right Omni arguments, or something else. Any guidance here?

[sudomateo]

Now that the user-data is sorted out I want to circle back to the machine "<ID>" has no platform set error that I'm seeing with the nocloud images since it's preventing me from using nocloud images with Omin. I'm unsure if this is a Talos or Omni issue so I'll explain what I'm seeing and we can go from there.

I manually created two machines using the nocloud image, one configured to join Omni via NoCloud cloud-init configuration and the other configured to join Omni via kernel arguments. This was irrespective of the infrastructure provider, so we can rule that out. Both machines successfully joined Omni and are listed under the Self-Managed section. Note that there's no platform label set which I believe to be a symptom of the problem.

I then created two one-node Kubernetes clusters, one using the talos-nocloud-userdata machine and the other using the talos-nocloud-kernel-args machine. The clusters show as Not Ready and their machines show as Configuring indefinitely.

When I click on the machines I see the machine "<ID>" has no platform set error and they never become configured to create the Kubernetes cluster.

I attached the console logs from each of those Talos machines below. I was going to log this as a Talos issue, but I figured it would be good to discuss this further to see if we can determine whether this is a Talos or Omni issue, or both.

• talos-nocloud-userdata.log
• talos-nocloud-kernel-args.log

[smira]

I wonder if this is yet another nocloud bug that it fails to work without network config provided.

Can you try to do talosctl -n UUID get platformmetadata -o yaml while the machine is connected to Omni (maintenance mode is fine)?

(Also if there is a way for me to test on Oxide with Talos images as close as possible to your flow, I'm happy to do it)

I expect that it comes back empty for you, which might be the root cause here.

[sudomateo]

Empty for both.

> talosctl --talosconfig ~/Downloads/talosconfig.yaml --nodes 2df12d04-d7e3-4dc9-95b6-d48b427bd0e0 get platformmetadata -o yaml
> talosctl --talosconfig ~/Downloads/talosconfig.yaml --nodes 2df12d04-d7e3-4dc9-95b6-d48b427bd0e0 get hostname
NODE                                   NAMESPACE   TYPE             ID         VERSION   HOSTNAME                    DOMAINNAME
2df12d04-d7e3-4dc9-95b6-d48b427bd0e0   network     HostnameStatus   hostname   1         talos-nocloud-kernel-args


> talosctl --talosconfig ~/Downloads/talosconfig.yaml --nodes 662eb983-8c92-4932-9843-13ff1133690a get platformmetadata -o yaml
> talosctl --talosconfig ~/Downloads/talosconfig.yaml --nodes 662eb983-8c92-4932-9843-13ff1133690a get hostname
NODE                                   NAMESPACE   TYPE             ID         VERSION   HOSTNAME                 DOMAINNAME
662eb983-8c92-4932-9843-13ff1133690a   network     HostnameStatus   hostname   1         talos-nocloud-userdata

(Also if there is a way for me to test on Oxide with Talos images as close as possible to your flow, I'm happy to do it)

Yep! I can get you access. Send me an email at [email protected] and I'll get you credentials.

[smira]

siderolabs/talos#11959 this should fix this, and we can backport it to 1.11.x, I will be waiting for a way to verify the fix.

[sudomateo]

I built a Talos image using those changes and it successfully joined Omni via user-data and did not encounter the machine "<ID>" has no platform set error. It also successfully installed the Kubernetes components and I could access the cluster using kubectl.

One thing I'll note is that Talos logged upgrade failed: locked and then, ~2 minutes later continued and rebooted into the upgraded Talos version. Here are the logs for that.

• talos.log

I also gave you Oxide credentials that you can use to test Talos directly in Oxide. Let me know if you have any questions.

[sudomateo]

Missing Processor Information

One thing that's bothering me is Oxide instances that are connected to Omni show has having no processor detected.

Processors are seemingly loaded from pollProcessors.

omni/internal/backend/runtime/omni/controllers/omni/internal/task/machine/poll.go

Lines 217 to 240 in 3f1b96d

	func pollProcessors(ctx context.Context, c *client.Client, info *Info) error {
	info.Processors = nil
	return forEachResource(
	ctx,
	c,
	hardware.NamespaceName,
	hardware.ProcessorType,
	func(r *hardware.Processor) error {
	if r.TypedSpec().CoreCount == 0 || r.TypedSpec().MaxSpeed == 0 {
	return nil
	}
	info.Processors = append(info.Processors, &specs.MachineStatusSpec_HardwareStatus_Processor{
	CoreCount: r.TypedSpec().CoreCount,
	ThreadCount: r.TypedSpec().ThreadCount,
	Frequency: r.TypedSpec().MaxSpeed,
	Manufacturer: r.TypedSpec().Manufacturer,
	Description: fmt.Sprintf("%s %s", r.TypedSpec().Manufacturer, r.TypedSpec().ProductName),
	})
	return nil
	})
	}

Which uses SMBIOS retrieval under the hood I believe.

I might not spend much more time on this but the metal vs nocloud instances appear to report back CPU information.

metal:

~> talosctl --talosconfig ~/Downloads/talos-default-talosconfig.yaml --nodes 172.30.0.7 get cpu
NODE         NAMESPACE   TYPE        ID   VERSION   MANUFACTURER                   MODEL                              CORES   THREADS
172.30.0.7   hardware    Processor        1         Advanced Micro Devices, Inc.   AMD EPYC 7713P 64-Core Processor   4       4

~> talosctl --talosconfig ~/Downloads/talos-default-talosconfig.yaml --nodes 172.30.0.7 get version
NODE         NAMESPACE   TYPE      ID        VERSION   VERSION
172.30.0.7   runtime     Version   version   1         v1.11.1

nocloud:

~> talosctl --talosconfig ./talosconfig --endpoints 172.21.252.175 --nodes 172.21.252.175 get cpu
NODE             NAMESPACE   TYPE        ID   VERSION   MANUFACTURER                   MODEL                              CORES   THREADS
172.21.252.175   hardware    Processor        1         Advanced Micro Devices, Inc.   AMD EPYC 7713P 64-Core Processor   4       4

~> talosctl --talosconfig ./talosconfig --endpoints 172.21.252.175 --nodes 172.21.252.175 get version
NODE             NAMESPACE   TYPE      ID        VERSION   VERSION
172.21.252.175   runtime     Version   version   1         v1.11.1

[Unix4ever]

Omni should be using the same API. Not sure why it doesn't work. Might need to check the Omni logs.

[sudomateo]

Is checking the Omni logs something I can do on SaaS Omni? The only logs I see in the UI are machine logs which I already have from the machine's console itself.

[rothgar]

No, the SaaS instance logs are something we can check. Let me see what we can get.

[smira]

so I got some testing going with Oxide - and the SMBIOS data doesn't contain CPU frequency, which is correctly handled by Talos, but Omni filters out processors without CPU speed which I think is a bug (#1688).

[sudomateo]

so I got some testing going with Oxide - and the SMBIOS data doesn't contain CPU frequency, which is correctly handled by Talos, but Omni filters out processors without CPU speed which I think is a bug (#1688).

Thank you for that! I opened oxidecomputer/propolis#962 to add such information to SMBIOS.

[sudomateo]

What is resource.Resource meant to represent?

I understand the goal of an infrastructure provider is to be a binary that does the following.

1. Accept OMNI_ENDPOINT and OMNI_SERVICE_ACCOUNT_KEY configuration.
2. Implement the provision.Provisioner interface.
3. Create a new infrastructure provider using infra.NewProvider
4. Call infra.Provider.Run.

I see that the provision.Provisioner interface is generic over a resource.Resource type which is itself an interface. Here are their definitions.

type Provisioner[T resource.Resource] interface {
	ProvisionSteps() []Step[T]
	Deprovision(context.Context, *zap.Logger, T, *infra.MachineRequest) error
}

type Resource interface {
	Metadata() *Metadata
	Spec() any
	DeepCopy() Resource
}

Looking at other infrastructure providers I see most implement the resource.Resource interface using the following pattern.

type Instance = typed.Resource[InstanceSpec, InstanceExtension]

type InstanceSpec = protobuf.ResourceSpec[oxideResource.Instance, *oxideResource.Instance]

type InstanceExtension struct{}

func (InstanceExtension) ResourceDefinition() meta.ResourceDefinitionSpec {
        return meta.ResourceDefinitionSpec{
                Type:             infra.ResourceType("Instance", ID),
                Aliases:          []resource.Type{},
                DefaultNamespace: infra.ResourceNamespace(ID),
                PrintColumns:     []meta.PrintColumn{},
        }
}

In this, oxideResource.Instance is the compiled Go code from the following protobuf specification.

syntax = "proto3";
package oxide.resource;

option go_package = "github.com/oxidecomputer/omni-infra-provider-oxide/internal/resource";

message Instance {
  string uuid = 1;
  string id = 2;
}

What exactly is this resource.Resource meant to represent? Is it a singular Oxide instance running Talos Linux? Is it just some generic "machine"? I see other infrastructure providers call this type Machine and have some fields in its protobuf specification but I'm not sure what fields are required and optional. For example, I see other infrastructure providers use a field uuid, presumably for idempotency, but I'm not sure what the significance of that field is for an infrastructure provider. I could use some description of the data flow and schemas here if those exist.

[Unix4ever]

It's the state of the machine needed for the specific provider. If you need to save anything for the machine you create in Oxide you can add the fields to this resource. Totally optional though.

[Unix4ever]

For example for KubeVirt we save VM volume UUID which is a hashed value of it's installer path.

[sudomateo]

This is much more clear now after our discussions and me using an infrastructure provider more. Thank you!

[sudomateo]

What is provision.Context used for?

Each step in the ProvisionSteps method takes a provision.Context. I see that I can read/write the state from my resource.Resource implementation but I'm not sure what state, if any, I'm supposed to store. What does Omni expect from an infrastructure provider here?

[sudomateo]

Thank you! Please let me know if I can assist you.

[Unix4ever]

I've fixed the deprovision flow. You can update the Omni client again.
I'm still not sure why provision is called twice, will continue looking.

[Unix4ever]

Ok, the issue with the duplicate calls was also fixed. Please update the library to the latest main.

[sudomateo]

Thank you so much! Traveling today but I'll get it updated and tested in the airport.

[sudomateo]

Works well! Updated the client in oxidecomputer/omni-infra-provider-oxide#5.

[sudomateo]

How does one create multiple instances of an infrastructure provider with unique provider IDs?

I read the Infrastructure Providers documentation where I saw the following image showcasing the deployment of multiple instances of the bare-metal infrastructure provider across multiple regions.

I would like to do this with the Oxide infrastructure provider. The motivation is that users with multiple Oxide racks may have them in different locations where a single infrastructure provider may not be able to reach all of the API endpoints on all of the racks. That, and even if they were able to reach multiple racks, each silo within a rack requires different API credentials which I believe are best set when running the infrastructure provider, not via machine classes.

I see that the bare-metal provider accomplishes this using a mutable ProviderID. That's the approach I settled on myself after a bit of research and AI prompting. It works but I just wanted to ask the following questions.

1. Is this the proper approach to doing this? Should I instead be doing something else?
2. I see that each concrete type that implements resource.Resource has a ResourceDefinition method that contains a namespace. Is this namespace something that I as a provider developer need to care about?

[rothgar]

The idea is one infrastructure provider per "zone" or at least in an environment that has network access to the APIs. We don't want to try and do fancy tunneling or bridging between zones and customers can define those however they'd like. It does add some complexity to managing providers but we hope to solve that in the future with easier update mechanisms.

I don't believe the namespace is intended to be used for the resources and I honestly don't know why it exists.

[utkuozdemir]

It is correct that you need to use a dedicated ID per zone, one instance one ID basically. It is done via that mutable ProviderID, yes.

I can give you some info on the namespace thing: there are two namespaces which interest an infra provider:

1. infra-provider: Contains the resources shared between the Omni and the provider, and a provider can only access the resources in that namespace if they have the label omni.sidero.dev/infra-provider-id matching theirs.
2. infra-provider:<provider-id>: This is the namespace dedicated for a single provider to do whatever they want - they can define their resources as they want/need. It is basically a persistent storage for the providers provided by Omni, so they won't need their own persistence.

The namespace you see here calls here and eventually evaluates to infra-provider:bare-metal basically.

[sudomateo]

Thank you for that information! That's much clearer now and lets me know that I don't need to mess with the namespace in the infrastructure provider, only ensure each instance has its own provider ID.

[sudomateo]

What's the concurrency model of an infrastructure provider?

Is it expected that an infrastructure provider support concurrent calls to Provision and Deprovision (e.g., for different machines)? I ask because right now I'm downloading the Talos image and uploading it to Oxide during Provision and it's not concurrency safe. That is, if I created two clusters at the same time with the same Talos version and schematic I'll have a race condition of uploading the image and creating an image of the same name. I'd like to better understand the concurrency model here so I can update the code to account for situations like this.

I'd like to reiterate that I really appreciate everyone's time and effort thus far on this discussion and the linked issues and pull requests. If this is not the proper mechanism for these questions please feel free to let me know what I can do better.

[Unix4ever]

Yeah, it can provision many machines in parallel. You can probably use singleflight module to handle image uploads.

[sudomateo]

Perfect! Thank you for letting me know.