config: move to a structured config file approach

[samlaf]

Right now proxy is configured using urfave flags/env-vars.
This has served us well, and there are good reasons for using this simple approach (see https://www.arp242.net/flags-config.html), but I think the proxy's feature set has outgrown this simple approach and warrants a proper hierarchical structured config file.

People who want to quickly spin up the proxy are referred to our .env.example.holesky file, where the config structure is encoded with string prefixes:

This is very yucky and bug-prone.

We should allow proxy to load up its config from a toml or other structured format.
Then we could have a nice detailed configuration example documentation, like e.g. kurtosis' ethereum-package.
We could then create holesky and mainnet examples of such a config file to help kick-start people with.

[samlaf]

Current internal config (from #340) looks like (actually this is missing some fields which were prob not set in this specific test... AppConfig contains all the fields):

{
  "ServerConfig": {
    "DisperseV2": true,
    "Host": "127.0.0.1",
    "Port": 0
  },
  "ClientConfigV1": {
    "EdaClientCfg": {
      "RPC": "disperser-holesky.eigenda.xyz:443",
      "ResponseTimeout": 10000000000,
      "ConfirmationTimeout": 900000000000,
      "StatusQueryTimeout": 2700000000000,
      "StatusQueryRetryInterval": 1000000000,
      "EthRpcUrl": "",
      "SvcManagerAddr": "0xD4A7E1Bd8015057293f0D0A557088c286942e84b",
      "WaitForConfirmationDepth": 1,
      "WaitForFinalization": false,
      "CustomQuorumIDs": [],
      "SignerPrivateKeyHex": "",
      "DisableTLS": false,
      "PutBlobEncodingVersion": 0,
      "DisablePointVerificationMode": false
    },
    "MaxBlobSizeBytes": 16000000,
    "PutRetries": 3
  },
  "VerifierConfigV1": {
    "VerifyCerts": false,
    "RPCURL": "",
    "SvcManagerAddr": "0xD4A7E1Bd8015057293f0D0A557088c286942e84b",
    "EthConfirmationDepth": 1,
    "WaitForFinalization": false,
    "MaxBlobSizeBytes": 0
  },
  "KzgConfig": {
    "G1Path": "../resources/g1.point",
    "G2Path": "../resources/g2.point",
    "G2TrailingPath": "../resources/g2.trailing.point",
    "G1PowerOf2Path": "",
    "G2PowerOf2Path": "",
    "CacheDir": "../resources/SRSTables",
    "NumWorker": 10,
    "SRSOrder": 268435456,
    "SRSNumberToLoad": 500000,
    "Verbose": false,
    "PreloadEncoder": false,
    "LoadG2Points": true
  },
  "ClientConfigV2": {
    "Enabled": true,
    "DisperserClientCfg": {
      "Hostname": "",
      "Port": "",
      "UseSecureGrpcFlag": false
    },
    "PayloadDisperserCfg": {
      "PayloadPolynomialForm": 0,
      "BlobVersion": 0,
      "DisperseBlobTimeout": 120000000000,
      "BlobCertifiedTimeout": 120000000000,
      "BlobStatusPollInterval": 1000000000,
      "ContractCallTimeout": 5000000000
    },
    "RelayPayloadRetrieverCfg": {
      "PayloadPolynomialForm": 0,
      "BlobVersion": 0,
      "RelayTimeout": 5000000000
    },
    "PutRetries": 1,
    "MaxBlobSizeBytes": 16000000,
    "EigenDACertVerifierAddress": ""
  },
  "MemstoreConfig": {
    "MaxBlobSizeBytes": 16000000,
    "BlobExpiration": "336h0m0s",
    "PutLatency": "0s",
    "GetLatency": "0s",
    "PutReturnsFailoverError": false
  },
  "StorageConfig": {
    "AsyncPutWorkers": 0,
    "FallbackTargets": [],
    "CacheTargets": [],
    "RedisConfig": {
      "Endpoint": "",
      "Password": "",
      "DB": 0,
      "Eviction": 86400000000000
    },
    "S3Config": {
      "CredentialType": "unknown",
      "Endpoint": "",
      "EnableTLS": false,
      "AccessKeyID": "",
      "AccessKeySecret": "",
      "Bucket": "",
      "Path": ""
    }
  }
}

We have an "external config", defined via flags/env-vars, which populates this internal config. And this is the config we print on initialization, so that users can eye-it and make sure they started the binary with correct config.

There are 2 things that we could improve:

1. improve the internal config (more consistent, better naming, better documentation)
   1.1 Do we ever want to expose this internal config to users in case they need maximal configurability? I don't think so, but not 100% sure.
2. improve the mapping from external-config -> internal-config
   2.1 We could create an explicit external-config struct, define the mapping from external->internal config. And then the flags/env-vars would map 1:1 to the external-config. I think this approach would be much easier to maintain and understand. See the optimism-example section at the bottom of this comment.

Why external->internal needed?

One simple example: MaxBlobSizeBytes - This parameter appears in three different places in the internal config:

1. ClientConfigV1.MaxBlobSizeBytes
2. MemstoreConfig.MaxBlobSizeBytes
3. ClientConfigV2.MaxBlobSizeBytes
   If we exposed the external config to users, it would be easy for them to update one and forget to update the others. So in the external config we would have a single fields. This is prob what we currently do with a single --max-blob-size-bytes flag, but I'm not even 100% sure, and it's prob hard to make sure. It shouldn't be.

Ditto for ClientConfigV1.EdaClientCfg.SvcManagerAddr and VerifierConfigV1.SvcManagerAddr. Prob lots of fields duplicated between v1 and v2 clients as well.

External vs Internal

I also think that the way the fields are categorized should be different in internal vs external perspective. High-level fields of the internal struct should be services/components. Whereas for the external config, might make more sense to group based on features.

Internal high-level fields:

ServerConfig
ClientConfigV1
VerifierConfigV1
KzgConfig
ClientConfigV2
MemstoreConfig
StorageConfig
   RedisConfig
   S3Config

Feels like some of this could be improved. Why is Memstore not part of StorageConfig for example? And should the ClientConfigV1 not own the Verifier and KzgConfig (like the v2 one does)?

External high-level fields might instead be based on external dependencies (not sure here...?):

EthConfig (would contain eth_rpc_url, svc_mgr_addr, etc)
DisperserConfig
etc

Really not sure about this proposal, but it feels like the duplicates in the internal-config are precisely coming from external dependencies being needed by multiple internal services?

Optimism example

Just realized optimism does use this external->internal approach for some of their services. For example the BatcherService contains a config.go file which contains the external config, along with flags to populate that config. And then in the Batcher's main function, it constructs this config by parsing the flags, and then passes it to the BatcherService constructor, which does the external->internal mapping.

And in tests, they can just create the external config (so called CLIConfig) and use the same constructor as the prod code.

[litt3]

I generally think this proposal is a major improvement.

2.1 do we want to actually expose the external-config to users? Probably not... But what we could do is to create an explicit external-config struct, define the mapping from external->internal config. And then the flags/env-vars would map 1:1 to the external-config. I think this approach would be much easier to maintain and understand.

When you say we probably shouldn't expose the external config, are you implying that there shouldn't be an option to configure the system with a config file? It seems to me that your proposal would trivially support users sidestepping flags and env vars entirely, in favor of a maintaining a single config file which directly maps to the concept of the "external config".

And should the ClientConfigV1 not own the Verifier and KzgConfig (like the v2 one does)?

As we discussed, for this particular example, KzgConfig is actually shared v1 and v2.

[samlaf]

When you say we probably shouldn't expose the external config, are you implying that there shouldn't be an option to configure the system with a config file? It seems to me that your proposal would trivially support users sidestepping flags and env vars entirely, in favor of a maintaining a single config file which directly maps to the concept of the "external config".

Good catch; not sure what I was thinking here... will update: the external config should definitely be exposed.

[samlaf]

Just realized google's SRE book contains a very similar distinction between internal and external config:

It is also acceptable for the configuration language to be static data. For example, both your infrastructure and interface might use plain JSON. However, avoid tight coupling between the data format you use as the interface and the data format you use internally. For example, you may use a data structure internally that contains the data structure consumed from configuration. The internal data structure might also contain completely implementation-specific data that never needs to be surfaced outside of the system.