Error while running core-eval to execute a null upgrade: The 'governedParams' term must be an object like Electorate …

[Jorge-Lopes]

Description

While running a core eval to execute a null upgrade to the KREAd contract, the chain logs display the error below.
I am failing to pinpoint where and why the error is being triggered, but my debug attempt can be seen in the section Additional Context.

Note that before the error is triggered, we can see the message that vat v54 upgraded from incarnation 0 to 1, but later it logged kernel: kernel terminating vat v54 (failure=true).

Error message:

v54: Error: The 'governedParams' term must be an object like {"Electorate":{"type":"invitation","value":{"brand":"[Alleged: Zoe Invitation brand]","value":[{"description":"questionPoser","handle":"[Alleged: InvitationHandle]","installation":"[Alleged: BundleIDInstallation]","instance":"[Alleged: InstanceHandle]"}]}}}, but was {"Electorate":{"type":"invitation","value":{"brand":"[Alleged: Zoe Invitation brand]","value":[{"description":"questionPoser","handle":"[Alleged: InvitationHandle]","installation":"[Alleged: BundleIDInstallation]","instance":"[Alleged: InstanceHandle]"}]}}}
 at apply ()
 at Error (/bundled-source/.../node_modules/ses/src/error/tame-error-constructor.js:60)
 at makeError (/bundled-source/.../node_modules/ses/src/error/assert.js:352)
 at fail (/bundled-source/.../node_modules/ses/src/error/assert.js:482)
 at (.../governance/src/contractHelper.js:44)
 at ()

The complete log history can be seen below.

To Reproduce

Steps to reproduce the behavior:

1. Checkout to branch 'jorge/10392-kread-upgrade'
2. Open terminal 1:

> cd a3p-integration
> yarn build
> docker buildx bake --load test-acceptance
> docker run -p 26657:26657 -p 9090:9090 -p 1317:1317 -it --rm --name a3p -w /usr/src/a3p-integration/proposals/z:acceptance  --entrypoint bash --mount type=bind,src=.,dst=/usr/src/a3p-integration ghcr.io/agoric/agoric-3-proposals:test-acceptance
> source /usr/src/upgrade-test-scripts/env_setup.sh

3. Open terminal 2:

> cd a3p-integration
> docker exec -w /usr/src/upgrade-test-scripts -it a3p bash start_agd.sh

4. Terminal 1:

>  yarn ava kread.test.js

5. See error being generated on chain logs at terminal 2

Expected behavior

The core-eval should be properly executed and the KREAd vat incarnation should have increased from 0 to 1.

Platform Environment

• OS: MacOs 15.1
• Node: v20.11.0
• agoric-sdk: agoric-upgrade-16av-823-g8943c95a9

Additional context

At the core-eval builder I am fetching the kreadKitP and the kreadCommitteeCreatorFacet, both available at the bootstrap powers.
The kreadKitP hold the privateArgs used to initiate the contract and through the kreadCommitteeCreatorFacet I get a new initialPoserInvitation to include in the privateArgs argument passed to upgradeContract.

The bundleID passed can be confirmed at the KREAd Agoric Mainnet release, more specifically at the start-kread.js. The same bundleID can be seen being used at the agoric-3-proposals 53:kread-start.

The initialPoserInvitation is then used by KREAd contract as an argument for handleParamGovernance

  const { makeDurableGovernorFacet } = handleParamGovernance(
    zcf,
    privateArgs.initialPoserInvitation,
    {},
  );

The error message present on the chain logs point us to /governance/src/contractHelper.js:44 which regards the validateElectorate function of contractHelper. This function is called during the execution of facetHelpers function, which is the return value of handleParamGovernance .

I noticed that the KREAd contract did not provided the storageNode and marshaller to handleParamGovernance, and that may resulted in a faulty paramManager object and consequently an unexpected behavior at validateElectorate due to:
const invitation = paramManager.getInternalParamValue(CONTRACT_ELECTORATE) .

Although I am not certain of my suspicion, due to the fact that this problem was not raised when the contract was first initiated, and the fact that no other issue was triggered during the execution of makeParamManagerFromTerms

Screenshots

2024-11-07T18:51:43.240Z SwingSet: vat: v1: evaluateBundleCap { manifestBundleRef: { bundleID: 'b1-ebdae5ff750e5f5a1edac4d020eb35c02dc9c7321c3a6fde9362f4d935003a332a5d8250e73ea4074bdc501c4841374e564eb4624dbf71ee89ff66362357f34e' }, manifestGetterName: 'getManifestForUpgradeKread', vatAdminSvc: Promise [Promise] {} }
2024-11-07T18:51:43.319Z SwingSet: vat: v1: execute { manifestGetterName: 'getManifestForUpgradeKread', bundleExports: [ 'getManifestForUpgradeKread', 'upgradeKread' ] }
2024-11-07T18:51:43.319Z SwingSet: vat: v1: coreProposal: upgradeKread
2024-11-07T18:51:43.320Z SwingSet: vat: v1: LOG: upgrading Kread
2024-11-07T18:51:43.465Z SwingSet: vat: v1: LOG: KREAd adminFacet Object [Alleged: adminFacet] {}
2024-11-07T18:51:43.479Z SwingSet: vat: v1: LOG: KREAd privateArgs { powers: { storageNode: Object [Alleged: ChainStorageNode] {}, marshaller: Object [Alleged: Board readonlyMarshaller] {} }, clock: Object [Alleged: timerClock] {}, seed: 303 }
2024-11-07T18:51:43.479Z SwingSet: vat: v1: LOG: KREAd initialPoserInvitation Object [Alleged: Zoe Invitation payment] {}
2024-11-07T18:51:43.498Z SwingSet: kernel: attempting to upgrade vat v54 from incarnation 0 to source b1-5ce9bb36ceb21c4af80b3c11098ac049ddfaa1898cf7a9e33d100c44f5fc68e5a14a96a5d2f9af0117929b3e5b4ab58c4a404416fb5688b03a2c0e398194e6b2
2024-11-07T18:51:44.963Z SwingSet: vat: v54: provideRecorderKits { infoKit: 'info', characterKit: 'character', itemKit: 'item', marketCharacterKit: 'market-characters', marketItemKit: 'market-items', marketCharacterMetricsKit: 'market-metrics-character', marketItemMetricsKit: 'market-metrics-item' } { characterKit: { payload: [ { name: { payload: [Array] }, character: { payload: [Array] }, inventoryKit: { payload: [Array] }, history: { payload: [Array] } } ] }, itemKit: { payload: [ { id: { payload: 0 }, item: { payload: [Array] }, history: { payload: [Array] } } ] }, marketCharacterKit: { payload: [ { payload: [ [Object], [Object] ] } ] }, marketItemKit: { payload: [ { payload: [ [Object], [Object] ] } ] }, marketCharacterMetricsKit: { payload: [ { amountSold: { payload: 0 }, collectionSize: { payload: 0 }, averageLevel: { payload: 0 }, marketplaceAverageLevel: { payload: 0 }, latestSalePrice: { payload: 0 }, putForSaleCount: { payload: 0 } } ] }, marketItemMetricsKit: { payload: [ { amountSold: { payload: 0 }, collectionSize: { payload: 0 }, averageLevel: { payload: 0 }, marketplaceAverageLevel: { payload: 0 }, latestSalePrice: { payload: 0 }, putForSaleCount: { payload: 0 } } ] } }
2024-11-07T18:51:45.195Z SwingSet: kernel: vat v54 upgraded from incarnation 0 to 1 with source b1-5ce9bb36ceb21c4af80b3c11098ac049ddfaa1898cf7a9e33d100c44f5fc68e5a14a96a5d2f9af0117929b3e5b4ab58c4a404416fb5688b03a2c0e398194e6b2
2024-11-07T18:51:45.482Z SwingSet: ls: v54: Logging sent error stack (Error#1)
2024-11-07T18:51:45.482Z SwingSet: ls: v54: Error#1: The 'governedParams' term must be an object like { Electorate: { type: 'invitation', value: { brand: Object [Alleged: Zoe Invitation brand] {}, value: [ { description: 'questionPoser', handle: [Object], installation: [Object], instance: [Object] } ] } } } , but was { Electorate: { type: 'invitation', value: { brand: Object [Alleged: Zoe Invitation brand] {}, value: [ { description: 'questionPoser', handle: [Object], installation: [Object], instance: [Object] } ] } } }
2024-11-07T18:51:45.482Z SwingSet: ls: v54: Error: The 'governedParams' term must be an object like {"Electorate":{"type":"invitation","value":{"brand":"[Alleged: Zoe Invitation brand]","value":[{"description":"questionPoser","handle":"[Alleged: InvitationHandle]","installation":"[Alleged: BundleIDInstallation]","instance":"[Alleged: InstanceHandle]"}]}}}, but was {"Electorate":{"type":"invitation","value":{"brand":"[Alleged: Zoe Invitation brand]","value":[{"description":"questionPoser","handle":"[Alleged: InvitationHandle]","installation":"[Alleged: BundleIDInstallation]","instance":"[Alleged: InstanceHandle]"}]}}}
 at apply ()
 at Error (/bundled-source/.../node_modules/ses/src/error/tame-error-constructor.js:60)
 at makeError (/bundled-source/.../node_modules/ses/src/error/assert.js:352)
 at fail (/bundled-source/.../node_modules/ses/src/error/assert.js:482)
 at (.../governance/src/contractHelper.js:44)
 at ()

2024-11-07T18:51:45.483Z SwingSet: ls: v54: Error#1 ERROR_NOTE: Sent as error:liveSlots:v54#70001
2024-11-07T18:51:45.483Z SwingSet: xsnap: v54: Error#1 ERROR_NOTE: Sent as error:liveSlots:v54#70002
2024-11-07T18:51:45.483Z SwingSet: ls: v54: Logging sent error stack (Error#1)
2024-11-07T18:51:45.485Z SwingSet: kernel: kernel terminating vat v54 (failure=true)
2024-11-07T18:51:46.386Z SwingSet: vat: v1: LOG: KREAd contract upgraded!
2024-11-07T18:51:46.388Z SwingSet: ls: v2: Logging sent error stack (RemoteError#1)
2024-11-07T18:51:46.388Z SwingSet: ls: v2: RemoteError#1: The 'governedParams' term must be an object like {"Electorate":{"type":"invitation","value":{"brand":"[Alleged: Zoe Invitation brand]","value":[{"description":"questionPoser","handle":"[Alleged: InvitationHandle]","installation":"[Alleged: BundleIDInstallation]","instance":"[Alleged: InstanceHandle]"}]}}}, but was {"Electorate":{"type":"invitation","value":{"brand":"[Alleged: Zoe Invitation brand]","value":[{"description":"questionPoser","handle":"[Alleged: InvitationHandle]","installation":"[Alleged: BundleIDInstallation]","instance":"[Alleged: InstanceHandle]"}]}}}
2024-11-07T18:51:46.388Z SwingSet: ls: v2: Error: The 'governedParams' term must be an object like {"Electorate":{"type":"invitation","value":{"brand":"[Alleged: Zoe Invitation brand]","value":[{"description":"questionPoser","handle":"[Alleged: InvitationHandle]","installation":"[Alleged: BundleIDInstallation]","instance":"[Alleged: InstanceHandle]"}]}}}, but was {"Electorate":{"type":"invitation","value":{"brand":"[Alleged: Zoe Invitation brand]","value":[{"description":"questionPoser","handle":"[Alleged: InvitationHandle]","installation":"[Alleged: BundleIDInstallation]","instance":"[Alleged: InstanceHandle]"}]}}}
 at construct ()
 at Error (/bundled-source/.../node_modules/ses/src/error/tame-error-constructor.js:56)
 at makeError (/bundled-source/.../node_modules/ses/src/error/assert.js:243)
 at decodeErrorCommon (/bundled-source/.../node_modules/@endo/marshal/src/marshal.js:281)
 at decodeFromSmallcaps (/bundled-source/.../node_modules/@endo/marshal/src/encodeToSmallcaps.js:434)
 at map ()
 at map ()
 at fromCapData (/bundled-source/.../node_modules/@endo/marshal/src/marshal.js:356)
 at deliver (/bundled-source/.../packages/swingset-liveslots/src/liveslots.js:1084)
 at dispatchToUserspace (/bundled-source/.../packages/swingset-liveslots/src/liveslots.js:1509)
 at runWithoutMetering (/bundled-source/.../packages/swingset-xsnap-supervisor/lib/supervisor-subprocess-xsnap.js:60)
 at ()

[Chris-Hibbert]

The error term must be an object like "Electorate" is usually due to not providing a fresh invitation. I improved that error message when upgrading vaults.

At the core-eval builder I am fetching the kreadKitP and the kreadCommitteeCreatorFacet, both available at the bootstrap powers.
The kreadKitP hold the privateArgs used to initiate the contract and through the kreadCommitteeCreatorFacet I get a new initialPoserInvitation to include in the privateArgs argument passed to upgradeContract.

The original privateArgs from bootstrapPowers contains an initialPoserInvitation that has been consumed, so you need a fresh one, which can be built with const poserInvitationP = E(electorateCreatorFacet).getPoserInvitation();.

[Jorge-Lopes]

Sorry if I am misunderstanding your suggestion, but I think that I am already doing it at the core-eval builder

[Chris-Hibbert]

I think that I am already doing it at the core-eval builder

You are correct. I was mis-reading it.

Hmm. This contract doesn't use paramGovernance. (It does have API governance, of a sort.) But there's nothing here that would respond to paramManager.getParams(). I also don't see why anyone would try to run vaildateElectorate() for this contract.

In fact, nothing here imports contractHelper.js. Let's figure out what code is running on chain. vat54 does seem to be KREAd. It does have provideRecorderKits, but rc1 has its only console.log commented out, so something changed. Maybe they added a paramManager.

As I recall from upgrading vaultFactory, after upgrade, care has to be taken with the new invitation in order to create a new paramManager. We need to find the code that creates the paramManager, and see if the new invitation from privateArgs is being passed through.