Can no longer run pulumi refresh/pulumi up after upgrading to pulumi-gcp 8.15.0.

[schplat]

Describe what happened

Every run ends with the following error:

` Detected that /home/jnelson/.pulumi/plugins/resource-gcp-v8.15.0/pulumi-resource-gcp exited prematurely.
This is always a bug in the provider. Please report the issue to the provider author as appropriate.

To assist with debugging we have dumped the STDOUT and STDERR streams of the plugin:

panic: fatal: A failure has occurred: Unexpected multiple elements in array with MaxItems=1

goroutine 370 [running]:
github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/go/common/util/contract/failfast.go:23
github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Failf({0x5848913?, 0xc0016dcdc0?}, {0x0?, 0xc0088b6a50?, 0x568b580?})
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/go/common/util/contract/fail.go:32 +0xcb
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.MakeTerraformOutput.func1({0x6212aa0, 0xc0018f3d70}, {0x4aea5e0, 0xc006d1a360}, {0x6211218, 0xc0016dcdc0}, 0x0, 0xc0089902d0, 0x1)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/schema.go:1192 +0xe74
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.MakeTerraformOutput({0x61efd38?, 0xc0088b6a50?}, {0x6212aa0?, 0xc0018f3d70?}, {0x4aea5e0?, 0xc006d1a360?}, {0x6211218, 0xc0016dcdc0}, 0xc00226e410?, 0xc0089902d0, ...)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/schema.go:1231 +0x9b
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.MakeTerraformOutputs({0x61efd38, 0xc0088b6a50}, {0x6212aa0, 0xc0018f3d70}, 0xc00919a148, {0x61f6720, 0xc0010d9470}, 0x0, 0xc0089902d0, 0x1)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/schema.go:1079 +0x1bd
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.MakeTerraformOutput.func1({0x6212aa0, 0xc0018f3d70}, {0x4e51c60, 0xc0024bde60}, {0x62112d0, 0xc008004c00}, 0x0, 0xc0089902d0, 0x1)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/schema.go:1211 +0x9cc
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.MakeTerraformOutput({0x61efd38?, 0xc0088b6a50?}, {0x6212aa0?, 0xc0018f3d70?}, {0x4e51c60?, 0xc0024bde60?}, {0x62112d0, 0xc008004c00}, 0x0?, 0xc0089902d0, ...)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/schema.go:1231 +0x9b
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.MakeTerraformOutput.func1({0x6212aa0, 0xc0018f3d70}, {0x4aea5e0, 0xc006d1a378}, {0x6211218, 0xc0016dd400}, 0x0, 0xc0089902d0, 0x1)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/schema.go:1181 +0xd8c
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.MakeTerraformOutput({0x61efd38?, 0xc0088b6a50?}, {0x6212aa0?, 0xc0018f3d70?}, {0x4aea5e0?, 0xc006d1a378?}, {0x6211218, 0xc0016dd400}, 0x4?, 0xc0089902d0, ...)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/schema.go:1231 +0x9b
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.MakeTerraformOutputs({0x61efd38, 0xc0088b6a50}, {0x6212aa0, 0xc0018f3d70}, 0xc00919ab10, {0x61f6720, 0xc0010d93e0}, 0x0, 0xc0089902d0, 0x1)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/schema.go:1079 +0x1bd
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.MakeTerraformOutput.func1({0x6212aa0, 0xc0018f3d70}, {0x4e51c60, 0xc0024bde30}, {0x62112d0, 0xc008004b40}, 0x0, 0xc0089902d0, 0x1)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/schema.go:1211 +0x9cc
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.MakeTerraformOutput({0x61efd38?, 0xc0088b6a50?}, {0x6212aa0?, 0xc0018f3d70?}, {0x4e51c60?, 0xc0024bde30?}, {0x62112d0, 0xc008004b40}, 0xc007ca5e50?, 0xc0089902d0, ...)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/schema.go:1231 +0x9b
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.MakeTerraformOutput.func1({0x6212aa0, 0xc0018f3d70}, {0x4aea5e0, 0xc006d1a390}, {0x6211218, 0xc0016dd7c0}, 0x0, 0xc0089902d0, 0x1)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/schema.go:1181 +0xd8c
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.MakeTerraformOutput({0x61efd38?, 0xc0088b6a50?}, {0x6212aa0?, 0xc0018f3d70?}, {0x4aea5e0?, 0xc006d1a390?}, {0x6211218, 0xc0016dd7c0}, 0xc00919b1b0?, 0xc0089902d0, ...)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/schema.go:1231 +0x9b
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.MakeTerraformOutputs({0x61efd38, 0xc0088b6a50}, {0x6212aa0, 0xc0018f3d70}, 0xc0024bde00, {0x61f6720, 0xc0010d93b0}, 0xc00229b110, 0xc0089902d0, 0x1)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/schema.go:1079 +0x1bd
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.MakeTerraformResult({0x61efd38?, 0xc0088b6a50?}, {0x6212aa0?, 0xc0018f3d70?}, {0x61f0838, 0xc008aa1040}, {0x61f6720?, 0xc0010d93b0?}, 0xc00229b110?, 0xc0089902d0, ...)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/schema.go:1048 +0x10f
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Update(0xc0009e1988, {0x61efd38?, 0xc0088b6630?}, 0xc003591d90)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/provider.go:1718 +0xffa
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/x/muxer.(*muxer).Update.func1({0x6214bb8?, 0xc0009e1988?})
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/x/muxer/muxer.go:386 +0x33
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/x/muxer.resourceMethod[...](0xc0046b1e00?, 0x54dc940, 0xc0009596b8?)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/x/muxer/muxer.go:333 +0xb5
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/x/muxer.(*muxer).Update(0xb17b1b0?, {0x61efd38?, 0xc0088b6630?}, 0x4bf9e20?)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/x/muxer/muxer.go:385 +0x4a
github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler.func1({0x61efd38?, 0xc0088b6630?}, {0x556dc80?, 0xc003591d90?})
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:927 +0xce
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x61efd38, 0xc008887920}, {0x556dc80, 0xc003591d90}, 0xc008874ce0, 0xc0088595f0)
	/home/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57 +0x3c4
github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler({0x5653ea0, 0xc0046b1e00}, {0x61efd38, 0xc008887920}, 0xc00866ff00, 0xc002b848a0)
	/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:929 +0x143
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000c7e000, {0x61efd38, 0xc007acd680}, {0x6207760, 0xc0018c4680}, 0xc008900ea0, 0xc0023473e0, 0xb1cff00, 0x0)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1394 +0xe2b
google.golang.org/grpc.(*Server).handleStream(0xc000c7e000, {0x6207760, 0xc0018c4680}, 0xc008900ea0)
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1805 +0xe8b
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1029 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 54
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1040 +0x125`

Sample program

I'm not even sure what specific component is causing this, so can't really provide a sample program, and I'm not going to provide my entire code base here.

Log output

No response

Affected Resource(s)

No response

Output of pulumi about

CLI
Version 3.147.0
Go Version go1.23.5
Go Compiler gc

Plugins
KIND NAME VERSION
resource gcp 8.15.0
resource mailgun 3.5.7
language python 3.147.0

Host
OS ubuntu
Version 22.04
Arch x86_64

This project is written in python: executable='/home/jnelson/git//venv/bin/python' version='3.10.12'

Additional context

While not happening on every stack I have, it is happening within a few stacks.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[guineveresaenger]

Hi @schplat, thank you for filing this issue. We understand that this is a frustrating experience.

We see this kind of bug when a resource field has changed type in a certain way.
As our tests have not tripped over this, it would in fact help us help you faster if we had a minimal reproduction of this behavior, so we can see which resources need to be fixed.

[guineveresaenger]

Hello again -

The team has taken a look at this and we cannot see anything immediately obvious that sticks out. This kind of error occurs when upstream changes the behavior of MaxItems = 1 between versions, and the culprit here is likely a single resource.

We don't currently have a way of telling which resource is failing in this way, so in order for us to gain traction here we do need the following:

• The version of pulumi-gcp that you're upgrading from

• Could you run pulumi up with PULUMI_DEBUG_GRPC=pulumi-logs.json and send us the file via gist?

• A minimal repro based on one of the stacks that are failing would be ideal - we don't need your entire code base. Depending on what version you're upgrading from, it's likely just a single resource that's stuck.

We're also working on better tooling around panics as a long-term investment to help with issues like this.

Thank you for your patience!

[schplat]

Upgrading from pulumi-gcp version 7.38.0

I will have to do the pulumi up at some point over the weekend. I managed to roll back to 7.38, and fortunately the state only needed to be modified in one location between the two versions to make it compatible with 7.38 again, I need it to be in a working state for the time being.

The minimal repro will still be difficult, even from my end, as pulumi is managing ~40 resources even in its smallest stack configuration, and I have no idea which resource it's dying on. The panic is attached to the top level Stack URN on the output, so it's not like I can use that to even narrow it down.. Is there a bisection that can be done between 7.38.0 and 8.15.0 that might show which resources added a MaxItems = 1 in order to narrow down what I have to look for?

[t0yv0]

@schplat we've added some instrumentation to narrow down the panic and label it with a specific resource type, name, and lifecycle method. https://github.com/pulumi/pulumi-gcp/tree/refs/tags/v8.17.0-alpha.2 version of the provider is available with this instrumentation, and this will be mainlined with v8.17.0. Unfortunately we were not able to repro as yet. There are too many resources involved in MaxItems=1.