New component: google pub sub push receiver

[constanca-m]

The purpose and use-cases of the new component

This component receives notifications from Pub/Sub, and decodes the messages given the encoding.

The component has support for notifications that:

• Directly contain a log placed in Pub/Sub
• Contain a reference to an object placed in Google Cloud Storage, in which case the receiver fetches the object, parses it, and passes it to the new consumer.

Roughly, the architecture is like this:

Difference to the current google pub sub receiver

The main difference between the current googlecloudpubsub receiver and the proposed new receiver lies in how they receive messages from Google Pub/Sub. The existing receiver uses a pull subscription model, where the receiver itself initiates requests to Pub/Sub to fetch new messages. This means it must regularly poll for new data, which can introduce delays between when a log is published and when it is received. In contrast, the proposed receiver is designed for push subscriptions, where Pub/Sub actively pushes messages to the receiver as soon as they are available. This allows for near real-time delivery, as the receiver is immediately notified and can process logs right away. In essence, the pull model is receiver-initiated and may be delayed, while the push model is Pub/Sub-initiated and enables immediate, event-driven log ingestion. You can read more about Pull/Push subscriptions in the official documentation.

Receiver Type	Who Initiates?	Delivery Speed	Use Case Example
Pull (current google pub sub receiver)	Receiver	Polling/Delayed	Periodic log collection
Push (new proposed receiver)	Pub/Sub	Immediate/Real-time	Event-driven log ingest

Example configuration for the component

An example configuration is:

receivers:
  googlepubsubpush:
    endpoint: :8080
    encoding: googlecloudlogentry_encoding

exporters:
  otlphttp:
    endpoint: "https://my-backend:443"

extensions:
  googlecloudlogentry_encoding:

service:
  extensions:
    - googlecloudlogentry_encoding
  pipelines:
    logs:
      receivers: [googlepubsubpush]
      exporters:
        - otlphttp
  telemetry:
    logs:
      level: warn

This means that logs received by googlepubsubpush are parsed using googlecloudlogentry_encoding, and then forward to otlphttp.

Telemetry data types supported

It will start with support for logs.

Later we may want to add metrics.

Code Owner(s)

@constanca-m @axw

Sponsor (optional)

@axw

Additional context

We use this internally at Elastic to parse logs placed in GCS or sent directly to Pub/Sub.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[braydonk]

For public record: In private CNCF discussions I stated my desire for this to somehow be combined with the existing pubsub receiver; while they share no technical similarity in their implementation, I believe the existence of two receivers called googlecloudpubsubsomething is going to invite user confusion. However, after evaluating the config surface of the proposed component, I don't think there's any way to hypothetically reconcile it with the existing PubSub receiver that doesn't outright break and hinder the configuration experience. As a result, I'm okay with it living as a separate component, but I'd ask that it be made as explicit as possible what the difference is between this component and the existing PubSub component (I think both READMEs should contain information about what differentiates it from its counterpart).

In the CNCF Slack @axw stated he'd be willing to sponsor the component. Since I am imminently signing on to more work on the Collector Stability Roadmap, I would appreciate if you would be willing to be the primary sponsor of this component. I am still happy to be involved in a consulting capacity in the implementation reviews and on any GCP-specific behaviours that we'd like to get right (I am working on a couple things to standardize across all googlecloud components, like the authentication codepath for example).

[axw]

I'd ask that it be made as explicit as possible what the difference is between this component and the existing PubSub component (I think both READMEs should contain information about what differentiates it from its counterpart).

💯 definitely agree. As mentioned on Slack, I think we could look into where they can/should be unified down the line. My instinct is that they're better separate, but that's mostly from a technical perspective - fewer dependencies, smaller binary size, that sort of thing. I agree it's not great in terms of discoverability though. If we do want to unify them, then we'll need a concerted effort to break the existing googlecloudpubsub receiver config to make room for push/pull config.