Refactor pubsub protocol to use new upstream v2 library #1180

philicious · 2025-08-18T17:27:11Z

The GCP GO SDK introduced a new v2 version of their PubSub library with the v1.50 release of the pubsub pkg. Deprecating the old interface and encouraging to upgrade within a 1yr grace period. See googleapis/google-cloud-go#12218 for reference

For downstream users (like me) being able to update to the new v2 library while also using the cloudevents SDK, its necessary to upgrade the pubsub protocol in the cloudevents SDK or we cannot pass in types to funcs like WithReceiveSettings anymore.

Sooo I took the chance and migrated the code to use pubsub/v2.
Noteworthy changes:

upstream dropped the Exists functions on Topic/Subscription and suggest optimistically using them and handling NotFound errors.
the CreateX funcs got moved to an admin plane
Topic and Subscription names now need to be fully-qualified names, including projectID
Topic and Subscription types got renamed to Publisher/Subscriber

(https://github.com/googleapis/google-cloud-go/blob/main/pubsub/MIGRATING.md)

PS: as a separate commit, I added a fix for the existing ReceiveSettings assertion that didnt actually test for the passed in ReceiveSettings anymore as the NumGoRoutines override was removed after these assertions were initially added

tyrone-anz · 2025-08-20T23:51:53Z

Just here to say we have same issue. We are unable to upgrade to libraries using pubsub/v2 because the cloudevents SDK is not yet supporting it.

protocol/pubsub/v2/go.mod

embano1 · 2025-08-24T07:00:21Z

Topic and Subscription names now need to be fully-qualified names, including projectID

Can you please update the public config parameters/related options to indicate this and reduce developer experience issues?

Topic and Subscription types got renamed to Publisher/Subscriber

Seems those are mainly used internally and not really breaking the public API (despite the upgrade to v2 obviously being considered a breaking change)?

philicious · 2025-08-24T14:18:54Z

Topic and Subscription names now need to be fully-qualified names, including projectID

Can you please update the public config parameters/related options to indicate this and reduce developer experience issues?

imho there is no change in cloudevents public config by that change. It still requires a ProjectID and topic/subscription ID as formerly, which are then passed down to PubSub SDK. But now they get concat before being passed down and formerly PubSub SDK did that internally I assume.

Topic and Subscription types got renamed to Publisher/Subscriber

Seems those are mainly used internally and not really breaking the public API (despite the upgrade to v2 obviously being considered a breaking change)?

Correct: The new Publisher and Subscriber types loosely reference a Topic/Subscription and some funcs got moved from the latter to the new types. However those new handlers are not exposed to the cloudevents public interface

embano1 · 2025-08-30T07:42:54Z

imho there is no change in cloudevents public config by that change. It still requires a ProjectID and topic/subscription ID as formerly

But IMHO that contradicts your comment now need to be fully-qualified names, including projectID which seemed not to be the case before? If these strings now require a different (semantical) format, it should be documented in the option/API and would be considered breaking.

philicious · 2025-08-30T09:13:23Z

But IMHO that contradicts your comment now need to be fully-qualified names, including projectID which seemed not to be the case before? If these strings now require a different (semantical) format, it should be documented in the option/API and would be considered breaking.

I was referring to the changes required for the migration to PubSub v2 library: it now requires them to be fully-qualified but as the cloudevents user supplies both projectID and topicID/subID anyways, we can simply construct the FQDN by the same supplied options as before.
So no breaking code change.

embano1

Just a nit, otherwise LGTM if CI passes and assuming you've verified this change in a real PubSub environment?

embano1 · 2025-08-31T14:39:31Z

protocol/pubsub/v2/internal/connection.go

+		if st, ok := status.FromError(err); ok && st.Code() == codes.NotFound {
+			return false, nil // Topic does not exist.
+		}
+		return false, fmt.Errorf("unable to get topic %q, %v", topicID, err)


nit: use %w for error wrapping?

good idea but since the calling function bubbles-up any caught errors and wraps them again with %v, it would need changes there too. So I'll leave it as is as its likely not used with errors.Is anywhere or would have caused issues before

philicious · 2025-09-01T15:59:20Z

@embano1 thanks for the review and approval 👍
The changes were tested with our (internal) ETL-pipeline library and events are still happily flowing. Topic/Sub creation wasnt explicitly tested but are covered by tests in cloudevents SDK so imho it should be fine to merge.

you might need to kick-off the remaining CI workflows?

embano1 · 2025-09-02T06:10:27Z

Thx, I'll merge once #1183 is merged as we're upgrading tooling and requiring updated workflows to pass.

See googleapis/google-cloud-go#12218 for reference Signed-off-by: Philipp Schuler <philipp.schuler@command-b.de>

Signed-off-by: Philipp Schuler <philipp.schuler@command-b.de>

philicious · 2025-09-02T11:18:07Z

Thx, I'll merge once #1183 is merged as we're upgrading tooling and requiring updated workflows to pass.

I rebased my branch onto latest master after you merged. I also added a little chore commit for replacing deprecated grpc calls in two of the tests. Hope you dont mind

embano1 · 2025-09-02T17:43:12Z

Thx, running CI with auto-merge.

philicious · 2025-09-09T11:52:48Z

@embano1 whats the general release schedule for the cloudevents SDK? (This here kinda blocks our upgrade path)
Best regards!

embano1 · 2025-09-09T19:22:01Z

We can cut a release any time cc/ @duglin

duglin · 2025-09-12T11:54:32Z

I have no objection to creating a new release

philicious · 2025-09-22T13:05:23Z

@embano1 I would like to kindly ask for a new release as there doesnt seem to be any objection :) Thanks in advance for your efforts!

embano1 · 2025-09-22T14:41:26Z

On it. Sorry for my silence, been traveling.

embano1 · 2025-09-22T14:48:34Z

Done, just don't have the time to create release notes ATM: https://github.com/cloudevents/sdk-go/releases/tag/v2.16.2

philicious requested a review from a team as a code owner August 18, 2025 17:27

philicious force-pushed the refactor-pubsubv2 branch 3 times, most recently from b93aca6 to d97e465 Compare August 18, 2025 22:53

philicious changed the title ~~Refactor pubsub protocol to use new upstream v2 interface~~ Refactor pubsub protocol to use new upstream v2 library Aug 18, 2025

embano1 reviewed Aug 24, 2025

View reviewed changes

protocol/pubsub/v2/go.mod Outdated Show resolved Hide resolved

philicious force-pushed the refactor-pubsubv2 branch from 9dd1209 to 20ff54a Compare August 24, 2025 14:08

philicious requested a review from embano1 August 24, 2025 14:19

philicious force-pushed the refactor-pubsubv2 branch from 20ff54a to ba7fb2f Compare August 25, 2025 10:50

embano1 approved these changes Aug 31, 2025

View reviewed changes

embano1 previously approved these changes Sep 2, 2025

View reviewed changes

philicious dismissed embano1’s stale review via c09a7f2 September 2, 2025 11:15

philicious force-pushed the refactor-pubsubv2 branch from ba7fb2f to c09a7f2 Compare September 2, 2025 11:15

philicious added 3 commits September 2, 2025 13:16

Refactor pubsub protocol to use new upstream v2 library

ae89eba

See googleapis/google-cloud-go#12218 for reference Signed-off-by: Philipp Schuler <philipp.schuler@command-b.de>

fix: Properly test that supplied ReceiveSettings are being used

f0bd406

Signed-off-by: Philipp Schuler <philipp.schuler@command-b.de>

chore: Replace deprecated grpc.Dial calls

35623f2

Signed-off-by: Philipp Schuler <philipp.schuler@command-b.de>

philicious force-pushed the refactor-pubsubv2 branch from c09a7f2 to 35623f2 Compare September 2, 2025 11:16

philicious mentioned this pull request Sep 2, 2025

add support for publishSettings #1072

Open

embano1 approved these changes Sep 2, 2025

View reviewed changes

embano1 enabled auto-merge September 2, 2025 17:43

embano1 merged commit 945d930 into cloudevents:main Sep 2, 2025
9 checks passed

philicious deleted the refactor-pubsubv2 branch September 12, 2025 12:37

Refactor pubsub protocol to use new upstream v2 library #1180

Refactor pubsub protocol to use new upstream v2 library #1180

Uh oh!

Conversation

philicious commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tyrone-anz commented Aug 20, 2025

Uh oh!

Uh oh!

embano1 commented Aug 24, 2025

Uh oh!

philicious commented Aug 24, 2025

Uh oh!

embano1 commented Aug 30, 2025

Uh oh!

philicious commented Aug 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

embano1 left a comment

Choose a reason for hiding this comment

Uh oh!

embano1 Aug 31, 2025

Choose a reason for hiding this comment

Uh oh!

philicious Sep 1, 2025

Choose a reason for hiding this comment

Uh oh!

philicious commented Sep 1, 2025

Uh oh!

embano1 commented Sep 2, 2025

Uh oh!

philicious commented Sep 2, 2025

Uh oh!

embano1 commented Sep 2, 2025

Uh oh!

Uh oh!

philicious commented Sep 9, 2025

Uh oh!

embano1 commented Sep 9, 2025

Uh oh!

duglin commented Sep 12, 2025

Uh oh!

philicious commented Sep 22, 2025

Uh oh!

embano1 commented Sep 22, 2025

Uh oh!

embano1 commented Sep 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

philicious commented Aug 18, 2025 •

edited

Loading

philicious commented Aug 30, 2025 •

edited

Loading