Push isn't working with pubsub emulator

[Eyad Tomeh]

Hello!

I'm trying to test locally withe the emulator which works fine for create topics/subscriptions, publishing and receiving. But I get an error when it tries to push a message to an endpoint

[pubsub] Exception in thread "pool-1-thread-14" java.lang.NoClassDefFoundError: javax/xml/bind/DatatypeConverter

[pubsub]        at com.google.cloud.pubsub.testing.v1.HttpEndpointPusher.toJsonStringForPush(HttpEndpointPusher.java:100)

[pubsub]        at com.google.cloud.pubsub.testing.v1.HttpEndpointPusher.push(HttpEndpointPusher.java:53)

[pubsub]        at com.google.cloud.pubsub.testing.v1.PushLoop.push(PushLoop.java:135)

[pubsub]        at com.google.cloud.pubsub.testing.v1.PushLoop.dispatchMessages(PushLoop.java:116)

[pubsub]        at com.google.cloud.pubsub.testing.v1.PushLoop.run(PushLoop.java:73)

[pubsub]        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)

[pubsub]        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

[pubsub]        at java.base/java.lang.Thread.run(Thread.java:844)

[pubsub] Exception in thread "pool-1-thread-13" java.lang.NoClassDefFoundError: javax/xml/bind/DatatypeConverter

[pubsub]        at com.google.cloud.pubsub.testing.v1.HttpEndpointPusher.toJsonStringForPush(HttpEndpointPusher.java:100)

[pubsub]        at com.google.cloud.pubsub.testing.v1.HttpEndpointPusher.push(HttpEndpointPusher.java:53)

[pubsub]        at com.google.cloud.pubsub.testing.v1.PushLoop.push(PushLoop.java:135)

[pubsub]        at com.google.cloud.pubsub.testing.v1.PushLoop.dispatchMessages(PushLoop.java:116)

[pubsub]        at com.google.cloud.pubsub.testing.v1.PushLoop.run(PushLoop.java:73)

[pubsub]        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)

[pubsub]        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

[pubsub]        at java.base/java.lang.Thread.run(Thread.java:844)

Is there something I'm doing wrong or is it just that the emulator doesn't work with push subscriptions?

[Steeve]

Hi Eyad,

I was able to find this public article [1] which explains how to use the emulator to publish a message locally. However, nothing is mentioned regarding pushing a message to an endpoint locally which makes me believe it is mostly likely currently not supported. I would recommend you file a Feature Request using the Issue Tracker link [2] below towards the Pub/Sub Systems Engineer.

[1]: https://cloud.google.com/pubsub/docs/emulator#using_the_emulator

[2]: https://issuetracker.google.com/savedsearches/559741

[Paul Harwood]

I did it. I got the same error. To make it work I had to change this file

cloud-pubsub-emulator

which was in 

..../google-cloud-sdk/platform/pubsub-emulator/bin

to include the line

DEFAULT_JVM_OPTS="--add-modules java.xml.bind"

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/bebe6ac8-3d87-4de5-bd81-b4f431c61c9e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eyad Tomeh]

@Paul

This works!! Thanks a lot!

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

[Tianzi Cai]

Hey, question about the endpoint - where is the push endpoint? In the emulator? Or outside of it?

[Eyad Tomeh]

What do you mean by "In the emulator"?

[Paul Harwood]

The push endpoint is any arbitrary URL and for emulation is usually something like http://localhost:8060/_ah/push-handlers/'name'?token='token' and - as you say - definitely not a part of the emulator.

The emulator should be able to call any reachable URL as specified in the pushConfig of the subscription.

@Paul

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/bebe6ac8-3d87-4de5-bd81-b4f431c61c9e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/64ae3f99-d289-418a-b80c-e69ab371e0c4%40googlegroups.com.

[Tianzi Cai]

Hmm, is it a fake push endpoint starting with "localhost:"? Or a real push endpoint you have somewhere? 

[Paul Harwood]

That is a real push endpoint running on my dev machine. I am not sure what a fake endpoint would look like, It could be on another machine - but in that case why not use a dev environment in the cloud. For me the whole point of emulator was to have everything working for real on one machine so that I can develop off net. I do not have to change any code when I distro to live.

@Paul

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/bebe6ac8-3d87-4de5-bd81-b4f431c61c9e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/0e18b4a8-bcce-48ab-9148-b30473a6c63e%40googlegroups.com.

[Tianzi Cai]

Thanks Paul. FWIW, I tested push in the emulator not using Java but Python. 

My emulator was started on `http://localhost:8085/`

My push endpoint was set to `http://localhost:8060/august`

Push was successful without me having to modify in `/google-cloud-sdk/platform/pubsub-emulator/bin/cloud-pubsub-emulator`.  I would be interested in knowing why Java is different. 

@Paul

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/bebe6ac8-3d87-4de5-bd81-b4f431c61c9e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

[Andrey Bulezyuk]

Hey, i have similar setup with NodeJS on local machine. Push does not work for ~localhost:8080. I'll try add handler for endpoints like "_ah/*" in my code. Maybe pubsub emulator routes to "endpoint"+"_ah"? Had to rewrite my code to use Pull Subscription..which is sad, Push-Sub has a feature of retrying unack. messages...

[Ali T (Cloud Platform Support)]

Although you can not receive push messages locally, you can simulate a push message through an HTTP request. The following documentation[1] gives a detailed explanation on how to work around this limitation.

[1] https://cloud.google.com/appengine/docs/flexible/nodejs/writing-and-responding-to-pub-sub-messages#simulate_push_notifications

[Paul Harwood]

Hmmm - I know that you can receive push messages locally from a local pub/sub emulator since I do it all of the time. I think that refers to the fact that you cannot receive push messages locally from the cloud implementation of pub/sub.

@Paul

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/bebe6ac8-3d87-4de5-bd81-b4f431c61c9e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/0e18b4a8-bcce-48ab-9148-b30473a6c63e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/39cf0227-bcd0-47d2-8780-b6853ef1448e%40googlegroups.com.

[Edouard Mehlman]

Would you know how to do it with java 11 ? 

--add-modules  no longer exists and I can't figure add how to use maven to add the corresponding java dependencies (https://stackoverflow.com/questions/48204141/replacements-for-deprecated-jpms-modules-with-java-ee-apis)

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

[Jinjun (Cloud Platform Support)]

You can open a Feature Request at https://cloud.google.com/support/docs/issue-trackers#feature_requests for the Pub/Sub team to add code samples for Java in the emulator documentation [1].

[1] https://cloud.google.com/pubsub/docs/emulator#using_the_emulator

[Eyad Tomeh]

I'm having the same problem with Java 11.
I'm still using Java 10 just to avoid this issue...

[Ignacio Tartavull]

How did you use python?

[sope...@gmail.com]

Buckets to journal logging API service?

[Tianzi Cai]

As of today, I can no longer use the Pub/Sub emulator to push to any ports on the localhost. Unless a solution is found, we shall tell people to stop using the emulator this way.