Publish JSON Output to Topic

[Matt T]

Hi,

I'm new to pubsub, and pretty new to Python, so my question may be quite basic!  I'm trying with little success to publish a json load to a pubsub topic.

The publisher method requires a bytestring and I have a dict.

I've tried various combinations of encoding, iterating through the dict string conversion etc..etc.. but I'm not getting very far.

What's best practise when dealing with a JSON payload, is it to parse it and send as individual messages for each line in the JSON script? There's no nesting in the JSON structure and I'm aiming to consume the payload in Data Flow and so some transformations.

Any guidance appreciated.

res = urllib.request.urlopen('https://data.cityofchicago.org/resource/8v9j-bter.json')res_body = res.read()# https://docs.python.org/3/library/json.htmltraffic = json.loads(res_body.decode("utf-8"))publisher = pubsub_v1.PublisherClient()topic_path = publisher.topic_path( project, topic)for key in traffic: print(key.items()) #publisher.publish(topic_path, bytes(traffic))#

Regards,

M.

[Kir Titievsky]

Matt, please start with a google search on how to convert a python dict into a string.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-dis...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/41b11340-c66e-4822-b635-261be8cd0e31%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kir Titievsky | Product Manager | Google Cloud Pub/Sub

[George (Cloud Platform Support)]

The methods dict.keys() and dict.values() return lists of the keys or values explicitly, and you may use them to transform dict data in strings. You have more detail in the "Python Dict and File" online document.  

[Alex Martelli]

The `json` module of the Python standard library is a good way to parse JSON strings into Python data structures, and viceversa to produce JSON bytestrings from Python data structures. For example:

>>> d={'a':23, 'b':[3,5,7]}
>>> import json
>>> json.dumps(d)
'{"a": 23, "b": [3, 5, 7]}'
>>> 

This is with Python 2.7 so there's no `b` prefix in the bytestring representation at the end, but that's the only difference: if you repeat this in Python 3.* it will be just the same except for that leading `b` in the representation of the bytestring result.

Alex

[Matt T]

Thanks for the feedback I have looked at string conversion however the solution was to convert to string and then encode...

for key in traffic: publisher.publish(topic_path, str.encode(str(key),"utf-8"))

This works nicely.

M.

On Tuesday, May 22, 2018 at 11:00:45 PM UTC+8, Kir Titievsky wrote:

Matt, please start with a google search on how to convert a python dict into a string.

On Tue, May 22, 2018 at 10:59 AM Matt T <matt.gm...@gmail.com> wrote:

Hi,

I'm new to pubsub, and pretty new to Python, so my question may be quite basic!  I'm trying with little success to publish a json load to a pubsub topic.

The publisher method requires a bytestring and I have a dict.

I've tried various combinations of encoding, iterating through the dict string conversion etc..etc.. but I'm not getting very far.

What's best practise when dealing with a JSON payload, is it to parse it and send as individual messages for each line in the JSON script? There's no nesting in the JSON structure and I'm aiming to consume the payload in Data Flow and so some transformations.

Any guidance appreciated.

res = urllib.request.urlopen('https://data.cityofchicago.org/resource/8v9j-bter.json')res_body = res.read()# https://docs.python.org/3/library/json.htmltraffic = json.loads(res_body.decode("utf-8"))publisher = pubsub_v1.PublisherClient()topic_path = publisher.topic_path( project, topic)for key in traffic: print(key.items()) #publisher.publish(topic_path, bytes(traffic))#

Regards,

M.

--
You received this message because you are subscribed to the Google Groups "Google Cloud Pub/Sub Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cloud-pubsub-discuss+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cloud-pubsub-discuss/41b11340-c66e-4822-b635-261be8cd0e31%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.