weewx.restx - Post data should be bytes

[Joe Williams]

Hey all,     just updating a little plugin that I had running on an old version of Weewx and Python2

It seems some code that I was using to do the rest call has changed between Python2 and 3,  and Im seeing the following stack when the request is called. 

 weewx[14173] ERROR weewx.restx: *** Traceback (most recent call last):
 weewx[14173] ERROR weewx.restx: ***   File "/usr/share/weewx/weewx/restx.py", line 381, in run_loop
 weewx[14173] ERROR weewx.restx: ***     self.process_record(_record, dbmanager)
 weewx[14173] ERROR weewx.restx: ***   File "/usr/share/weewx/user/lametric.py", line 130, in process_record
 weewx[14173] ERROR weewx.restx: ***     self.post_with_retries(req)
 weewx[14173] ERROR weewx.restx: ***   File "/usr/share/weewx/weewx/restx.py", line 475, in post_with_retries
 weewx[14173] ERROR weewx.restx: ***     _response = self.post_request(request, data)
 weewx[14173] ERROR weewx.restx: ***   File "/usr/share/weewx/weewx/restx.py", line 537, in post_request
 weewx[14173] ERROR weewx.restx: ***     _response = urllib.request.urlopen(request, data=data_bytes, timeout=self.timeout)
 weewx[14173] ERROR weewx.restx: ***   File "/usr/lib/python3.9/urllib/request.py", line 214, in urlopen
 weewx[14173] ERROR weewx.restx: ***     return opener.open(url, data, timeout)
 weewx[14173] ERROR weewx.restx: ***   File "/usr/lib/python3.9/urllib/request.py", line 514, in open
 weewx[14173] ERROR weewx.restx: ***     req = meth(req)
 weewx[14173] ERROR weewx.restx: ***   File "/usr/lib/python3.9/urllib/request.py", line 1277, in do_request_
 weewx[14173] ERROR weewx.restx: ***     raise TypeError(msg)
 weewx[14173] ERROR weewx.restx: *** TypeError: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

The code I was using was the following....   

def process_record(self, record, dbm):
        r = self.get_record(record, dbm)
        data = self.get_data(r)
        if self.skip_upload:
            loginf("skipping upload")
            return
        #logdbg('PR ---- using device_key: %s' % self.device_key)
        #logdbg('PR ---- using server_url: %s' % self.server_url)
        req = six.moves.urllib.request.Request(self.server_url, data)
        req.get_method = lambda: 'POST'
        req.add_header("User-Agent", "weewx/%s" % weewx.__version__)

        auth_header = '%s:%s' % ('dev',self.device_key)
       
        #update
        str_to_bytes = auth_header.encode('ascii')
        b64s = base64.b64encode(str_to_bytes)
        req.add_header("Authorization", "Basic %s" % b64s)
        #end update
       
        req.add_header("Content-Type", "application/json")
        self.post_with_retries(req)

It was working with Python2 version of Weewx.     If anyone has some thoughts on this,  or an alternative way to make the rest call,  it would be much appreciated.  

Cheers,

-Joe

[Joe Williams]

Sorry,  I should have posted that I'm running Weewx 4.5.1 with Python 3.9.2

[Joe Williams]

So I think i was able to work around the issue...    What I am doing is passing a JSON response back to a device using the rest API,  so I am setting the header to be json.   Instead of sending back a string in the request.get_data,   I used the six.ensure_binary to convert the string into bytes and returned the bytes instead of the string,  and now I'm not seeing the error anymore.   Now I'm getting a 400: bad request code back,  but I will have to troubleshoot this one further.

result = " { \"model\": { \"frames\": [{\"icon\": \"%s\", \"text\": \"WeeWx Data\"}, { \"icon\":\"2497\", \"text\":\"%s\"}, { \"icon\":\"20768\", \"text\":\"%s\"}, { \"icon\":\"1153\", \"text\":\"%s\"}, { \"icon\":\"72\", \"text\":\"%s\"}, { \"icon\":\"2423\", \"text\":\"%s\"} ],\"sound\": {\"category\":\"notifications\",\"id\":\"%s\"} } }" % (self.icon, values['temp'], values['pressure'], values['wind speed'], values['rain today'], values['humidity'],  self.sound)

        logdbg('Result: %s' % result)
        result_bytes = six.ensure_binary(result)
        logdbg('bResult: %s' % result_bytes)
        return result_bytes

[Vince Skahan]

Check out https://stackoverflow.com/questions/30760728/python-3-urllib-produces-typeerror-post-data-should-be-bytes-or-an-iterable-of