Incorrect padding error

[Teddy-Schmitz]

Hi,
This is in regards to the pubsub module. When I send a test message with data "testing 1 2 3" from the web console to a simple python application it fails to decode the message and throws a

binascii.Error: Incorrect padding

Error from

  File "/python3.5/site-packages/google/cloud/pubsub/message.py", line 88, in from_api_repr
    data = base64.b64decode(api_repr.get('data', b''))
  File "/lib/python3.5/base64.py", line 88, in b64decode

I get the same error if I send the same message using the Go libraries.
I do not get the error if I send the message using python however.
All I'm doing is receiving and printing out the messages.

[dhermes]

When I send a test message with data "testing 1 2 3"

Can you give some code snippets indicating how you do this? Thanks.

[Teddy-Schmitz]

I was using the Google Cloud console to do so by clicking on the publish button. I also did it using the Go library I'll get you the code for that later today.

[Teddy-Schmitz]

func (q *PubSubClient) PublishTransaction(t *db.TrxModel) error {
    ctx := context.Background()
    data, err := json.Marshal(t)
    if err != nil {
        return err
    }

    _, err = q.client.Topic(q.config.TrxTopic).Publish(ctx, &pubsub.Message{
        Data: data,
    })
    return err
}

Here is what the go code looks like, we sent the data "testing 1 2 3" .

I would also add that "testing 1 2 3 4" works fine so I'm guessing its something to do with how other languages pad the base64 encoded bytestring. Maybe you can do something like this,

http://stackoverflow.com/questions/2941995/python-ignore-incorrect-padding-error-when-base64-decoding

To check for incorrect padding and add them in as necessary?

[daspecster]

@Teddy-Schmitz, this is actually the github repository for the Google Cloud Python library.

In your Go example(which I'm not super familiar with Go just FYI), it appears that you may need to cast data to bytes.

See:

• https://godoc.org/cloud.google.com/go/pubsub#hdr-Publishing,
• https://github.com/GoogleCloudPlatform/google-cloud-go/blob/5af4269f950e91e917bab77f1138139023c868c2/pubsub/message.go#L32

The issue's for google-cloud-go are here https://github.com/GoogleCloudPlatform/google-cloud-go/issues.

@omaray, do you know who to contact about this being an issue in the console?

[dhermes]

@daspecster I'd guess this is our problem too. It seems like we ASSUME that base64 values are properly padded but other libraries may strip padding to reduce payload size?

[daspecster]

@dhermes, oh I see. For some reason I dove head first into the Go code. (Go is intriguing).

@Teddy-Schmitz sorry about that, I think I understand the problem better now.

It appears that the console is sending message data that is not base64 encoded at all.

$ python ../../test_pubsub_padding.py
DATA RECIEVED:
testing 1 2 3
Traceback (most recent call last):
  File "../../test_pubsub_padding.py", line 15, in <module>
    pulled = subscription.pull(max_messages=2)
  File "/Users/daspecster/.virtualenvs/google-cloud-pubsub/lib/python2.7/site-packages/google_cloud_pubsub-0.20.0-py2.7.egg/google/cloud/pubsub/subscription.py", line 327, in pull
    for info in response]
  File "/Users/daspecster/.virtualenvs/google-cloud-pubsub/lib/python2.7/site-packages/google_cloud_pubsub-0.20.0-py2.7.egg/google/cloud/pubsub/message.py", line 90, in from_api_repr
    data = base64.b64decode(api_repr.get('data', b''))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/base64.py", line 76, in b64decode
    raise TypeError(msg)
TypeError: Incorrect padding

The docs for GRPC and REST say that...

The message payload. For JSON requests, the value of this field must be base64-encoded.

See: https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#pubsubmessage

So I guess it's 'assumed' that you should base64 encode your message before typing/pasting it in the console?

I have a branch with a fix(if you can call it that) but I think the right solution is to talk to the console.cloud.google.com/cloudpubsub people.

[Teddy-Schmitz]

@daspecster sorry for the confusion, I don't think I explained the problem properly.

I have done some further testing, if I send the same message using the Google Console the Go Library will correctly return me the message. So I'm guessing it is being base64 encoded at the server receiving the POST request. As for the Go library I took a quick peek at their source code and it is base64 encoding the payload before sending it off to pubsub. I believe @dhermes is right and other languages strip off the padding to save space. If the console wasn't base64 encoding the message at all Go would also fail with an error.

At this point its pretty dangerous to pass messages from non-python code to python.

[sturench]

👍 this seems to be an issue related to the Cloud Console - but the Python Library should handle this more gracefully. Looks like the proposed fix by @daspecster would handle it.

[daspecster]

OK great, I'll get that rolling then!

[tseaver]

FWIW, stripping off the padding to "save bandwidth" is utterly bogus for a JSON-based protocol.