Support marshalling inferred updates of NULL values#71
Support marshalling inferred updates of NULL values#71Jawshua wants to merge 8 commits intoNextdoor:masterfrom
Conversation
|
Hi @Jawshua, thank you for contributing. The change sounds reasonable. Could you also please add an integration test since this is fairly important change. Also, I'm wondering if the user should be able to specify how they want to differentiate nulls as. eg "null", empty string, or something else just in case the word "null" is valid in their use case. |
The existing versions didn't play nice with Docker on M1 (ARM) * localstack to 0.14 * postgres to 13.6 * bats-core to 1.5.0
…ng literal update
|
Hello again. Apologies - only just getting back to this now after finally grabbing some free cycles. I've added an integration test for the new flag behaviour, and tweaked some existing ones the tweak I made to the unchanged toast datum detection. I had to bump the test dependencies in order to get the suite running on an M1 MacBook - I hope that is ok. Re: allowing users to provide their own values - this is already covered by the database with the Anyone that is currently checking |
| @@ -1,6 +1,6 @@ | |||
| FROM postgres:10.13 | |||
| FROM postgres:13.6 | |||
There was a problem hiding this comment.
I ran the integration tests and I'm having a couple related issues because postgres takes a little longer to startup. First the docker build of the postgres container times out because bootstrap.sh times out waiting for postgres to start. Increasing the loop counter in bootstrap.sh helped. The second issue I had was when I ran the integration tests pg-bifrost errored creating the replication slot because postgres wasn't ready yet. I think the create function could use a retrier or loop.
bifrost | time="2022-03-07T22:00:43Z" level=error msg="Unable to create replication slot 'pg_bifrost'" package=app
bifrost | time="2022-03-07T22:00:43Z" level=fatal msg="failed to connect to `host=postgres user=replication database=postgres`: server error (FATAL: the database system is starting up (SQLSTATE 57P03))" package=app
Within an
UPDATEoperation, downstream consumers currently cannot differentiate between an unchanged column and an update fromNULLto notNULL. This is of course only relevant for tables that have setREPLICA IDENTITY FULL.This is due to an interaction quirk that arises from the following details:
NULLvalues fromold-key.oldkey being omitted from the published output represents an unchanged column.This PR attempts to address this by adding a new
infer-updated-nullssetting, which causes the marshaller to inject aNULLvalue for each column missing from the decodedold-key.Another way I'd considered solving this was by adding a setting to always output the
oldvalue, even if the value was unchanged, but this seems like the more efficient approach.