Using the templates found here
Running the commands
job = client.load_table_from_storage(
'load-from-storage' + datetime.now().strftime('%Y%m%d%H%M'),
table, gsbucket)
job.skip_leading_rows = 0
job.writeDisposition = 'WRITE_TRUNCATE'
job.field_delimiter = ','
job.begin()This piece of code appends to my table instead of overwriting it like the docs say.
WRITE_TRUNCATE: If the table already exists, BigQuery overwrites the table data.
I know this because if I use:
table.reload()
print(table.num_rows)
after the job it has increased by 2 million - the size of the table.
I use a workaround as follows:
if disposition == 'WRITE_TRUNCATE':
schema = table.schema
while table.exists():
table.delete()
table = dataset.table(table_name, schema=schema)
table.create()Which seems to work fine, and the while loop is just me making certain that the table is deleted but when I tested it without the while loop it also worked.
Anyway, the issue is WRITE_TRUNCATE isn't doing what it say it does in the docs.
Using the templates found here
Running the commands
This piece of code appends to my table instead of overwriting it like the docs say.
I know this because if I use:
after the job it has increased by 2 million - the size of the table.
I use a workaround as follows:
Which seems to work fine, and the while loop is just me making certain that the table is deleted but when I tested it without the while loop it also worked.
Anyway, the issue is WRITE_TRUNCATE isn't doing what it say it does in the docs.