How to create external table in BigQuery using Dataproc Pyspark

Question

My use case includes creating an external table in Bigquery using Pyspark code. The data source is Google cloud storage bucket where JSON data is sitting. I am reading the JSON data into a data frame and want to create an external Bigquery table. As of now, the table is getting created but it is not an external one.

df_view.write\
    .format("com.google.cloud.spark.bigquery")\
    .option('table', 'xyz-abc-abc:xyz_zone.test_table_yyyy')\
    .option("temporaryGcsBucket","abcd-xml-abc-warehouse")\
    .save(mode='append',path='gs://xxxxxxxxx/')

P.S. - I am using spark-bigquery connector to achieve my goal.

Please let me know in case anyone has faced the same issue.

David Rabinowitz · Accepted Answer · 2020-02-26 16:09:30Z

1

At the moment the spark-bigquery-connector does not support writing to an external table. Please create an issue and we will try to add it soon.

You can of course do it in two steps:

Write the JSON files to GCS.
Use the BigQuery API in order to create the external table.

answered Feb 26, 2020 at 16:09

David Rabinowitz

30.5k16 gold badges95 silver badges125 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Pritam Over a year ago

Thanks David. I will raise a ticket with Google Cloud support team.

Collectives™ on Stack Overflow

How to create external table in BigQuery using Dataproc Pyspark

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related