3 questions from the last 7 days
Best practices
0
votes
3
replies
62
views
Pushing down filters in RDBMS with Java Spark
I have been working as a Data Engineer and got this issue.
I came across a use case where I have a view(lets name it as inputView) which is created by reading data from some source.
Now somewhere ...
1
vote
0
answers
52
views
Databricks always loads built-in BigQuery connector (0.22.2), can’t override with 0.43.x
I am using Databricks Runtime 15.4 (Spark 3.5 / Scala 2.12) on AWS.
My goal is to use the latest Google BigQuery connector because I need the direct write method (BigQuery Storage Write API):
option(&...
- 157
0
votes
0
answers
31
views
PySpark 3.5.5 CharType in read.csv schema definition
I'm using a PySpark notebook inside of Azure Synapse.
This is my schema definition
qcew_schema = StructType([
StructField( 'area_fips', dataType = CharType(5), ...
