0

In Databricks, I uploaded my source data under a volume folder (e.g., /Volumes/my_catalog/my_schema/landing_source/). Now, I want to create a DataFrame or table using this volume path as the source.

1. Using Spark API:

I am able to successfully create a DataFrame using the spark.read API:

df = spark.read.format("csv") \
    .option("header", "true") \
    .load("/Volumes/my_catalog/my_schema/landing_source/")

2. Using SQL CREATE TABLE with datasource:

When I try to create a table directly using SQL with the volume path as the data source:

CREATE TABLE IF NOT EXISTS my_catalog.my_schema.my_table
USING CSV
OPTIONS (
  path "/Volumes/my_catalog/my_schema/landing_source/",
  header "true",
  inferSchema "true"
)

I get the following exception:

[RequestId=4800ec53-1a9d-946e-b61f-0ab7c93b88fc ErrorClass=INVALID_PARAMETER_VALUE.INVALID_PARAMETER_VALUE] Missing cloud file system scheme JVM stacktrace: com.databricks.sql.managedcatalog.UnityCatalogServiceException at com.databricks.managedcatalog.ErrorDetailsHandler.wrapServiceException(ErrorDetailsHandler.scala:111) at com.databricks.managedcatalog.ErrorDetailsHandler.wrapServiceException$(ErrorDetailsHandler.scala:66) at com.databricks.managedcatalog.ManagedCatalogClientImpl.wrapServiceException(ManagedCatalogClientImpl.scala:266) at com.databricks.managedcatalog.ManagedCatalogClientImpl.recordAndWrapExceptionBase(ManagedCatalogClientImpl.scala:7309) at com.databricks.managedcatalog.ManagedCatalogClientImpl.recordAndWrapException(ManagedCatalogClientImpl.scala:7295) at com.databricks.managedcatalog.ManagedCatalogClientImpl.generateTemporaryPathCredentials(ManagedCatalogClientImpl.scala:6277)

Currently i am using Databricks trial account for my learning. How can i fix this issue.

Improve this question
4
  • Why are you trying to create a table in a path location which already exists and has CSV files? Commented Jul 20 at 12:02
  • required to read CSV file in job Commented Jul 21 at 16:05
  • Instead of putting the path in OPTIONS, try using the LOCATION parameter Commented Aug 16 at 5:19
  • did you find a solution? Commented Oct 27 at 14:50
Related questions
1
Spark (Databricks) unmanaged table from SQL not processing headers
0
How to read a dataframe with inferschema as true
0
Is there any difference between using "true" and True in PySpark Load/Save functions?
Load 3 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.