How to covert dataframe datatypes to String?

Question

I have a hive Table having Date and Timestamp datatypes. I am creating DataFrame using below java code:

SparkConf conf = new SparkConf(true).setMaster("yarn-cluster").setAppName("SAMPLE_APP");
SparkContext sc = new SparkContext(conf);
HiveContext hc = new HiveContext(sc);
DataFrame df = hc.table("testdb.tbl1");

Dataframe schema:

 df.printSchema
root
 |-- c_date: date (nullable = true)
 |-- c_timestamp: timestamp (nullable = true)

I want to covert these columns to String. How can I achieve this?

I need this because of issue : Spark csv data validation failed for date and timestamp data types of Hive

karthik manchala · Accepted Answer · 2017-03-24 10:05:40Z

2

You can do the following:

df.withColumn("c_date", df.col("c_date").cast(StringType))

answered Mar 24, 2017 at 10:05

karthik manchala

13.7k1 gold badge34 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

ROOT · Accepted Answer · 2017-03-24 08:42:30Z

2

In scala, we generally cast datatypes like this:

df.select($"date".cast(StringType).as("new_date"))

answered Mar 24, 2017 at 8:42

ROOT

1,7734 gold badges35 silver badges60 bronze badges

Collectives™ on Stack Overflow

How to covert dataframe datatypes to String?

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related