Storing Result of SQL Query in RDD

Question

Afternoon All,

I am attempting to call some Spark SQL on a SchemaRDD, and then the result stored in an RDD. The below line is producing the expected values, so I know the SQL is generating the correct table. Now I just need to store it.

sqlContext.sql("select encounter.Member_ID AS patientID, encounter.Encounter_DateTime AS date, diag.code from encounter join diag on encounter.Encounter_ID = diag.Encounter_ID").show(1)

Pawan B · Accepted Answer · 2016-10-09 05:06:01Z

1

sqlContext.sql gives the DataFrame, you can call .rdd() to get the RDD[Row] .

You can try this:

 val queryResult = sqlContext.sql("select encounter.Member_ID AS patientID, encounter.Encounter_DateTime AS date, diag.code from encounter join diag on encounter.Encounter_ID = diag.Encounter_ID")

 val rdd: RDD[Row] = queryResult.rdd

Remove the show function on DataFrame since it displays the content of the DataFrame to stdout

edited Oct 9, 2016 at 5:06

answered Oct 9, 2016 at 4:54

Pawan B

4,6332 gold badges27 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

mongolol Over a year ago

Thanks for the response! Dropped the .show(1) at the end as well.

Pawan B Over a year ago

remove the .show() at the end. show method shows you some random rows from the dataframe

mongolol Over a year ago

Got it to work for RDD[Row]. Now I am trying to match this table with the following class of RDD - case class Diagnostic(patientID:String, date: Date, code: String). Attempting to build the RDD with

val res1: RDD[Diagnostic] = sqlContext.sql("select encounter.Member_ID AS patientID, encounter.Encounter_DateTime AS date, diag.code from encounter join diag on encounter.Encounter_ID = diag.Encounter_ID").rdd

Pawan B Over a year ago

i m nt sure if it will match since queryResult.rdd returns you the RDD[Row]

Collectives™ on Stack Overflow

Storing Result of SQL Query in RDD

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related