spark.sql cannot execute a basic SQL call. Issue on FROM

Question

I basically want to get the name, number of stars, and the number of reviews, of the restaurants with 5 stars and more than 1000 reviews.

  def fiveStarBusinessesSQL():DataFrame = {
    spark.sql("SELECT name, stars, review_count FROM yelpBusinessesView WHERE stars == 5 && review_count >= 1000")
  }

It makes no sense to me why I get the error. It is a basic SQL call, as basic as it can get IMO.

Here's the error I get:

Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException: 
mismatched input 'FROM' expecting <EOF>(line 1, pos 33)

== SQL ==
SELECT name, stars, review_count FROM yelpBusinessesView WHERE stars == 5 && review_count >= 1000
---------------------------------^^^

I'm working on the Yelp Dataset. Here's an example of what's in yelpBusinessesView

{"business_id":"1SWheh84yJXfytovILXOAQ","name":"Arizona Biltmore Golf Club","address":"2818 E Camino Acequia Drive","city":"Phoenix","state":"AZ","postal_code":"85016","latitude":33.5221425,"longitude":-112.0184807,"stars":3.0,"review_count":5,"is_open":0,"attributes":{"GoodForKids":"False"},"categories":"Golf, Active Life","hours":null}

Try changing && to AND.

Shaido
– Shaido

2019-10-18 05:46:11 +00:00
Commented Oct 18, 2019 at 5:46 — Shaido
– Shaido, Commented Oct 18, 2019 at 5:46
That did the trick. Thanks a lot!

doomdaam
– doomdaam

2019-10-18 06:02:36 +00:00
Commented Oct 18, 2019 at 6:02 — doomdaam
– doomdaam, Commented Oct 18, 2019 at 6:02
Happy to help :)

Shaido
– Shaido

2019-10-18 07:39:59 +00:00
Commented Oct 18, 2019 at 7:39 — Shaido
– Shaido, Commented Oct 18, 2019 at 7:39

test · Accepted Answer · 2019-10-18 09:32:24Z

1

Use String Interpolation while dealing with plain SQL queries

answered Oct 18, 2019 at 9:32

test

261 bronze badge

Sign up to request clarification or add additional context in comments.

2 Comments

papanito Over a year ago

Can you give some more details how this exactly should be done, would be helpful for others

doomdaam Over a year ago

Welcome to S.O. Let me be the first one to hand you some rep.

Collectives™ on Stack Overflow

spark.sql cannot execute a basic SQL call. Issue on FROM

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related