5

Maybe this question is strongly related to How to set pivotMaxValues in pyspark?. But I haven't find any clear answer about this.

So, how I change spark.sql.pivotMaxValues in my code or configuration file? Where can I find the configuration file?

Improve this question
2
  • 8
    If your are using Spark 2.0+ you can do that spark.conf.set('spark.sql.pivotMaxValues', u'50000') where spark is your sparkSession Commented Aug 7, 2018 at 13:43
  • This comment also works at the start of a pyspark notebook. Commented Jan 11, 2020 at 0:31
Related questions
136
importing pyspark in python shell
106
Cannot find col function in pyspark
169
How to set Apache Spark Executor memory
Load 7 more related questions Show fewer related questions

0

Reset to default

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.