How to set pyspark logging level to debug?

Question

Can anyone help me with the spark configuration needed to set logging level to debug and capture more logs

Does this answer your question? How can set the default spark logging level? — William Prigol Lopes
– William Prigol Lopes, Commented May 6, 2020 at 18:51

notNull · Accepted Answer · 2020-05-06 18:39:23Z

8

Set setLogLevel property to DEBUG in sparksession.

from pyspark.sql import SparkSession
spark = SparkSession.builder.master('local').appName('test').getOrCreate()
spark.sparkContext.setLogLevel("DEBUG")

#or

log4j = sc._jvm.org.apache.log4j
log4j.LogManager.getRootLogger().setLevel(log4j.Level.DEBUG)

#for custom log
log4j.LogManager.getRootLogger().info("custom log")
#2020-05-06 13:43:42 INFO  root:? - custom log

edited May 6, 2020 at 18:39

answered May 6, 2020 at 18:19

notNull

31.8k4 gold badges41 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user13485171 Over a year ago

Thanks. But i think this line is creating end less console output in my case. The error was around "connection error"

notNull Over a year ago

@user13485171, Could you update the question with steps you are performing and error?

user13485171 Over a year ago

I would like to but i can't as that's little confidential My code looks like Setting environment variables Creating spark session similarly Then i tried to change log level So with the new code recreted the issue I think it's more because of my server settings/permission I'll take this up with my IT and update you why it happened

Collectives™ on Stack Overflow

How to set pyspark logging level to debug?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related