1

I am submitting java jar using spark-submit to spark standalone cluster manager. But I am not able to provide custom log4j.xml file to it. If i used --files option then i need to copy that log4j file to all machines at that same position or if I give it using hdfs:// path then it is not taking that file as log4j and switch to default log4j file.

I also used -Dlog4j.configuration with both local and hdfs path but same issue is happening as using --files.

Please help me if someone know the solution

Thanks in advance.

Improve this question

1 Answer 1

Reset to default
1

Spark does not support writing logs in HDFS using log4j rather you can write the logs local in your unix box.

The porperties for specifying log4j in spark-submit command are:

--conf "spark.driver.extraJavaOptions=-Dlog4j.configuration= Location of your log4j.properties file"

--conf "spark.executor.extraJavaOptions=-Dlog4j.configuration= location of your log4j.properties file"

You have to create a custom log4j.properties file not log4j.xml in your local unix box.

If you want to create custom logs in HDFS you can create a interface in java or trait in scala to specify the loging levels and write the log to HDFS for more reference you can check this question.

Improve this answer
Sign up to request clarification or add additional context in comments.

9 Comments

I am writing log into unix box, i am just reading the log4j file from HDFS location. As using log4j.configuration it search the file from local fs, so i need to copy the saem file to all my worker nodes. So I am searching a way to submit log4j file from HDFS or from client machine, so that it will not needed to copy on all nodes
no need of copying the log4j file to all the nodes. The node from which you submit jobs it must be fixed so the file should be in that node only and whenever you submit it a job spark loads the external jars r files that you specify in spark-submit into the cluster for running in distributed mode.
yes I am using the fixed node, but when i gave the path of log4j file, it will search that path on driver node which was different from the submitted node due to which I got below error: log4j:ERROR Could not parse url [file:/data/log4j.xml]. java.io.FileNotFoundException: /data/log4j.xml (No such file or directory)
Specify the log4j.properties file through the above parameters in my answer not through --files parameter. can you please show me your spark-submit command?
this should be the command spark-submit --class xyz --master "yarn" --deploy-mode "client" --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=../conf/spark-log4j.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=../conf/spark-log4j.properties" ../lib/xyz.jar
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.