How to summarize Spark executor computing time

Question

I have a spark job running with 260K tasks. I can check individual task executor computing time from Spark UI. For the purpose of calculate resource usage of the whole job, how to summarize all computing time?

chintan shah · Accepted Answer · 2025-07-30 17:41:24Z

0

Using Spark UI → Stages tab

Go to the Spark UI.
Open the Stages tab.
For each stage:
- Look at the column “Executor CPU Time” (or sometimes called “Task Time”).
- This shows total executor CPU time in milliseconds for all tasks in that stage.
Sum these values for all stages to get total compute time of the job.
- Alternatively, you can download the Spark event logs and script the parsing to extract CPU time per stage.

answered Jul 30 at 17:41

chintan shah

212 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to summarize Spark executor computing time

1 Answer 1

Using Spark UI → Stages tab

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Using Spark UI → Stages tab

Comments

Your Answer

Sign up or log in

Post as a guest

Related