How to handle environmental variable in spark scala

Question

I am very new to spark. Please help me with below query:

Suppose I have different file path in dev and prod and i do not want to hardcode the same in my spark scala code rather I want to use variable in my code so my doubt is that where I define these variables for dev and prod and how to access them in my code.

Vikramsinh Shinde · Accepted Answer · 2022-06-29 18:49:28Z

0

Here are a few options:

Pass variables as command line parameters to your Spark application
Use the property/config file to keep those variables
Create environmental variables for these variables

For implementation, follow these articles.

edited Jun 29, 2022 at 18:49

answered Jun 29, 2022 at 3:45

Vikramsinh Shinde

2,8832 gold badges26 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Ankit Gupta Over a year ago

@vikramsinhThanks for the response, it would be very helpful if you share how to set env variable for the variables and use them in the application or share some reference

Vikramsinh Shinde Over a year ago

Updated the answer to include a few useful links

Collectives™ on Stack Overflow

How to handle environmental variable in spark scala

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related