passing argument from shell script to hive script

Question

I've a concern which can be categorized in 2 ways: My requirement is of passing argument from shell script to hive script. OR within one shell script I should include variable's value in hive statement.

I'll explain with an example for both:

1) Passing argument from shell script to hiveQL->

My test Hive QL:
select count(*) from demodb.demo_table limit ${hiveconf:num}

My test shell script:

cnt=1
sh -c 'hive -hiveconf num=$cnt -f countTable.hql'

So basically I want to include the value of 'cnt' in the HQL, which is not happening in this case. I get the error as:

FAILED: ParseException line 2:0 mismatched input '<EOF>' expecting Number near 'limit' in limit clause

I'm sure the error means that the variable's value isn't getting passed on.

2) Passing argument directly within the shell script->

cnt=1
hive -e 'select count(*) from demodb.demo_table limit $cnt'

In both the above cases, I couldn't pass the argument value. Any ideas??

PS: I know the query seems absurd of including the 'limit' in count but I have rephrased the problem I actually have. The requirement remains intact of passing the argument.

Any ideas, anyone?

Thanks in advance.

can you do echo in second case and see what you are getting: echo 'select count(*) from demodb.demo_table limit $cnt' — Amar
– Amar, Commented Jun 2, 2015 at 9:25

sras · Accepted Answer · 2015-06-02 11:20:37Z

9

Set the variable this way:

#!/bin/bash
cnt=3
echo "Executing the hive query - starts"
hive -hiveconf num=$cnt -e ' set num; select * from demodb.demo_table limit ${hiveconf:num}'
echo "Executing the hive query - ends"

answered Jun 2, 2015 at 11:20

sras

8387 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Charles Duffy Over a year ago

num="$cnt", if you want to make the answer more generic (able to pass things that aren't just numbers and may contain whitespace, glob characters, etc).

knowone Over a year ago

Thanks sras, this is perfect. I never used multiple options for hive from command line. Totally agree with Charles; single quotes suppresses information.

sras Over a year ago

to be more precise, we can use hivevar name space rather than hiveconf name space. However both will work for your query.

Charles Duffy · Accepted Answer · 2015-06-03 00:23:44Z

2

This works, if put in a file named hivetest.sh, then invoked with sh hivetest.sh:

cnt=2
hive -e "select * from demodb.demo_table limit $cnt"

You are using single quotes instead of double. Using double quotes for OPTION #1 also works fine.

edited Jun 3, 2015 at 0:23

Charles Duffy

299k43 gold badges441 silver badges496 bronze badges

answered Jun 3, 2015 at 0:00

Madhu

214 bronze badges

Comments

y durga prasad · Accepted Answer · 2017-11-23 07:26:45Z

1

hadoop@osboxes:~$ export val=2;

hadoop@osboxes:~$ hive -e "select * from bms.bms1 where max_seq=$val";

or

vi test.sh
#########
export val=2
hive -e "select * from bms.bms1 where max_seq=$val";
#####################

edited Nov 23, 2017 at 7:26

answered Nov 23, 2017 at 7:16

y durga prasad

1,2229 silver badges11 bronze badges

Comments

Aman · Accepted Answer · 2015-06-02 12:24:44Z

0

Try this
cnt=1

hive -hiveconf number=$cnt select * from demodb.demo_table limit ${hiveconf:number}

answered Jun 2, 2015 at 12:24

Aman

3,2717 gold badges39 silver badges60 bronze badges

Collectives™ on Stack Overflow

passing argument from shell script to hive script

4 Answers 4

3 Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related