How to pass a command line argument into a kdb function

Question

I wrote a shell script to collect data on each character of the alphabet from a large table. As a result of the function being memory intensive, I'd like to partition the characters of the alphabet so that each character is concurrently called using a different port number. However I can't seem to successfully pass the desired command line argument into my function. Testing on a single port and a small table, what I've tried is the following...

On the server: I set up my dummy table and defined a function...

ts:([]sym:1000?`A`Ab`B`Bc`C`Ca`X`Xz`Y`Yx`Z`Zy;price:1000?100.0;num:til 1000)
collect:{[x;y]select from x where sym like y}

On the client: I open a connection handle, use the .z.X namespace to return strings of the raw, unfiltered command line arguments, index into it and store as a variable, and then attempt to pass that variable into my server side function. I've tried storing as a character and a symbol. It will run without error but neither returns any data when called. I save this file as collector.q..

/ start connection handles to remote server
h:hopen `::5000
/ index into command line arguments to get partition letter. Store as character and symbol
part:.z.X[6]
symPart:`$part
/ call server side functions
fetched:h (`collect; `ts; "symPart*")
/ close connection handle
hclose h

my shell script looks like the following...

#!/bin/sh
port=$1
partition=$2

for x in {A..Z}
do
  echo running partition $x on port $port
  $QHOME/l64/q collector.q -p $port -partition $x > ./stdout$port.log 2>&1 &
  port=$(($port + 1))
done

After running the shell script, when calling the fetched function on the client, what's returned is an empty table...

q)fetched
symbol price ID
---------------
q)

Mark Kelly · Accepted Answer · 2019-01-22 20:01:07Z

I think the first issue is that you are not getting the correct value for part from the command line. .z.X returns the command line as a list of tokens while .z.x returns the same but without the q command and file name.

q test.q -p 5000 -partition a

q).z.X
"/opt/kdb/3.5/l32/q"
"test.q"
"-p"
"5000"
"-partition"
,"a"
q).z.X[6]
""

Use .Q.opt function to turn the command line parameters into a dictionary that much more reliable.

q)params:.Q.opt .z.X
q)params
p        | "5000"
partition| ,"a"
q)`$first params`partition
`a

The second issue is that "symPart*" won't evaluate the variable symPart as it is inside quotes. All kdb+ sees is a string. Each loop of the bash script will send the same command (`collect; `ts; "symPart*") over the handle, with the collect checking against a "symPart" partition which I guess doesn't exist.

To pass it in as a variable you can change it to symPart,"*", but in this case symPart needs to be a string, not a symbol so you can remove the casting.

You can amend your script as follows:

/ start connection handles to remote server
h:hopen `::5000
/ convert command line arguments into dictionary and index to get partition letter
part:first .Q.opt[.z.X]`partition;
/ call server side functions
fetched:h (`collect; `ts; symPart,"*")
/ close connection handle
hclose h

Or to do this all in a single q script you could do the following:

/ start connection handles to remote server
h:hopen `::5000

/ call server side functions
fetched:.Q.a!h each (`collect; `ts),/: enlist each .Q.a,\:"*"
/ close connection handle
hclose h

.Q.a holds all the lowercase letters in a string.

q).Q.a
"abcdefghijklmnopqrstuvwxyz"

We can create all the wildcards using the each left \: with , (join) to join each of them to "*" and then create multiple commands to send over the handle using each right /: to join the unchanging left hand side to all of the different wildcards.

q).Q.a,\:"*"
"a*"
"b*"
"c*"
"d*"
..

q)(`collect; `ts),/: enlist each .Q.a,\:"*"
`collect `ts "a*"
`collect `ts "b*"
`collect `ts "c*"
..

Note the output of fetched will be a dictionary with keys as the partitions and the fetched result of each partition as the corresponding value.

Collectives™ on Stack Overflow

How to pass a command line argument into a kdb function

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related