Can't get IFS to work when converting array to string

Question

Below is a bash shell script for taking in a csv file and spitting out rows formatted the way I want (Some more changes are there, but I only kept the array affecting ones below to show).

FILENAME=$1
cat $FILENAME | while read LINE
do
    OIFS=$IFS;
    IFS=","
    columns=( $LINE )
    date=${columns[4]//\"/}
    columns[13]=${columns[13]//\"/}
    columns[4]=$(date -d $date +%s)
    newline=${columns[*]}
    echo $newline
    IFS=$OIFS;
done

I'm using GNU bash v 4.1.2(1)-release for CentOS 6.3. I've tried putting quotes like

newline="${columns[*]}"

Still no luck.

Following is sample data line

112110120001299169,112110119001295978,11,"121.119.163.146.1322221980963094","2012/11/01"

It seems like it should be outputting the array into a comma delimited string. Instead, the string is space delimited. Anyone know the reason why?

I suspect it has something to do with the fact that if I echo out $IFS in script it's an empty string, but when I echo out "${IFS}" it's then the comma I expect.

Edit: Solution

I found the solution. When echoing out $newline, I have to use quotes around it, i.e.

echo "$newline"

Otherwise, it uses the default blanks. I believe it has something to do with bash only subbing in for the IFS when you force it to with the quotes.

Gordon Davisson · Accepted Answer · 2012-12-22 03:24:21Z

2

I'm not clear on why, but bash only seems to use the first character of IFS as a delimiter when expanding ${array[*]} when it's in double-quotes:

$ columns=(a b "c d e" f)
$ IFS=,
$ echo ${columns[*]}
a b c d e f
$ echo "${columns[*]}"
a,b,c d e,f
$ newline=${columns[*]}; echo "$newline"
a b c d e f
$ newline="${columns[*]}"; echo "$newline"
a,b,c d e,f

Fortunately, the solution is simple: use double-quotes (newline="${columns[*]}")

(BTW, my testing was all on bash v3 and v2, as I don't have v4 handy; so it might be different for you.) (UPDATE: tested on bash v4.2.10, same results.)

edited Dec 22, 2012 at 3:24

answered Dec 22, 2012 at 3:03

Gordon Davisson

128k16 gold badges141 silver badges168 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

gniourf_gniourf Over a year ago

If you want to be clear why, just read the bash reference manual. It's all in there.

ormaaj Over a year ago

That's POSIX behavior for the expansion of "$*". Array expansions in essentially every shell with ksh-like arrays are meant to behave analogously to the @ and * special parameters. (except zsh which injects a lot of changes)

Gordon Davisson Over a year ago

@gniourf_gniourf, @ormaaj: The behavior in double-quotes makes sense (& follows the manual). After thinking about it, echo ${columns[*]} also makes sense (bash expands each element to a word, passes them to echo that way, echo prints them with spaces between). But I don't understand the newline=${columns[*]} result -- the manual says [*] without double-quotes expands each element to a separate word, but on the RHS of an assignment word splitting doesn't take place after expansion (if it did, only the first word would be assigned to newline), so this seems to be an undefined situation.

saccharine Over a year ago

Doesn't work. I've already tired using double quotes. It works for me in shell as well, but it doesn't work in script.

saccharine Over a year ago

Actually, nevermind, it does work. I was echoing out newline without the quotes. It seems bash is handling things in the back quite differently from what I'd expect.

|

shellter · Accepted Answer · 2012-12-22 04:10:36Z

Edit Thanks to @GordonDavidson, Removed erroneous comments about how IFS works in bash.

awk has a very nice pair of vars, name FS=","; OFS="|" that do perform this transformation. You'll have to construct awk -F, '{"date -d "$date" +%s" | getline columns[4]}' or similar to call external programs and fill variables. Not quite as intuitive as the shell's c[4]=$(date ...), but awk is a very good tool to learn for data manipulations like you have outlined in your question.

Something like

#!/bin/awk -f
{
  # columns=( $LINE )
  split($0, columns)

  # date=${columns[4]//\"/}
  myDcolucolumns[4] ; gsub(/\"/, "", myDate)
  # gcolumns[13]=${columns[13]//\"/}
  gsub(/\"/,""columns[13]}
  # columns[4]=$(date -d $date +%s)
  "date -d '"$date"' +%s" | getline columns[4]

  #Don_t_need_this newline=${columns[*]}

  #echo $newline
} print  $0

used like

 cat myFile | myAwkScript

should achieve the same result.

Sorry but I don't have the time, OR the sample data to test this right now. Feel free to reply with error messages that you get, and I'll see if I can help.

You might also consider updating your posting with 1 line of sample data, and a date value you want to process. IHTH

Collectives™ on Stack Overflow

Can't get IFS to work when converting array to string

2 Answers 2

6 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related