Bash array from CSV and awk

Question

I'm trying to create an array in BASH with the following command:

# Normally a loop, but for this example, n=1
arr=( `awk 'NR == n' n=1 "$file" | tr ',' " "` );
echo ${arr[@]};
echo ${arr[2]};

$file contains:

"1","test_id = \"5\"","test_id <> \"5\""
"2","test_id = \"6\"","test_id <> \"6\""

Output:

"1" "test_id = \"5\"" "test_id <> \"5\""
"test_id

I'm expecting output of ${arr[2]} to be test_id <> "5" but I actually get "test_id.

However, when I manually input the data like so, I get the correct result:

arr=( "1" "test_id = \"5\"" "test_id <> \"5\"" );
echo ${arr[@]};
echo ${arr[2]};

Output:

1 test_id = "5" test_id <> "5"
test_id <> "5"

Any help would be appreciated,

Thanks.

The shell splits on blanks — those that you add when you replace the commas and those that are already in the fields. — Jonathan Leffler
– Jonathan Leffler, Commented Nov 5, 2015 at 0:41
But how does that explain why the second command works? It contains blanks within the strings.. And it doesn't get split. — Travis Liew
– Travis Liew, Commented Nov 5, 2015 at 0:45
In the second case, the shell is interpreting the quotes. In the first, it does not. It's tricky, but that's the trouble. You'll also note I didn't quote a fix; that's because it isn't particularly easy to come up with one. In fact, given that you have nested double quotes in your data, and it's only a matter of time before there's a comma inside one of the fields, there really isn't a suitable way to deal with it in shell. At this point, you need a tool that handles CSV more sanely. That might be Python, or Perl, or a custom program such as csvfix. — Jonathan Leffler
– Jonathan Leffler, Commented Nov 5, 2015 at 0:49
I also observe that you've got a non-standard variant of CSV format. Normally, a double quote inside a field is represented by "" and not \". That's going to give some headaches too. What happens with commas and backslashes inside the field? I don't think your variant is too hard to handle, but because it doesn't match the default, you will need to tweak tools such as Python's CSV module to handle it properly. Just a heads-up. — Jonathan Leffler
– Jonathan Leffler, Commented Nov 5, 2015 at 1:01

joepd · Accepted Answer · 2015-11-05 21:23:40Z

1

Thew correct answer is of course in the comments: Use a real CSV-parser. That being said, this would be a contrived way of solving it:

. <(
      echo "arr=(";
      awk -F, 'NR==n && $1=$1' n=1 file
      echo ")"
)

Sourcing the result of this process substitution subtly changes the order in which the stages of quote interpretation happens. It often helps with these intricate issues. Ah, and to get rid of the unnnecesary tr call, awk is instructed to split on comma's. $1=$1 always results true, and causes awk to re-evaluate its line, so the input field separator is changed for the output field separator.

answered Nov 5, 2015 at 21:23

joepd

4,8612 gold badges28 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Bash array from CSV and awk

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related