Bash variable with newlines not being represented correctly within csv file

Question

I'm trying to stick the text output of a linux command into a single cell of a csv file. No matter how I try to escape quotes, the newline characters do not seem to be registered by the csv file.

For example if the output of one of these commands is:

155 C T
2518 T C

It is being printed within the csv cell as:

155 C T 2518 T C

for filename in "$1"/*JGI*.vcf; do
    counter=$((counter + 1))

    depth_score="$(samtools depth "${filename%.*}.bam" | python depth.py)"
    variant_array="$(bcftools query -f '%POS %REF %ALT\n' "$filename")"
    column_name="$(basename "$filename")"

    echo "$counter/$total_colonies" # echo current/total
    echo "$variant_array"
    echo ""$variant_array""
    echo "$column_name, "$variant_array", $depth_score" >> output.csv
done

Here is the for loop that's giving me some errors.

The result of the $variant_array is the problem.

I have two print logs above where I write to the output.csv to see what's going on.

"$variant_array" yields the desired behavior, but for some reason when I echo that within another set of quotes to write it to the csv file, it ignores the newlines...

Add a shebang and then paste your script there: shellcheck.net — Cyrus
– Cyrus, Commented May 24, 2019 at 4:38

Michael Back · Accepted Answer · 2019-05-24 04:43:43Z

2

Quotes do not escape this way... So perhaps the following will work for you?

echo '"'"$variant_array"'"'
echo "$column_name", '"'"$variant_array"'"', "$depth_score" >> output.csv

Here is another form of escaping quotes:

echo \""$variant_array"\"
echo "$column_name", \""$variant_array"\", "$depth_score" >> output.csv

Here is another idea:

echo "\"$variant_array\""
echo "$column_name, \"$variant_array\", $depth_score" >> output.csv

edited May 24, 2019 at 4:43

answered May 24, 2019 at 4:38

Michael Back

1,8712 gold badges16 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

John1024 Over a year ago

Good point. In a CSV file, the OP probably does want double-quotes to appear in the output.

John1024 Over a year ago

Possibly simpler: echo "\"$variant_array\""

kbw Over a year ago

Both the second two were perfect solutions. Thanks!

John1024 · Accepted Answer · 2019-05-24 04:38:01Z

0

The problem is that $variant_array is not "within another set of quotes." It is entirely unquoted.

Replace:

echo "$column_name, "$variant_array", $depth_score" >> output.csv

with:

echo "$column_name", "$variant_array", "$depth_score" >> output.csv

or, simpler still:

echo "$column_name, $variant_array, $depth_score" >> output.csv

answered May 24, 2019 at 4:38

John1024

115k15 gold badges152 silver badges183 bronze badges

1 Comment

John1024 Over a year ago

It happens to all of us at one time or another.

Collectives™ on Stack Overflow

Bash variable with newlines not being represented correctly within csv file

2 Answers 2

3 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related