Using awk or perl to extract specific columns from CSV (parsing)

I'd suggest python csv module:

#!/usr/bin/env python3
import csv
rdr = csv.reader(open('input.csv'), escapechar='\\')
wtr = csv.writer(open('output.csv', 'w'), escapechar='\\', doublequote=False)
for row in rdr:
    wtr.writerow(row[0:1]+row[2:4])

output.csv

John \"Super\" Doe,123 ABC Street,123-456-7890
"Jane, Mary",132 CBS Street,333-111-5332
"Smith \"Jr.\", Jane",,555-876-1233
"Lee, Jack",123 Sesame St,

The following command will extract the required fields(e.g., first, third and fourth) separated by delimiter ',' from sample.csv file and displays the output in the console. cut -f1,3,4 -d',' sample.txt If you want to store the output in a new csv file, then redirect the output to a file as below cut -f1,3,4 -d',' sample.txt > newSample.csv

Before I post, I see now that this is an old question bumped by an already deleted answer, however, I thought I would still use the opportunity to show off Tie::Array::CSV which make CSV file manipulation as easy as working with Perl arrays. Full disclosure: I'm the author.

Anyway here is the script. The OP's data required changing the escape character and Perl indexes arrays starting at 0, but other than that this should be quite readable.

#!/usr/bin/env perl

use strict;
use warnings;

use Tie::Array::CSV;

my $opts = { text_csv => { escape_char => '\\' } };

tie my @input,  'Tie::Array::CSV', 'data', $opts or die "Cannot open file 'data': $!";
tie my @output, 'Tie::Array::CSV', 'out',  $opts or die "Cannot open file 'out': $!";

for my $row (@input) {
  my @slice = @{ $row }[0,2,3];
  push @output, \@slice;
}

That said, I think that last loop doesn't loose too much readability if I convert it to the (IMO) more impressive form:

push @output, [ @{$_}[0,2,3] ] for @input;

csvkit is a tool that handles csv files and allows such operations (among other features).

see csvcut. Its command line interface is compact and it handles the multitude of csv formats (tsv, other delimiters, encodings, escape chars etc.)

What you asked for can be done using:

csvcut --columns 0,2,3 input.csv

GNU awk solution. Just using the wheel as a wheel. You can define what fields should look like using FPAT, like this:

$ awk -vFPAT='[^,]+|"[^"]*"' -vOFS=, '{print $1, $3, $4}' file

which results in:

"John \"Super\" Doe","123 ABC Street",123-456-7890
"Jane, Mary",132 CBS Street,333-111-5332
"Smith \"Jr.\",35,555-876-1233
"Lee, Jack",123 Sesame St,""

Explanation of the regex:

[^,]+           # 1 or more occurrences of anything that's not a comma, 
|               # OR
"[^"]*"         # 0 or more characters unequal to '"' enclosed by '"'

Read about FPAT in the gawk manual

Now, walking you through your script. Basically it tries to rewrite what your fields look like. At first, you split by ",", which obviously causes some problems. Next, it looks for fields that are not properly closed by a '"'.

BEGIN{OFS=FS =","}                        # set field sep (FS) and output field 
                                          #   sep to ,
/"/{                                      # for each line matching '"'
    for(i=1;i<=NF;i++){                   # loop through fields 1 to NF
        if($i ~ /^"[^"]+$/){              # IF field $i start with '"', followed by
                                          #   non-quotes
            for(x=i+1;x<=NF;x++){         # loop through ALL following fields
                $i=$i","$x                # concatenate field $i with ALL following 
                                          #   fields, separated by ","
                if($i ~ /"+$/){           # IF field $i ends with '"'
                    z = x - (i + 1) + 1   # z is index of field we're looking at next
                    for(y=i+1;y<=NF;y++)  
                        $y = $(y + z)     # change contents of following fields to 
                                          #   contents of field, z steps further
                                          #   down the line
                    break                 # break out of for(x) loop
                }
            }
            NF = NF - z                   # reset number of fields
            i=x                           # continue loop for(i) at index x
        }
    }
 print $1,$3,$4
}

You script fails on this input line:

"Smith \"Jr.\", Jane",35,,555-876-1233,"F",

simply because $i ~ /^"[^"]+$/ fails on $1.

I hope you agree with me that rewriting the fields like this can be tricky. More than that, it's like "O, I like awk, but I'm going to use it like C/perl/python." Using FPAT is a shorter solution, to say the least.

I made some mistakes hopefully corrected now.

awk '{sub(/y",""/,"y\42")sub(/,2.|,3./,"")sub(/,".",.*/,"")}1' file

"John \"Super\" Doe","123 ABC Street",123-456-7890
"Jane, Mary",132 CBS Street,333-111-5332
"Smith \"Jr.\", Jane",,555-876-1233
"Lee, Jack",123 Sesame St,""