How to skip header of CSV file while appending it to another CSV file using python

Question

I am receiving a CSV response from API. I want to append the received CSV response to already present CSV file in my machine. Problem i am facing is it is also appending the header from second CSV file. I want to remove that header. I have also attached screenshot of how my csv looks after appending.

screenshot

click here to see the screenshot screenshot of the response recieved I am trying this code

response = requests.get(Link)
actual_file = glob.glob(path_to_data+'\\Data\\*')
new_target_file = path_to_data+'\\Data'+'\\'+State+'_'+date+'_'+St_Id+'.csv'
# Write to .CSV
if not os.path.exists(new_target_file):
   for x in response.text.split('\n')[1:]:
       f = open(actual_file[0], "a")
       f.write(x)
   f.close()
   os.rename(actual_file[0],new_target_file)
else: 
   logging.warning("File already exist")

There isn't an obvious direct problem. Have you tried fixing the f = open() line? It does not belong in your loop. You might want to do appenedFile = actual_file[0]; with open(appendedFile) as f: for x ..... — Charles Merriam
– Charles Merriam, Commented Jun 19, 2017 at 0:54
Please provide an extract of the first several lines of a sample response (file). Although you have used slicing [1:] to skip the first line of response, you might need to skip more than one line of response. — ChuckCottrill
– ChuckCottrill, Commented Jun 19, 2017 at 1:06
@CharlesMerriam I have tried your approach . It is removing the header but it only appending 1 row to the file . here is the code response = requests.get(Link) actual_file = glob.glob(path_to_data+'\\Data\\*') new_target_file = path_to_data+'\\Data'+'\\'+State+'_'+date+'_'+St_Id+'.csv' appendedfile = actual_file[0] # Write to .CSV if not os.path.exists(new_target_file): with open(appendedfile, "a")as f: for x in response.text.split('\n')[1:]: f.write(x) — PriyalChaudhari
– PriyalChaudhari, Commented Jun 19, 2017 at 1:08
@ChuckCottrill I am uploading the screenshot in question as there is no space here — PriyalChaudhari
– PriyalChaudhari, Commented Jun 19, 2017 at 1:14
Maybe there's a newline at the beginning of the response text - try response.text.strip().split("\n")[1:] instead. — zwer
– zwer, Commented Jun 19, 2017 at 1:25

ChuckCottrill · Accepted Answer · 2017-06-19 01:48:24Z

0

The problem you have is recognizing the response, which lines to skip, and which lines to retain. You have implemented slicing to extract response lines after the first line, assuming that the actual content starts on the second line. Based upon your described symptoms, this is not (always) the case.

#fully describe header here,
header="STATION,STATION_ELEVATION,LATITUDE,LONGITUDE,..."

def isheader(line,header,delim=','):
    l = line.split(delim) #may want to fold case
    h = header.split(delim) #may want to fold case
    n = sum(list(set(l) & set(h)))
    return n==len(h)

response = requests.get(Link)
actual_file = glob.glob(path_to_data+'\\Data\\*')
new_target_file = path_to_data+'\\Data'+'\\'+State+'_'+date+'_'+St_Id+'.csv'
# Write to .CSV
if not os.path.exists(new_target_file):
    with open(actual_file[0], "a") as f:
        for x in response.text.split('\n')[1:]:
            if len(x) < 2: continue #skip empty lines
            if isheader(x,header,','): continue #skip header
            f.write(x+'\n')
    #with performs close automatically
    os.rename(actual_file[0],new_target_file)
else:
    logging.warning("File already exist")

Take a look at this question for how to use with open, How to open a file using the open with statement

Here is an example of how to compare two lists (such as a header list to a row list), Comparing two lists in Python

answered Jun 19, 2017 at 1:48

ChuckCottrill

4,4792 gold badges30 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

PriyalChaudhari Over a year ago

f.write(x+'\n') this alone solved the problem in my code. I will try your approch as well. Thank you for response.

Collectives™ on Stack Overflow

How to skip header of CSV file while appending it to another CSV file using python

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related