3

I have a create a python script to clean a csv file. code in the script file "CleanCSV.py" as below

import csv

filepath_i = 'C:\Source Files\Data Source\Flat File Source\PatientRecords.csv'
filepath_o = 'C:\Python\PatientRecords.csv'
rows = []
with open(filepath_i, 'r', newline='') as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',', quotechar='"')
    with open(filepath_o, 'w', newline='' ) as writeFile:
        writer = csv.writer(writeFile, lineterminator='\r')
        for row in csv_reader:
            row[3] = row[3].replace("\n","").replace("\r","") 
            rows.append(row) 
        writer.writerows(rows)

This is working fine when ran from python editor. but not creating file when ran from command line like below.

C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64>python C:\Python\CleanCSV.py 

I tried this as well

C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64>pythonw C:\Python\CleanCSV.pyw 

I provided full access to folder but still its not creating any file at destination. Please let me know if I am missing anything.

Also please suggest if this code can be optimized. I can't use external packages like pandas so I did it with csv. Thanks in advance.

Extension

When I changed to x for write setting

with open(filepath_o, 'x', newline='' ) as writeFile:

to my surprise I got this error

File "CleanCSV.py", line 8, in <module>
    with open(filepath_o, 'x', newline='' ) as writeFile:
FileExistsError: [Errno 17] File exists: 'C:\\Python\\PatientRecords1.csv'

but I don't see the file in the directory. even after setting hidden files to true. So I ran this script.

from pathlib import Path
config = Path(filepath_o )
if config.is_file():
    print('yes')
    print(config)
else:
    print('no')

got this ouptut, but there is no file in the directory!! puzzled.

yes
C:\Python\PatientRecords1.csv

Extension 2

Rewrote script to try with directories

with open(filepath_i,'r') as csv_file:
    csv_reader = csv.DictReader(csv_file, delimiter=',', quotechar='"')

    with open('PatientRecords1.csv', 'w') as writeFile:
        fieldnames = ['DRG Definition','Provider Id','Provider Name','Provider Street Address','Provider City','Provider State','Provider Zip Code','Hospital Referral Region Description','Hospital Category','Hospital Type', 'Total Discharges' ,'Covered Charges' , 'Total Payments' ,'Medicare Payments']
        writer = csv.DictWriter(writeFile,fieldnames=fieldnames)
        for row in csv_reader:
            row['Provider Street Address'] = row['Provider Street Address'].replace("\n","").replace("\r","") 
            writer.writerows(row)

But received this error

Traceback (most recent call last):
  File "CleanCSV.py", line 36, in <module>
    writer.writerows(row)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\csv.py", line 158, in writerows
    return self.writer.writerows(map(self._dict_to_list, rowdicts))
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\csv.py", line 148, in _dict_to_list
    wrong_fields = rowdict.keys() - self.fieldnames
AttributeError: 'str' object has no attribute 'keys'

Sample input file data

DRG Definition,Provider Id,Provider Name,Provider Street Address,Provider City,Provider State,Provider Zip Code,Hospital Referral Region Description,Hospital Category,Hospital Type, Total Discharges ,Covered Charges , Total Payments ,Medicare Payments
039 - EXTRACRANIAL PROCEDURES W/O CC/MCC,10001,SOUTHEAST ALABAMA MEDICAL CENTER,1108 ROSS CLARK CIRCLE,DOTHAN,AL,36301,AL - Dothan,Specialty Centers,Government Funded,91,"$32,963.07 ","$5,777.24 ","$4,763.73 "
039 - EXTRACRANIAL PROCEDURES W/O CC/MCC,10005,MARSHALL MEDICAL CENTER SOUTH,"2505 U S HIGHWAY 
431 NORTH",BOAZ,AL,35957,AL - Birmingham,Specialty Centers,Private Institution,14,"$15,131.85 ","$5,787.57 ","$4,976.71 "
039 - EXTRACRANIAL PROCEDURES W/O CC/MCC,10006,ELIZA COFFEE MEMORIAL HOSPITAL,205 MARENGO STREET,FLORENCE,AL,35631,AL - Birmingham,Rehabilitation Centers,Private Institution,24,"$37,560.37 ","$5,434.95 ","$4,453.79 "

Extension 3

Looks like file is created in the directory, I got output for this two peices of code. However I am unable to see that file, wondering why!!

with open(filepath_o,'r') as csv_file:
    csv_reader = csv.reader(csv_file, delimiter=',', quotechar='"')
    for row in csv_reader:
        print(row)

import os.path 
from os import path
print(path.exists(filepath_o ))
5
  • Why are you running from the Python install directory? Run from the actual script location. Make sure you've added Python to your PATH Commented Aug 2, 2019 at 12:02
  • should I copy python.exe to script file location 'C:\Python\PatientRecords.csv'. Commented Aug 2, 2019 at 12:11
  • 1
    No, follow this: geek-university.com/python/add-python-to-the-windows-path. This will allow you to run Python from anywhere Commented Aug 2, 2019 at 12:12
  • 1
    Once you get past your PATH problems, if you append the extensions .py and .pyw to the PATHEXT environment variable using ; as a separator (assuming they are not already there), you are stating that these file types should be considered executable. You should than be able to execute a python file as in the the second case you tried above (i.e. just entering the file name without preceding it with the python command) or double-clicking on a python file it within an explorer window. For this to work, of course, there must be an association between these file types and their "openers". Commented Aug 2, 2019 at 12:31
  • sure will do that, thanks. Commented Aug 2, 2019 at 13:10

2 Answers 2

1

Go to the location where your python script is present. Click on the address bar of folder and type cmd

Then the command prompt will be launched from the script's folder location

then type in cmd

python CleanCSV.py

NOTE : You need to have python added to environment variable.


If you're using Anaconda, follow the same above mentioned steps from Anaconda Prompt.

Sign up to request clarification or add additional context in comments.

6 Comments

I did this, added env variable to python.exe path and executed from script location. but it didn't create file..
Please look into my edit. It says file exists but I don't see the file.
import os os.listdir() Does the filename shows up ?
yes, it is showing up. even print is working on that file. please see the 3rd ext to my question for code.
this is the output I got ['CleanCSV.py', 'PatientRecords.csv']
|
0

That is puzzling. csv might not like your explicit r/w/x arguments in open. For example, instead of using open(filepath_i, 'r', newline=''), try open(filepath_i, newline='').

1 Comment

thanks, I tried and got this error. File "CleanCSV.py", line 13, in <module> writer.writerows(rows) io.UnsupportedOperation: not writable

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.