Execute python program with multiple files - Python - Bash

Question

I have hundreds of XML files and I would like to parse it into CSV files. I already code this program. To execute the python program I use this command (on VScode MS):

python ConvertXMLtoCSV.py -i Alarm120.xml -o Alarm120.csv

My question is, how change this script to integrate a sort of for loop to execute this program for each xml files ?

UPDATE

If my files and folders are organized like in the picture:

I tried this and execute the file .bat in windows10 but it does nothing:

#!/bin/bash
for xml_file in XML_Files/*.xml
do
   csv_file=${xml_file/.xml/.csv}
   python ConvertXMLtoCSV.py -i XML_Files/$xml_file -o CSV_Files/$csv_file
done

alani · Accepted Answer · 2020-10-04 11:51:36Z

2

If you want to keep your Python script as-is (process one file), and add the looping externally in bash, you could do:

#!/bin/bash
for xml_file in *.xml
do
   csv_file=${xml_file/.xml/.csv}
   python ConvertXMLtoCSV.py -i $xml_file -o $csv_file
done

answered Oct 4, 2020 at 11:51

alani

13.2k3 gold badges18 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Zebra125 Over a year ago

Thanks for your help, but I tried it again. Can you analyze my update please.

alani Over a year ago

You tagged bash so I wrote a bash script, assuming that you had already done what is necessary to run bash under windows. If this is not the case then you might need to install CygWin or similar. It is not a script that you can run as a Windows .bat file. If you can't run bash under Windows, then you can put your looping in the Python instead, as shown in the other answer.

Zebra125 Over a year ago

When I execute the code on cygwin I have "./Alarms_Script.sh: line 5: /cygdrive/c/Users/antho/AppData/Local/Microsoft/WindowsApps/python: Permission denied"

alani Over a year ago

@ThonyNadhir Sorry I don't know why you are getting permission denied when trying to run Python via CygWin. It looks like it might be necessary to install Python under Cygwin - wiki.usask.ca/display/MESH/… . Again, I was assuming from the fact that you tagged both bash and python that you had already got a setup in which these two would interoperate. However, if this is not the case then you do not need to use bash, because Python can do the looping for you, and what's in the other answer should be sufficient information on how to do that.

alani Over a year ago

@ThonyNadhir I guess that you do also have an option of writing an external script to do the looping (so as to leave your ConvertXMLtoCSV.py unchanged), but writing that script in python. I could provide an example of this as another answer if you want, but before I do that, please can you confirm whether in python you can do import os followed by os.system("python -c 'print(2+2)'") - it should print 4 (possibly followed by 0). This will test whether python can call python as an external process. If you can first confirm this, then I can provide an example of the full script.

|

Rolv Apneseth · Accepted Answer · 2020-10-04 11:57:48Z

2

Ideally the for loop would be included inside your ConvertXMLtoCSV.py itself. You can use this to find all xml files in a given directory:

for file in os.listdir(directory_path):
    if file.endswith(".xml"):
        # And here you can do your conversion

You could change the arguments given to the script to be the path of the directory the xml files are located in and the path for an output folder for the .csv files. For renaming, you can leave the files with the same name but give the .csv extension. i.e.

csv_name = file.replace(".xml", ".csv")

answered Oct 4, 2020 at 11:57

Rolv Apneseth

2,1282 gold badges10 silver badges22 bronze badges

Comments

alani · Accepted Answer · 2020-10-06 10:31:25Z

After discussion, it appears that you wish to use an external script so as to leave the original ConvertXMLtoCSV.py script unmodified (as required by other projects), but that although you tagged bash in the question, it turned out that you were not in fact able to use bash to invoke python when you tried it in your setup.

This being the case, it is possible to adapt Rolv Apneseth's answer so that you do the looping in Python, but inside a separate script (let's suppose that this is called convert_all.py), which then runs the unmodified ConvertXMLtoCSV.py as an external process. This way, the ConvertXMLtoCSV.py will still be set up to process only one file each time it is run.

To call an external process, you could either use os.system or subprocess.Popen, so here are two options.

Using os.system:

import os
import sys

directory_path = sys.argv[1]

for file in os.listdir(directory_path):
    if file.endswith(".xml"):
        csv_name = file.replace(".xml", ".csv")
        os.system(f'python ConvertXMLtoCSV.py -i {file} -o {csv_name}')

note: for versions of python too old to support f-strings, that last line could be changed to

        os.system('python ConvertXMLtoCSV.py -i {} -o {}'.format(file,csv_name))

Using subprocess.Popen:

import subprocess
import sys

directory_path = sys.argv[1]

for file in os.listdir(directory_path):
    if file.endswith(".xml"):
        csv_name = file.replace(".xml", ".csv")
        p = subprocess.Popen(['python', 'ConvertXMLtoCSV.py',
                              '-i', file,
                              '-o', csv_name])
        p.wait()

You could then run it using some command such as:

python convert_all.py C:/Users/myuser/Desktop/myfolder

or whatever the folder is where you have the XML files.

Collectives™ on Stack Overflow

Execute python program with multiple files - Python - Bash

3 Answers 3

8 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

8 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related