How to merge multiple files in Python

Question

Very simple question! I want to merge multiple JSON files and the

Check this out:

f1data = f2data = f3data = f4data = f5data = f6data = "" 
 
with open('1.json') as f1: 
  f1data = f1.read() 
with open('2.json') as f2: 
  f2data = f2.read() 
with open('3.json') as f3: 
  f3data = f3.read() 
with open('4.json') as f4: 
  f4data = f4.read() 
with open('5.json') as f5: 
  f5data = f5.read() 
with open('6.json') as f6: 
  f6data = f6.read() 
 
f1data += "\n"
f1data += f2data += f3data += f4data += f5data += f6data 
with open ('merged.json', 'a') as f3: 
  f3.write(f1data)

And the output should be like this:

[
    {
        "id": "1",
        "name": "John",
    },

    {
        "id": "2",
        "name": "Tom",
    }
]

The problem is that Visual Code Studio brings up a red line under: f1data += f2data += f3data += f4data += f5data += f6data

I have no idea why! And the code can't run! there is no error so i can troubleshoot..Any advice?

I would suggest to rename the post as "how to merge multiple Json files in Python" to be more clear and facilitate search. Adding json in tags as well. — Osamoele
– Osamoele, Commented Jul 25, 2021 at 9:16

Osamoele · Accepted Answer · 2021-07-25 09:12:16Z

2

There is several points to improve in this code:

You should consider doint it in a more "programatic" way:

if you declare a list with the names of the json files you want to access like this:

files_names = ["1","2","3","4","5","6"]

you can then do:

files_names = ["1","2","3","4","5","6"]
data = ""
for file_name in files_names :
   with open(file_name+".json" ,"r") as file_handle :
      temp_data = file_handle.read()
      data = data + temp_data 

with open ('merged.json', 'a') as file_handle : 
  file_handle.write(data)

which is more concise, more pythonic, and can be adapted easily if you ever need 7 json input files for example.

If your files are always 1,2,3,4, you can also just do the for loop iteration like this: knowing the highest json file number you want

max_file_name = 6
for file_name in range(1,max_file_name):#i added 1 as first arg of 
#range, assuming your files naming start at 1 and not at 0
   str(file_name) + ".json"

To be sure your Json is a valid json, you could use the json standard library. It will take a little bit more time though, as the file will be parsed instead of just dumped into another one, but if you do not have 100000 files to merge and you don't know for sure the code that creates your json files in the first plase is valid, you souldn't see the difference

To use it, just do

import json
max_file_name = 6
data = ""
for file_name in range(max_file_name):
   with open(str(file_name) + ".json" ,"r") as file_handle :
      temp_data = json.load(file_handle )
      data = {**data , **temp_data}
      # ** is used to "unload" every key value in a dict at runtime, 
      # as if you provided them one by one separated by comas :
      # data["key1"],data["key2"]...
      # doing so for both json objects and puttin them 
      # into a dictionnay is effectively just like merging them.

with open ('merged.json', 'a') as file_handle : 
  json.dump(data,file_handle)

edited Jul 25, 2021 at 9:12

answered Jul 25, 2021 at 8:49

Osamoele

5161 gold badge7 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Alon Gadot Over a year ago

I would also recommend using the standard library's json module, rather than working directly with strings.

Osamoele Over a year ago

@AlonGadot I added your suggestion as a point of improvement of Ivy Knm's actual code.

David Meu · Accepted Answer · 2021-07-25 08:56:36Z

1

You have several ways:

with open('a', 'w') as a, open('b', 'w') as b, ..:
    do_something()

files_list = ['a', 'b', ..]
for file in files_list:
    with open(file, 'w')...

Use contextlib.ExitStack

with ExitStack() as stack:
    files = [stack.enter_context(open(fname)) for fname in filenames]
    # Do something with "files"

edited Jul 25, 2021 at 8:56

answered Jul 25, 2021 at 8:51

David Meu

1,5309 silver badges19 bronze badges

Comments

user2668284 · Accepted Answer · 2021-07-25 09:15:26Z

1

The output to the 'merge' file won't be formatted exactly as you've specified but this shows the approach that I would personally use:-

import json
alist = []
with open('/Users/andy/merged.json', 'w') as outfile:
    for k in range(6):
        with open(f'/Users/andy/{k+1}.json') as infile:
            alist.append(json.load(infile))
    outfile.write(str(alist))

answered Jul 25, 2021 at 9:15

user2668284

Collectives™ on Stack Overflow

How to merge multiple files in Python

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related