0

I have log data in 7 separate files which I need to merge into one file before I can conduct any page analysis on it. Below I have provided an example of a line within one of the log data files

207.46.13.124 - - [01/Jun/2015:00:00:04 +0000] "GET /maritime/collections/hmsconway/hiscock/index.aspx HTTP/1.0" 200 "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

The names of the files are u_ex150601, u_ex150602, u_ex150603, u_ex1506014, u_ex150605, u_ex150606, and u_ex150607.

How do I merge all of these together using just Python?

0

1 Answer 1

3

I would personally recommend using bash. Something like,

cat u_ex15* >> all_logs

But in python you can simply use the following:

import glob
all_log_filename = 'all_logs'
log_files = glob.glob('u_ex15*')
for filename in log_files:
    with open(filename) as reader:
        with open(all_log_filename, 'a') as writer:
             writer.write(reader.read())

NOTE: This is the simplest case where your log files aren't that large. If your log files are large, you may want to iterate over each line or chunks so that you don't run into memory issues. The simplest thing you could do is something like:

for filename in log_files:
    with open(filename) as reader:
        with open(all_log_filename, 'a') as writer:
            for line in reader:
                writer.write(line)
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.