Write data to multiple files using multiple threads

Question

I want to create multiple file using multiple threads, and append data (after some operation is performed) to corresponding files from their respective threads.

I tried it, but data is getting messed between threads and correct data is not added to respective files.

import threading
import time

exitFlag = 0

class myThread (threading.Thread):
   def __init__(self, threadID, name, counter):
      threading.Thread.__init__(self)
      self.threadID = threadID
      self.name = name
      self.counter = counter
   def run(self):
      with open('file_'+count+'_logs.txt', 'a+') as result:
            result.write("Starting " + self.name)
            result.write("Exiting " + self.name)
      print ("Starting " + self.name)
      print_time(self.name, self.counter, 5)
      print ("Exiting " + self.name)

def print_time(threadName, delay, counter):
   while counter:
      if exitFlag:
         threadName.exit()
      time.sleep(delay)
      print ("%s: %s" % (threadName, time.ctime(time.time())))
      counter -= 1

myList = ['string0', 'string1', 'string2', 'string3']

if __name__ == "__main__":
    count = 0
    for data in myList:
        count += 1
        mythread = myThread(count, "Thread-" + str(count), count)
        mythread.start()
        mythread.join()

I expect that 4 files shall be created from 4 threads, and data from thread 1 should be written to file_1_logs.txt and so on... But while writing data, sometimes all data is written in a single file. How do I write this data to file correctly?

Your variable count does not exist inside the thread. Could it be that you mean self.counter? — Tomalak
– Tomalak, Commented Jul 22, 2019 at 12:00
count exists in the main, and is argument of the constructor — atg
– atg, Commented Jul 22, 2019 at 12:02
I know, but it does not exist in the thread. The thread does run separately from the main. Don't use global (or higher-scope) variables when you are working with threads. — Tomalak
– Tomalak, Commented Jul 22, 2019 at 12:06
It is a constraint, that I have to use higher scope variables... I get data from other function that controls my thread :( — atg
– atg, Commented Jul 22, 2019 at 12:11

Tomalak · Accepted Answer · 2019-07-22 15:03:22Z

Don't use higher-scope or global variables in threads. Every variable (that you want to modify) must be local to the thread.

That means you need to pass the initial values of everything to the Thread constructor. The same is true for functions like your print_time. Either everything the function needs to do its job is passed via the arguments – or you turn it into a class method.

Consider the following changes. Note how MyThread is completely self-contained:

from threading import Thread
from time import sleep
from datetime import datetime

class MyThread(Thread):
    def __init__(self, threadID, name, delay, data):
        Thread.__init__(self)
        self.threadID = threadID
        self.name = name
        self.data = data
        self.delay = delay
        self.logfile = None

    def run(self):
        with open('file_%s_logs.txt' % self.threadID, 'a+') as logfile:
            self.logfile = logfile
            self.log("Starting")
            self.print_time(5)
            self.log("Exiting")
            self.logfile = None

    def print_time(self, repeat):
        for c in range(repeat):
            sleep(self.delay)
            self.log(self.data)

    def log(self, message):
        now = datetime.now().isoformat()
        formatted_line = "%s:%s:%s" % (now, self.name, message)
        print(formatted_line)
        if self.logfile:
            self.logfile.write(formatted_line + '\n')


if __name__ == "__main__":
    myList = ['string0', 'string1', 'string2', 'string3']
    threads = []

    # spawn threads
    for idx, data in enumerate(myList):
        thread = MyThread(idx, "Thread-%s" % idx, idx, data)
        threads.append(thread)
        thread.start()

    # wait for threads to finish
    while True:
        if any(thread.is_alive() for thread in threads):
            sleep(0.1)
        else:
            print("All done.")
            break

Collectives™ on Stack Overflow

Write data to multiple files using multiple threads

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related