How to write two CSV concurrently in Python?

Question

I have two radios, sdr and sdr2, receiving data and I want to save that date (which are complex numbers) in a CSV file. I need to get the data from both radios at the same time,running a scan 5 times on each, so what I do in the main part of my code is:

#we save the sdr and sdr2 in the same array
radios = [ sdr, sdr2]
pool = ThreadPool(4)
#create an object of class Scan
s=Scan()
#pool.map(function, array)
pool.map(s.scan, radios)
pool.close() 
pool.join()

Then, the Scan function is:

class Scan: 
    def scan(self, object):   
      for i in range(0,1):
        #Read iq data
        samples = object.read_samples(256*1024)
        #print(samples)

       #get the maximum amplitude in frequency to save IQ samples if it's greater
       #than -1 dB
        sp = np.fft.fft(samples)
        ps_real=sp.real
        ps_imag=sp.imag
        sq=np.power(ps_real,2)+np.power(ps_imag,2)
        sqrt=np.sqrt(sq)
        psd=sqrt/1024
        value=[ps_real,ps_imag]
        max=np.max(psd)
        log=10*math.log10(max)
        print(value)
        current_time = time.strftime("%m.%d.%y-%H%M.csv", time.localtime())
        if log > -1:
            #save the IQ data in csv
            with open('%s' % current_time, 'w',newline='') as f:
                writer = csv.writer(f, delimiter=',')
                writer.writerows(zip(ps_real,ps_imag))

but what this does is getting the array (real,imag pairs) from the last iteration of one of the radios (I think it's only one) and save it in a unique CSV...I'd like to have 2 different CSVs, that's why I put the timestamp in the CSV name and I also need to record the data from any iteration. Any idea on how to fix this? Thanks!

Can you put the filename as a member of self and write to that csv file? — cup
– cup, Commented Jan 3, 2018 at 8:32

Netwave · Accepted Answer · 2018-01-03 09:53:17Z

1

You are opening the outfile in the same day and same hour and minute so you are writing to the same file in both jobs, just make the function use a id and pass it as argument:

class Scan: 
    def scan(self, id, object):
        ...
        current_time = time.strftime("%m.%d.%y-%H%M", time.localtime())
        if log > -1:
            #save the IQ data in csv
            with open('{}_{}.csv' .format(current_time, id), 'w',newline='') as f:
                ...

And then use a wrapper to unpack the ids from an enumerate to the radios when mapping it in the thread pool:

#we save the sdr and sdr2 in the same array
radios = [ sdr, sdr2]
pool = ThreadPool(4)
#create an object of class Scan
s=Scan()

def scan(args_tuple):
    global s
    id, code = args_tuple
    return s.scan(id, code)

pool.map(scan, enumerate(radios))
pool.close() 
pool.join()

edited Jan 3, 2018 at 9:53

answered Jan 3, 2018 at 8:36

Netwave

43.2k6 gold badges69 silver badges111 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Contact-us Behind The Sciences Over a year ago

Thanks a lot! I just don't understand where you place the second piece of code...would this go inside the class Scan? and the the pool.map inside the main or inside the class Scan? Because I don't see how you call the scan from the main. Thank you!

Contact-us Behind The Sciences Over a year ago

Thank you :) the code doesn't give any error but I get only a file: "01.03.18-1043.csv_0" and when I edit it to "01.03.18-1043.csv" and open it, I see the samples of the last iteration, I think only from one radio...

Netwave Over a year ago

@Contact-usBehindTheSciences, just realized you were using the '.csv' in the current time formatiing, now should be name it properly. Anyway, try now and compare both files, because you shold be getting 2 files, one for each sdr

Contact-us Behind The Sciences Over a year ago

it's working now :) thank you so much!! I tried to upvote your answer but I haven't got enough reputation, so I marked it as the answer :D

Collectives™ on Stack Overflow

How to write two CSV concurrently in Python?

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related