Writing a file to a USB stick in linux with Python?

Question

I'm having a lot more trouble than expected writing to a file than I expected. I have a small Single Board computer running on an arm processor with the Angstrom embedded distribution. I'm writing an application for it with python. The application should detect the USB stick and then dump the file to it. The issue I'm running into is that even when my script appears to write the file correctly, and the file appears to be there using "ls" and "cat" in Linux, when I remove the USB stick and try to look at the file in Windows or another linux distro it's empty or more often not there at all. My application is multithreaded. I create /media/mymntpnt as root (directory.)

Any help is appreciated.

Here are some snippets of my code:

This part is what I use to identify the USB stick:

class MediaScanner():
    def __init__(self):
        self.lok = thread.allocate_lock()
        self.running = True
        self.started = False

    def register_cb(self,func):
        self.cb = func

    def start(self):
        if not self.started:
            thread.start_new_thread(self.scan_thread,())

    def scan_thread(self):
        self.quit = False
        self.started = True
        last_devices = []
        while self.running:
            devices = self.scan_media()
            if (devices != last_devices):
                self.cb(devices) #call the callback as its own thread
            last_devices = devices

            time.sleep(0.1)

        self.quit = True    

    def stop(self):
        self.running = False
        while(not self.quit):
            pass
        return True

    def is_running(self):
        return self.running


    def scan_media(self):
        with self.lok:
            partitionsFile = open("/proc/partitions")
            lines = partitionsFile.readlines()[2:]#Skips the header lines
            devices = []
            for line in lines:
                words = [x.strip() for x in line.split()]
                minorNumber = int(words[1])
                deviceName = words[3]
                if minorNumber % 16 == 0:
                    path = "/sys/class/block/" + deviceName
                    if os.path.islink(path):
                        if os.path.realpath(path).find("/usb") > 0:
                            devices.append('/dev/'+deviceName)

            partitionsFile.close()

            return devices

And here's an example of my script to write the file:

from mediascanner import *
from util import *
from database import *
from log import *

ms = MediaScanner()

devices = ms.scan_media()

print devices

if devices:
    db = TestDatabase(init_tables=False)

    data = db.get_all_test_data()



    for device in devices:
        print device
        mount_partition(get_partition(device))
        write_and_verify('/media/mymntpnt/test_'+get_log_file_name(),flatten_tests(data))
        unmount_partition()

And here are my utility functions:

def write_and_verify(f_n,data):
    f = file(f_n,'w')
    f.write(data)
    f.flush()
    f.close()
    f = file(f_n,'r')
    verified = f.read()
    f.close()
    return  verified == data and f.closed


def get_partition(dev):
    os.system('fdisk -l %s > output' % dev)
    f = file('output')
    data = f.read()
    print data
    f.close()
    return data.split('\n')[-2].split()[0].strip()

def mount_partition(partition):
    os.system('mount %s /media/mymntpnt' % partition)


def unmount_partition():
    os.system('umount /media/mymntpnt')

have you tried sync and/or pmount and pumount? Kind of just brainstorming. — unmounted
– unmounted, Commented Jul 27, 2011 at 7:30
also look at os.fdatasync (probably amounts to something like os.system('fsync ...) on linux) — unmounted
– unmounted, Commented Jul 27, 2011 at 7:38
Is the umount always successful? Check the return value. Since your app is multithreaded, it could be that the partition is busy and cannot be unmounted. Just guessing... — codeape
– codeape, Commented Jul 27, 2011 at 8:39
The issue has little to do with Python. The OS is caching the data and not writing it to the USB stick for performance issues. You have to make sure that unmounting the drive is completed successfully. — Zaur Nasibov
– Zaur Nasibov, Commented Jul 27, 2011 at 9:22

Michael Dillon · Accepted Answer · 2011-08-04 02:01:41Z

1

The mistake is in the write function which should sync the file, before closing it, as seen below. The read will always succeed because the data is already in RAM. You might want to try something different to verify it such as checking the bytecount.

def write_and_verify(f_n,data):
    f = file(f_n,'w')
    f.write(data)
    f.flush()
    os.fsync(f.fileno())
    f.close()
    f = file(f_n,'r')
    verified = f.read()
    f.close()
    return  verified == data and f.closed

If you get the file info like this finfo = os.stat(f_n) then you can compare finfo.st_size with the number of bytes that you expected to write.

answered Aug 4, 2011 at 2:01

Michael Dillon

32.5k7 gold badges76 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Writing a file to a USB stick in linux with Python?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related