Scope of Global variables in Python in Windows

Question

I have a CLI script which I am using to push files into and s3 bucket. For larger files i am splitting the files into parts and uploading them in parallel. (Pasting code structure here. I tried to make a minimalist example but even that is 60 lines long)

def _upload_part(argFile, argBucket, max_attempts_limit, **core_chunk):
    #bunch of stuff
    pool = Pool(processes=parallel_processes)
    for i in range( chunk_amount ):
        #bunch of stuff
        pool.apply_async( _upload_for_multipart, [keyname, offset, mp, part_num, bytes] )
    pool.close()
    pool.join()

def _upload_for_multipart(keyname, offset, mp, part_num, bytes):
    #code to upload each part
    #log the status of each part to log files

def _get_logger( pdir, ldir, lname, level, fmt ):
    os.makedirs( logs_dir )
    logging.basicConfig(
        filename=os.path.join(logs_dir, lname),
        level=level,
        format=fmt
    )
    return logging.getLogger( lname )

#under main
if __name__ == "__main__":
    logneeds = dict( pdir=exec_dir, ldir='logs', lname='s3_cli.log', level='INFO',
                               fmt='%(asctime)s %(levelname)s: %(message)s' )
    logger = _get_logger(**logneeds)

The above code structure works in OSX and Linux but fails in windows. It says that the name 'logger' is not defined under the _upload_for_multipart function .Is there a difference between the way global variables are interpreted in windows and unix based OS?

Edit: Added working example here

@martineau - I can post the code here, but I couldnt make it any smaller than 60 lines of code... — Beginner
– Beginner, Commented Apr 24, 2015 at 17:11
Well, you should at least include the section that has the line that results in the name 'logger' is not defined error. Some folks upload longish pieces of code onto a website like pastebin or GitHub and include a url reference to it in their question. — martineau
– martineau, Commented Apr 24, 2015 at 17:25
@martineau - Pasting links to code isnt allowed on SO AFAIK and gets downvoted. People ask for a small example if code is too long — Beginner
– Beginner, Commented Apr 24, 2015 at 18:10

Community · Accepted Answer · 2017-05-23 11:58:28Z

2

What you're seeing is a difference in the way child processes get created on Windows vs. UNIX-y OSes (or any OS that supports fork). On Linux, fork is used to create the child process, which creates the child in a copy-on-write duplicate of the parent process. That means all the variables present in the parent will be present in the child.

Windows, however, does not support fork. In order to create the child, it spawns a brand-new process, and then re-imports the __main__ module of the parent in the child. In your case that means that the script running your program gets imported in the child. The creation of logger happens inside the if __name__ == "__main__": guard in the parent, which means it gets skipped when the child imports the module. There are a couple of places in the multiprocessing docs that mention this, which I explicitly list in this answer.

In order for logger to exist in the children, it needs to either get created at the top-level of the module (which would mean it runs every time you import it), or you need to explicitly create it in all the child processes, perhaps using the initializer argument to multiprocessing.Pool:

logger = None

def init_log():
    global logger
    logneeds = dict(pdir=exec_dir, ldir='logs', lname='s3_cli.log', level='INFO',
                    fmt='%(asctime)s %(levelname)s: %(message)s' )
    logger = _get_logger(**logneeds)

def _upload_part(argFile, argBucket, max_attempts_limit, **core_chunk):
    #bunch of stuff
    pool = Pool(processes=parallel_processes, initializer=init_log)
    for i in range( chunk_amount ):
        #bunch of stuff
        pool.apply_async( _upload_for_multipart, [keyname, offset, mp, part_num, bytes] )
    pool.close()
    pool.join()

def _upload_for_multipart(keyname, offset, mp, part_num, bytes):
    #code to upload each part
    #log the status of each part to log files

def _get_logger( pdir, ldir, lname, level, fmt ):
    os.makedirs( logs_dir )
    logging.basicConfig(
        filename=os.path.join(logs_dir, lname),
        level=level,
        format=fmt
    )
    return logging.getLogger( lname )

#under main
if __name__ == "__main__":
    init_log()

edited May 23, 2017 at 11:58

CommunityBot

11 silver badge

answered Apr 24, 2015 at 15:08

dano

95.5k21 gold badges234 silver badges231 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Beginner Over a year ago

Thank you for the explanation. i followed your advise and your given code and made the necessary changes in my code. However the child process throws an error saying - Nonetype object has no attribute info

dano Over a year ago

@Beginner Can you edit your question to include a complete example that reproduces the issue?

Beginner Over a year ago

- I added a link to the script after I got rid of 80% of the code.

dano Over a year ago

@Beginner The problem is here:Pool(processes=parallel_processes, initializer=init_log(logFile)). You're using initializer=init_log(logFile), which actually calls init_log in the parent process and passes the return value from that function to initializer. This means that logger isn't getting set up in the child at all. What you should be doing is just passing the init_log function to the child process, so that logger is actually set up in the child. So, you should do this instead: Pool(processes=parallel_processes, initializer=init_log, initargs=(logFile,))

Beginner Over a year ago

I had a feeling that passing initializer = ini_log(logFile) was not right. I fixed the issues. Now I run into a permissionError. I think this is a step forward

Collectives™ on Stack Overflow

Scope of Global variables in Python in Windows

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related