Limit python script RAM usage in Windows

Question

My program can allocate suddenly a lot of RAM based on usage. I want to limit the RAM it can take from the system.

I saw here: Limit RAM usage to python program

But it works only for Unix. Any solution for Windows?

Tried combining it with resource.RLIM_INFINITY? docs.python.org/3/library/resource.html#resource.RLIM_INFINITY — Sharan Arumugam
– Sharan Arumugam, Commented Mar 1, 2019 at 17:05
Rather than limiting the amount of memory you are able to utilise, I think it better for you to do the actions you want in a more efficient way that is not so consuming on the systems resources. Ramping up hardware is the worst kind of scalability. — Swift
– Swift, Commented Mar 1, 2019 at 17:44

Eryk Sun · Accepted Answer · 2019-03-02 13:14:25Z

7

A Job object supports limiting the committed memory of a process. In Python, we can implement this via PyWin32 or ctypes.

Note that prior to Windows 8 a process can only be in a single Job. A couple of common cases where this is a concern include the py.exe launcher (the default association for .py files), which runs python.exe in a Job, and the Task Scheduler service, which runs each task in a Job.

PyWin32 Example

import sys
import warnings

import winerror
import win32api
import win32job

g_hjob = None

def create_job(job_name='', breakaway='silent'):
    hjob = win32job.CreateJobObject(None, job_name)
    if breakaway:
        info = win32job.QueryInformationJobObject(hjob,
                    win32job.JobObjectExtendedLimitInformation)
        if breakaway == 'silent':
            info['BasicLimitInformation']['LimitFlags'] |= (
                win32job.JOB_OBJECT_LIMIT_SILENT_BREAKAWAY_OK)
        else:
            info['BasicLimitInformation']['LimitFlags'] |= (
                win32job.JOB_OBJECT_LIMIT_BREAKAWAY_OK)
        win32job.SetInformationJobObject(hjob,
            win32job.JobObjectExtendedLimitInformation, info)
    return hjob

def assign_job(hjob):
    global g_hjob
    hprocess = win32api.GetCurrentProcess()
    try:
        win32job.AssignProcessToJobObject(hjob, hprocess)
        g_hjob = hjob
    except win32job.error as e:
        if (e.winerror != winerror.ERROR_ACCESS_DENIED or
            sys.getwindowsversion() >= (6, 2) or
            not win32job.IsProcessInJob(hprocess, None)):
            raise
        warnings.warn('The process is already in a job. Nested jobs are not '
            'supported prior to Windows 8.')

def limit_memory(memory_limit):
    if g_hjob is None:
        return
    info = win32job.QueryInformationJobObject(g_hjob,
                win32job.JobObjectExtendedLimitInformation)
    info['ProcessMemoryLimit'] = memory_limit
    info['BasicLimitInformation']['LimitFlags'] |= (
        win32job.JOB_OBJECT_LIMIT_PROCESS_MEMORY)
    win32job.SetInformationJobObject(g_hjob,
        win32job.JobObjectExtendedLimitInformation, info)

def main():
    assign_job(create_job())
    memory_limit = 100 * 1024 * 1024 # 100 MiB
    limit_memory(memory_limit)
    try:
        bytearray(memory_limit)
    except MemoryError:
        print('Success: available memory is limited.')
    else:
        print('Failure: available memory is not limited.')
    return 0

if __name__ == '__main__':
    sys.exit(main())

edited Mar 2, 2019 at 13:14

answered Mar 2, 2019 at 12:58

Eryk Sun

34.5k7 gold badges97 silver badges113 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Lilianna Over a year ago

Thank you for this answer! However, I still don't understand how to use this code, say, I have a function run_pipeline() for which I want to set the RAM limit, where do I put run_pipeline() in this code?

Eryk Sun Over a year ago

@Lilianna, the process is assigned to a Job object via assign_job(create_job()). Then limit_memory(memory_limit) configures a memory limit that applies to all processes in the job. Subsequently, calling your run_pipeline function will be subject to the memory limit. Note that by default the job is created in silent breakaway mode, so only the current process will be in the job, i.e. you don't have to worry about child processes being affected by this limit.

Talha Anwar Over a year ago

any idea for this error: Assertion failed: error not defined [8] (C:\ci\zeromq_1599741307349\work\src\ip.cpp:481) I got this while running above code

Saaru Lindestøkke · Accepted Answer · 2024-05-01 15:32:12Z

3

I had about the same problem as the OP, except that I wanted to limit the amount of physical RAM used, not virtual. @eryksun's answer and limit_memory() function work great, but limit the total amount of allocatable memory (virtual memory). Here is an additional function to his/her code that limits the amount of physical memory (the "working set").

def limit_working_set(memory_limit):
    if g_hjob is None:
        return
    info = win32job.QueryInformationJobObject(g_hjob, win32job.JobObjectBasicLimitInformation)
    info['MinimumWorkingSetSize'] = 50 * 4096  # default minimum value
    info['MaximumWorkingSetSize'] = memory_limit
    info['LimitFlags'] = (win32job.JOB_OBJECT_LIMIT_WORKINGSET)
    win32job.SetInformationJobObject(g_hjob, win32job.JobObjectBasicLimitInformation, info)

By default this can only be done by Administrators. To enable this functionality for regular users it's possible to provide them with the needed rights through the group policy editor, as described in this procgov issue.

edited May 1, 2024 at 15:32

Saaru Lindestøkke

2,6051 gold badge33 silver badges64 bronze badges

answered May 16, 2019 at 12:29

cxxl

5,4183 gold badges36 silver badges58 bronze badges

3 Comments

Saaru Lindestøkke Over a year ago

Is there any way to run this as a non-admin? If a regular user runs the limit_working_set function they get an error pywintypes.error: (1314, 'SetInformationJobObject', 'A required privilege is not held by the client.')

cxxl Over a year ago

@Saaru: I don't see how, since this is a permission issue of the Windows API.

Saaru Lindestøkke Over a year ago

On the repository of a standalone tool that can limit the working set (procgov) I found this issue. If regular users are allowed to "Increase scheduling priority" policy it's possible for them to limit the workingset.

Collectives™ on Stack Overflow

Limit python script RAM usage in Windows

2 Answers 2

PyWin32 Example

3 Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

PyWin32 Example

3 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related