Force compiler when running python setup.py install

Question

Is there a way to explicitly force the compiler for building Cython extensions when running python setup.py install? Where setup.py is of the form:

import os.path
import numpy as np
from setuptools import setup, find_packages, Extension
from Cython.Distutils import build_ext

setup(name='test',
  packages=find_packages(),
  cmdclass={'build_ext': build_ext},
  ext_modules = [ Extension("test.func", ["test/func.pyx"]) ],
  include_dirs=[np.get_include()]
 )

I'm trying to install a package on Windows 8.1 x64 using Anaconda 3.16, Python 3.4, setuptools 18, NumPy 1.9 and Cython 0.24. The deployment script is adapted from the Cython wiki and this Stack Overflow answer:

Makefile.bat

:: create and activate a virtual environement with conda
conda create --yes -n test_env cython setuptools=18 pywin32 libpython numpy=1.9 python=3
call activate test_env

:: activate the MS SDK compiler as explained in the Cython wiki
cd C:\Program Files\Microsoft SDKs\Windows\v7.1\
set MSSdk=1
set DISTUTILS_USE_SDK=1
@call .\Bin\SetEnv /x64 /release

cd C:\test
python setup.py install

The problem is that in this case setup.py install still used the MinGW compiler included with conda instead of the MS Windows SDK 7.1 one.

So the DISTUTILS_USE_SDK=1 and MSSdk=1 don't seem to have an impact on the build. I'm not sure if activating the MS SDK from within a conda virtualenv might be an issue here.
Running python setup.py build_ext --compiler=msvc correctly builds the extension with the MS compiler, but subsequently running the setup.py install recompiles it with MinGW again. Same applies to python setup.py build --compiler=msvc.
Also tried running %COMSPEC% /E:ON /V:ON /K "%PROGRAMFILES%\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" as discussed in the answer linked above, but for me this produces a new terminal prompt, colored in yellow, and stops the install process.

Is there a way of forcing the compiler for building this package, for instance, by editing the setup.py?

Try using python setup.py build_ext --compiler=msvc --force to force the compiler to build/rebuild the package and see what happens. — romeric
– romeric, Commented Oct 25, 2015 at 16:24
@romeric So the build_ext builds the extension with MS VC as expected (both with and without the --force flag). The problem is that when I then run the install it gets rebuilt with mingw. — rth
– rth, Commented Oct 25, 2015 at 16:39

PeterE · Accepted Answer · 2015-10-27 16:41:34Z

24

+250

You can provide (default) command line arguments for distutils in a separate file called setup.cfg (placed parallel to your setup.py). See the docs for more information. To set the compiler use something like:

[build]
compiler=msvc

Now calling python setup.py build is equivalent to calling python setup.py build --compiler=msvc. (You can still direct distutils to use an other complier by calling python setup.py build --compiler=someothercompiler)

Now you have (successfully directed distutils to use a msvc compiler. Unfortunately there is no option to tell it which msvc compiler to use. Basically there are two options:

One: Do nothing and distutils will try to locate vcvarsall.bat and use that to setup an environment. vcvarsall.bat (and the compiler it sets the environment up for) are part of Visual Studio, so you have to have installed that for it to work.

Two: Install the Windows SDK and tell distutils to use that. Be aware that the name DISUTILS_USE_SDK is rather missleading (at least in my opinion). It does NOT in fact tell distutils to use the SDK (and it's setenv.bat) to setup an environment, rather it means that distutils should assume the environment has already been set up. That is why you have to use some kind of Makefile.bat as you have shown in the OP.

Side Note: The specific version of VisualStudio or the Windows SDK depends on the targeted python version.

answered Oct 27, 2015 at 16:41

PeterE

5,8656 gold badges34 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

rth Over a year ago

Thanks your answer, specifying the compiler in setup.cfg does solve the issue on Windows. The problem is that simply adding this file also breaks the install on Linux which should use the gcc complier and not MS VC. Of course it is possibly to generate the setup.cfg on the runtime, but I'm wondering if it's possible to hack the compiler option through setup.py which would be more flexible ...

PeterE Over a year ago

As an afterthought: I Suppose, that you could hack distutils to do to do you wish, we are talking about Python here after all. The point to target would probably be the build_ext command, as in recreate a modified version or derive a modify the existing one. It is my impression however that the entire distutils is rather complex (which probably stems from the nature of the thing, and wasn't avoidable) and thus difficult to understand at first glance. The MS compiler setup is rather convoluted and hasn't been touched often for fear of breaking backward compatibility (My opinion and guess)

Fred Schoen Over a year ago

setup.cfg is a file that the user has complete control of. You should not distribute it with your code.

Fred Schoen · Accepted Answer · 2017-05-30 11:36:43Z

3

As a remark: on linux, you can use many of the autoconf environment variables. For the compiler

CC=mpicc python setup.py build_ext -i

answered May 30, 2017 at 11:36

Fred Schoen

1,45215 silver badges18 bronze badges

Comments

B.Bocquet · Accepted Answer · 2018-09-13 21:48:47Z

2

Well I found a trick in my case : I wanted to use MSVC14.0 (from buildtools 2015) and NOT MSVC14.1 (buildtools 2017). I edited the file Lib\distutils_msvccompiler.py. There is a method

_find_vcvarsall

which is calling

best_version, best_dir = _find_vc2017()

I replaced this call by

best_version, best_dir = _find_vc2015()

Do not forget to undo this dirty trick once compiled.

answered Sep 13, 2018 at 21:48

B.Bocquet

215 bronze badges

Comments

Guillaume · Accepted Answer · 2020-07-22 08:20:29Z

0

I ended up putting this at the beginning of setup.py to force visual2015

useful when running in a non bat environment and starting vcvarsall from outside is not an option

if sys.platform == 'win32':
    # patch env with vcvarsall.bat from vs2015 (vc14)
    try:
        cmd = '"{}..\\..\\VC\\vcvarsall.bat" x86_amd64 >nul 2>&1 && set'.format(environ['VS140COMNTOOLS'])
        out = subprocess.check_output(cmd, stderr=subprocess.STDOUT, universal_newlines=True)
    except:
        print("Error executing {}".format(cmd))
        raise

    for key, _, value in (line.partition('=') for line in out.splitlines()):
        if key and value:
            os.environ[key] = value

    # inform setuptools that the env is already set
    os.environ['DISTUTILS_USE_SDK'] = '1'

edited Jul 22, 2020 at 8:20

answered Jul 19, 2020 at 19:21

Guillaume

1,0078 silver badges8 bronze badges

Comments

EvgenKo423 · Accepted Answer · 2022-07-20 17:57:35Z

0

There is a --skip-build option of install command which will help you with this. You can also specify multiple commands in a row, like so:

python setup.py build --compiler=msvc install --skip-build

Here is a list of supported compiler types (as returned by setup.py build_ext --help-compiler):

--compiler=bcpp     Borland C++ Compiler
--compiler=cygwin   Cygwin port of GNU C Compiler for Win32
--compiler=mingw32  Mingw32 port of GNU C Compiler for Win32
--compiler=msvc     Microsoft Visual C++
--compiler=unix     standard UNIX-style compiler

Note: To prevent compatibility issues extension modules should be built with the same compiler as the Python interpreter itself.

answered Jul 20, 2022 at 17:57

EvgenKo423

3,4002 gold badges20 silver badges28 bronze badges

Collectives™ on Stack Overflow

Force compiler when running python setup.py install

5 Answers 5

3 Comments

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

3 Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related