Practical use of PyVarObject, the variable-length subtype of cpython’s PyObject

Question

As someone who has been putting the Python C-API through its paces of late, I am curious about PyVarObject, which the documentation says is a subtype of PyObject used specifically for the definition of variable-length types – and though the caveat that:

This type does not often appear in the Python/C API

… there are an abundance of documented support structures, slot hooks, and other API minutiae that are dedicated to PyVarObject, to the point where I am genuinely curious if there are any Python-API-related problems (however specifically niche) for which it can notably ease the solution.

Your question is not completely clear or too broad. An example would be list. Personally I used it once I think. — tynn
– tynn, Commented Apr 23, 2016 at 15:46

DavidW · Accepted Answer · 2016-04-24 13:04:25Z

2

It's mostly for immutable container types (e.g. tuple and bytes). The idea is that the type can have an array at the end of it that contains all its data. When it is allocated (with PyObject_NewVar or similar) extra space is allocated for the (immutable) data it will contain, which simplifies memory management and presumably gains a bit of speed by only having to do a single memory allocation. Because the size never changes after construction it can take this shortcut without having to worry about invalidating references to it by reallocating the memory.

To confuse things slightly list is also a PyVarObject despite being mutable. However, it has an item size of 0 and so no extra space is actually allocated at the end of the object. Instead it stores a pointer to an array of PyObject*, allocated separately (which can be reallocated if the list grows without invalidating references to the list). Presumably it was made a PyVarObject just to have an ob_size field.

answered Apr 24, 2016 at 13:04

DavidW

31.2k7 gold badges64 silver badges99 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

fish2000 Over a year ago

Aha, interesting – so would you say that it’s more CPython implementation detail than useful PyObject subtype?

DavidW Over a year ago

Somewhere between. It certainly is a CPython implementation detail. But it does provide a small benefit for certain container types, so if you were implementing something like a (non-resizable) numpy array, you might well chose to use it yourself. I don't think anything it would be a huge loss not to have it though.

fish2000 Over a year ago

If I understand you correctly, the storage for a PyVarObject* op is everything after the PyObject_HEAD stuff in the memory region from the object pointer to e.g. op + op->ob_size, yes? … which OK wow, I find that personally very funny as I am currently working on exactly that sort of thing – I’m writing a Python extension in C++ to replace PIL for simple image I/O (backed by immutable arrays). It might be good to use a PyVarObject for implementing, like, a hyperslab selection in a Pythonic way (as an array index or somesuch). Neat!

DavidW Over a year ago

Yes - that's substantially it. The extra space allocated is op->ob_size times the tp_itemsize field of the TypeObject. Just remember that you can't resize it after you've allocated it.

fish2000 Over a year ago

Awesome. That works for me… I like to do allocations like Louis Khan did concrete, boundary-aligned and fearsomely immutable. Thanks for the cogent breakdown @DavidW – I did not expect PyVarObject to have such usefulness on offer.

Collectives™ on Stack Overflow

Practical use of PyVarObject, the variable-length subtype of cpython’s PyObject

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related