Using a Python Class to share data between Python scripts

Question

I am wondering if it makes sense in Python to use a class containing a list as if it is an in-memory database.

class MyData(object):
    def __init__(self):
        self._data = ['apple', 'orange', 'banana']

    @property
    def data(self):
        return self._data

    @classmethod
    def append(self, item):
        self._data.append(item)

Here's the scenario:

A first python script a.py will import this class MyData and use the data in its list.

A second python script b.py continuously poll an online API in an infinite loop, imports this same class and appends more items to the list.

When a.py runs another time, it should be able to read the newly added items in the list.

Question: Is this a good idea, or are we better off using a real database like MySQL/redis?

Update: The data does not need to be persisted to disk

I don't get how you will store state here. If a.py runs another time, it will here simply start again with the initial list. You can use code generators to alter the Python code, but that would be horrible design. — willeM_ Van Onsem
– willeM_ Van Onsem, Commented Dec 18, 2017 at 21:16
This is what databases were designed to do. You will have a much easier time using SQL. — Myles Hollowed
– Myles Hollowed, Commented Dec 18, 2017 at 21:17
Furthermore in many cases a database is meant to store data. It is done in such a way to make it easy to insert, update, remove, ... data, and run aggregates over it. Here you only have a list, so you would miss a lot of important functionality. Most databases detach data from a specific application: the database can be queried in a variety of ways each time retrieving other interesting subsets of data. — willeM_ Van Onsem
– willeM_ Van Onsem, Commented Dec 18, 2017 at 21:18
Finally whether it is stored in memory or on a disk does not depend on whether you use SQL. There are database managers for SQL that can store a database completely (or partially) in RAM memory. By default most database management systems will store a large portion of the indexes in memory. — willeM_ Van Onsem
– willeM_ Van Onsem, Commented Dec 18, 2017 at 21:20
why do you have a classmethod that expects a self argument? That's not how class methods work... — Tadhg McDonald-Jensen
– Tadhg McDonald-Jensen, Commented Dec 18, 2017 at 21:33

Kiersten Arnold · Accepted Answer · 2017-12-18 21:18:03Z

1

This will not work how you described it. a.py and b.py will be run be different python interpreters, thus not sharing memory (i.e. they won't share the class).

answered Dec 18, 2017 at 21:18

Kiersten Arnold

1,9141 gold badge13 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

BPL · Accepted Answer · 2017-12-18 21:38:59Z

If the idea is about running a.py and b.py on different python processes, you'd need to persist your data somewhere on disk, using interprocess communication or similars... My best advice? Use just your database through some of the zillion python modules allowing you to communicate to it, that's what they're made for.

Now, if the question is about sharing an object state between modules, here's a simple example which shows you how to achieve that:

storage.py:

class Storage(object):

    def __init__(self):
        self._data = ['apple', 'orange', 'banana']

    @property
    def data(self):
        return self._data

    def append(self, item):
        self._data.append(item)

Storage = MyData()

a.py:

from storage import Storage


class A():

    def __init__(self):
        self.count = 0

    def __str__(self):
        self.count += 1
        return f"read count={self.count} data={','.join(Storage.data)}"

b.py:

from storage import Storage


class B():

    def __init__(self):
        self.count = 0

    def write(self, item):
        Storage.append(item)

test.py:

from a import A
from b import B

if __name__ == "__main__":
    reader = A()
    writer = B()

    print(f"Initial state of storage {reader}")
    for i, item in enumerate(["pomegranate", "watermelon", "strawberry"]):
        writer.write(item)
        print(reader)

For more info about this topic you can check this Is there a simple, elegant way to define singletons? or similars SO threads talking about python&singletons.

Tadhg McDonald-Jensen · Accepted Answer · 2017-12-18 21:43:49Z

assuming there is nothing else that a.py needs to output, you can output the data you need to transfer by just printing it, for example:

#a.py
for i in range(6):
    print("data:{}".format(i))

Then by piping the output to b.py which can receive the input with input (raw_input with python2)

#b.py
try:
    while True:
        data = input()
        print("recieved data:", repr(data))
except EOFError: #raised when input stream is closed
    print("data stopped coming")

Then you can run them together using the | in a command prompt to pipe the data:

Tadhgs-MacBook-Pro:codes Tadhg$ python3 a.py | python3 b.py
recieved data: 'data:0'
recieved data: 'data:1'
recieved data: 'data:2'
recieved data: 'data:3'
recieved data: 'data:4'
recieved data: 'data:5'
data stopped coming

Collectives™ on Stack Overflow

Using a Python Class to share data between Python scripts

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related