Storing data structure (set) as a file but not loading it into memory

Your best bet is to store it to a database. MongoDB handles sets well. Then you can just query the database like you do a set.

you have to install

sudo apt install mongodb-server-core
pip3 install pymongo
And create a /data/db directory on your drive with the right permissions then run
mongod &
before this code will work:

from pymongo import MongoClient
client = MongoClient()

client = MongoClient('localhost', 27017)

#client = MongoClient('mongodb://localhost:27017')


db = client.pymongo_test

posts = db.posts
post_data = {
    'title': 'Python and MongoDB',
    'content': 'PyMongo is fun, you guys',
    'author': 'Bill'
}
result = posts.insert_one(post_data)
print('One post: {0}'.format(result.inserted_id))

bills_post = posts.find_one({'author': 'Bill'})
print(bills_post)

One post: 5dc61c0cc2b75ebc458da31f
{'_id': ObjectId('5dc61bf76071bde943ca262b'), 'title': 'Python and MongoDB', 'content': 'PyMongo is fun, you guys', 'author': 'Bill'}
``

How about this as an interim approach until you find a DB or indexed file solution.

1. Divide the keyword file into multiple files(37), based on the character the keyword starts with. e.g.

   • keys_startwith_a.txt /contains all values starting with 'A' or 'a' ...
   • keys_startwith_z.txt /contains all values starting with 'Z' or 'z'

   • keys_startwith_0.txt /contains all values starting with '0'

     ...

   • keys_startwith_9.txt /contains all values starting with '9'
   • keys_startwith_others.txt /contains all values starting with any other char

2. change the read mechanism for each file as stream. e.g.

io.read(file, buffer=1)

Now if you want to compare values, you just have to check the char the key starts with and compare against the values in corresponding file.

With this

• Your file loading is faster(each file footprint should be significantly reduced) and on demand(stream) i.e. only when required
• comparison is faster since check is against less no of values.
• since you have unlimited memory, even if all the files are loaded during comparison, it . won't create issues.
• Finally, you can perhaps use threads for even faster loading of files and can pass on the comparison tasks to multiple threads(if required)