Remote database access on python

Question

I have a large data set (185GB) that I plan to perform some machine learning algorithms on it. The data is on a local computer with restricted computational powers. I have access to a remote cluster where I can perform my computationally expensive algorithms. It has 1TB of memory and is pretty fast. But for some reasons I only have 2GB(!) of disk storage on the remote server.

I can connect to the cluster via SSH, is there any way on python that I can load the database to the RAM via SSH?

Any general tips on how to tackle this problem is much appreciated.

Maybe this is what you are looking for?

Ali
– Ali

2015-11-05 03:50:00 +00:00
Commented Nov 5, 2015 at 3:50 — Ali
– Ali, Commented Nov 5, 2015 at 3:50

CleverLikeAnOx · Accepted Answer · 2015-11-05 02:11:14Z

1

You may want to use paramiko so that you can connect with SSH from within Python. Then, you can run commands that output your data and read it from the stream. This would work better than copying the files over because it wont involve copying the data to disk. If the data is in files, then you can just use paramiko to cat the files and read the data from the stream.

answered Nov 5, 2015 at 2:11

CleverLikeAnOx

1,4961 gold badge10 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Remote database access on python

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related