Using python scipy from spark scala

Question

Is it possible to use python "scipy" linear algebra library from spark/scala script?

I need to use the rich sparse functions in the "scipy" library but my project is already in scala

Unlikely. The scipy code is written in Python with heavy use of numpy. numpy has a lot of compiled code. So does the scipy.sparse. Some of the linear algrabra stuff uses external compiled libraries, ones can be used by other code. But that's a c/c++/Fortran task. — hpaulj
– hpaulj, Commented Nov 2, 2016 at 0:56

Community · Accepted Answer · 2017-05-23 11:52:55Z

1

It's not feasible to use SciPy from Scala, because Python is no JVM language, but there seem to be ways. The closest you might get in pure Scala is to use Scalanlp-Breeze. You can check out their Comparison with Matlab and Numpy. Their is a SparseMatrix datastructure, you should look at that.

Make sure you properly install the native libraries if you want to get the full performance.

edited May 23, 2017 at 11:52

CommunityBot

11 silver badge

answered Nov 2, 2016 at 7:12

Raphael Roth

27.3k19 gold badges98 silver badges152 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2017-05-23 11:45:48Z

0

I think it is not possible to use scipy if you want to take the concurrent computing advantage of spark, because scipy is not designed for spark.

If you only want to use it in a local machine in scala script, you can try java-python-integration. Or you can use scipy-like lib in Java instead.

edited May 23, 2017 at 11:45

CommunityBot

11 silver badge

answered Nov 2, 2016 at 3:31

Joseph Guan

1064 bronze badges

Collectives™ on Stack Overflow

Using python scipy from spark scala

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related