Python: how do you store a sparse matrix using python?

Question

I have got an output using sparse matrix in python, i need to store this sparse matrix in my hard disk, how can i do it? if i should create a database then how should i do?? this is my code:

import nltk
import cPickle
import numpy
from scipy.sparse import lil_matrix
from nltk.corpus import wordnet as wn
from nltk.corpus import brown
f = open('spmatrix.pkl','wb')
def markov(L):
    count=0
    c=len(text1)
    for i in range(0,c-2):
        h=L.index(text1[i])
        k=L.index(text1[i+1])
        mat[h,k]=mat[h,k]+1//matrix
    cPickle.dump(mat,f,-1)



text = [w for g in brown.categories() for w in brown.words(categories=g)]
text1=text[1:500]
arr=set(text1)
arr=list(arr)
mat=lil_matrix((len(arr),len(arr)))
markov(arr)
f.close()

I need to store this "mat" in a file and should access the value of the matrix using the co-ordinates..

result of the sparse matrix is like this: `the result of sparse matrix are like this:

(173, 168) 2.0 (173, 169) 1.0 (173, 172) 1.0 (173, 237) 4.0 (174, 231) 1.0 (175, 141) 1.0 (176, 195) 1.0

but when i store it into a file and read the same i'm getting it like this:

(0, 68) 1.0 (0, 77) 1.0 (0, 95) 1.0 (0, 100)    1.0 (0, 103)    1.0 (0, 110) 1.0 (0, 112)   2.0 (0, 132)    1.0 (0, 133)    2.0 (0, 139)    1.0 (0, 146)    2.0 (0, 156)    1.0 (0, 157)    1.0 (0, 185)    1.0

Do you have a particular database in mind? What's the size of these matrices? Have you considered sqlite (which Python has built-in support for)? — NullUserException
– NullUserException, Commented Mar 2, 2011 at 6:26
i'm a newbie, i don't know much, well i'm just trying to find a solution to store my matrix in harddisk i don't want to run the program to produce the matrix again and again, if i can store the matrix i can just reference the matrix for the values next time.. pls suggest me.. thanks :) — Bhuvan raj
– Bhuvan raj, Commented Mar 2, 2011 at 9:06
Please see the following to format your code so it is readable: stackoverflow.com/editing-help, but basically just put 4 spaces before each line of code and then indent as you would otherwise making sure you leave a blank line between code and any normal text. — JoshAdel
– JoshAdel, Commented Mar 2, 2011 at 19:52

ide · Accepted Answer · 2011-03-02 06:33:11Z

7

Assuming you have a numpy matrix or ndarray, which your question and tags imply, there is a dump method and load function you can use:

your_matrix.dump('output.mat')
another_matrix = numpy.load('output.mat')

answered Mar 2, 2011 at 6:33

ide

20.9k6 gold badges72 silver badges113 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Bhuvan raj Over a year ago

so can i use this output.mat anytime i need in other programs??

ide Over a year ago

Yes. It is just the path to any file on your hard drive where you would like to store the data.

JoshAdel · Accepted Answer · 2011-03-02 20:37:55Z

4

Note: This answer is in response to the revise question that now provides code.

You should not call cPickle.dump() in your function. Create the sparse matrix and then dump its contents to the file.

Try:

def markov(L):
   count=0
   c=len(text1)
   for i in range(0,c-2):
       h=L.index(text1[i])
       k=L.index(text1[i+1])
       mat[h,k]=mat[h,k]+1 #matrix


text = [w for g in brown.categories() for w in brown.words(categories=g)]
text1=text[1:500]
arr=set(text1)
arr=list(arr)
mat=lil_matrix((len(arr),len(arr)))
markov(arr)
f = open('spmatrix.pkl','wb')
cPickle.dump(mat,f,-1)
f.close()

edited Mar 2, 2011 at 20:37

answered Mar 2, 2011 at 20:08

JoshAdel

69.1k27 gold badges146 silver badges146 bronze badges

2 Comments

Bhuvan raj Over a year ago

This code runs without errors infact it outputs the matrix to the file leaving a part though!..i'm not able to use this command to.. mat.dump('output.mat') where mat is my matrix.. This is the error i'm finding raise AttributeError, attr + " not found" AttributeError: dump not found

JoshAdel Over a year ago

You seem to be using some other method now as mat.dump() does not appear in your code or in my suggestion. I apologize, but I do not have the time to help you further. Best of luck.

Paul · Accepted Answer · 2011-03-02 06:51:07Z

2

pyTables is the Python interface to HDF5 data model and is pretty popular choice for and well-integrated with NumPy and SciPy. pyTables will let you access slices of databased arrays without needing to load the entire array back into memory.

I don't have any specific experience with sparse matrices per se and a quick Google search neither confirmed nor denied that sparse matrices are supported.

edited Mar 2, 2011 at 6:51

answered Mar 2, 2011 at 6:32

Paul

43.9k17 gold badges112 silver badges126 bronze badges

Comments

whatnick · Accepted Answer · 2011-03-02 06:54:14Z

2

Adding on the HDF5 support, Python also has NetCDF support which is ideal for matrix form data storage and quick access both sparse and dense. It is included in Python-x,y for windows, which a lot of scientific users of python end up with.

More numpy based examples can be found in this cookbook.

answered Mar 2, 2011 at 6:54

whatnick

5,5003 gold badges22 silver badges36 bronze badges

Comments

Andrea Zonca · Accepted Answer · 2011-03-02 07:08:11Z

2

For very big sparse matrices on clusters, you might use pytrilinos, it has a HDF5 interface which can dump a sparse matrix to disk, and works also if the matrix is distributed on different nodes.

http://trilinos.sandia.gov/packages/pytrilinos/development/EpetraExt.html#input-output-classes

answered Mar 2, 2011 at 7:08

Andrea Zonca

8,83310 gold badges46 silver badges73 bronze badges

Comments

JoshAdel · Accepted Answer · 2011-03-02 16:35:40Z

2

Depending on the size of the sparse matrix, I tend to just use cPickle to pickle the array:

import cPickle
f = open('spmatrix.pkl','wb')
cPickle.dump(your_matrix,f,-1)
f.close()

If I'm dealing with really large datasets then I tend to use netcdf4-python

Edit:

To then access the file again you would:

f = open('spmatrix.pkl','rb') # open the file in read binary mode
# load the data in the .pkl file into a new variable spmat
spmat = cPickle.load(f) 
f.close()

edited Mar 2, 2011 at 16:35

answered Mar 2, 2011 at 13:14

JoshAdel

69.1k27 gold badges146 silver badges146 bronze badges

8 Comments

Bhuvan raj Over a year ago

Thanks a lot :) :) if u could explain how to access that spmatrix.pkl it will be of great help.. thanks again :)

Bhuvan raj Over a year ago

this is the actual input to the file (0, 148) 1.0 (1, 48) 1.0 (1, 173) 1.0 (2, 173) 1.0 (3, 168) 1.0 (4, 61) 1.0 (4, 91) 1.0 (5, 136) 1.0 (6, 237) 2.0 (7, 111) 1.0 but after reading the data from .pkl file i'm getting (0, 148) 1.0 (0, 48) 1.0 (0, 173) 1.0 (0, 173) 1.0 (0, 168) 1.0 (0, 61) 1.0 (0, 91) 1.0 (0, 136) 1.0 (0, 237) 2.0 (0, 111) 1.0 what is the error i have done as u have instructed :(

JoshAdel Over a year ago

I don't understand what your input is. Is it text in a file that you read into python? Please provide the exact code you used to generate the pickle. It's probably best if you edit the original question so you can format the code properly.

JoshAdel Over a year ago

Please put the code in your original question using the proper stackoverflow formatting so that I can actually read what you wrote

Bhuvan raj Over a year ago

I want to write this matrix "mat" into a file and using the indexes of the matrix i should be able to access the values using another program!...

|

EricP · Accepted Answer · 2012-08-29 15:13:13Z

For me, using the -1 option in cPickle.dump function caused the pickled file to not be loadable afterwards.

The object I dumped through cPickle was an instance of scipy.sparse.dok_matrix.

Using only two arguments did the trick for me; documentation about pickle.dump() states the default value of the protocol parameter is 0.

Working on Windows 7, Python 2.7.2 (64 bits), and cPickle v 1.71.

Example:

>>> import cPickle
>>> print cPickle.__version__
1.71
>>> from scipy import sparse
>>> H = sparse.dok_matrix((135, 654), dtype='int32')
>>> H[33, 44] = 8
>>> H[123, 321] = -99
>>> print str(H)
  (123, 321)    -99
  (33, 44)  8
>>> fname = 'dok_matrix.pkl'
>>> f = open(fname, mode="wb")
>>> cPickle.dump(H, f)
>>> f.close()
>>> f = open(fname, mode="rb")
>>> M = cPickle.load(f)
>>> f.close()
>>> print str(M)
  (123, 321)    -99
  (33, 44)  8
>>> M == H
True
>>>

Collectives™ on Stack Overflow

Python: how do you store a sparse matrix using python?

7 Answers 7

2 Comments

2 Comments

Comments

Comments

Comments

8 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

2 Comments

2 Comments

Comments

Comments

Comments

8 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related