Python bug when creating matrices

Question

I have written a code in Python to create a transition probability matrix from the data, but I keep getting wrong values for two specific data points. I have spent several days on trying to figure out the problem, but with no success.

About the code: The input is 4 columns in csv file. After preparation of the data, the first two columns are the new and old state values. I need to calculate how often each old state value transfers to a new one (basically, how often each pair (x,y) occurs in the first two columns of the data). The values in these columns are from 0 to 99. In the trans_pr matrix I want to get a number how often a pair (x,y) occurs in the data and have this number at the corresponding coordinates (x,y) in the trans_pr matrix. Since the values are from 0 to 99 I can just add 1 to the matrix at this coordinates each time they occur in the data.

The problem: The code works fine, but I always get zeros at coordinates (:,29) and (:,58) and (29,:) and (58;:) despite having observations there. It also sometimes seems to add the number at this coordinates to the previous line. Again, doesn't make any sense to me.

I would be very grateful if anyone could help. (I am new to Python, therefore the code is probably inefficient, but only the bug is relevant.)

The code is as simple as it can be:

from numpy import *
import csv

my_data = genfromtxt('99c_test.csv', delimiter=',')

"""prepares data for further calculations"""
my_data1=zeros((len(my_data),4))
my_data1[1:,0]=100*my_data[1:,0]
my_data1[1:,1]=100*my_data[1:,3]
my_data1[1:,2]=my_data[1:,1]
my_data1[1:,3]=my_data[1:,2]
my_data2=my_data1
trans_pr=zeros((101,101))
print my_data2

"""fills the matrix with frequencies of observations"""

for i in range(len(my_data2)):
    trans_pr[my_data2[i,1],my_data2[i,0]]=trans_pr[my_data2[i,1],my_data2[i,0]]+1

c = csv.writer(open("trpr1.csv", "wb"))
c.writerows(trans_pr)

You can test the code with this input (just save it as csv file):

p_cent,p_euro,p_euro_old,p_cent_old
0.01,1,1,0.28
0.01,1,1,0.29
0.01,1,1,0.3
0.01,1,1,0.28
0.01,1,1,0.29
0.01,1,1,0.3
0.01,1,1,0.57
0.01,1,1,0.58
0.01,1,1,0.59
0.01,1,1,0.6

Is this currency? I'd suggest you have a look at the decimal Python library. — Skurmedel
– Skurmedel, Commented May 22, 2011 at 11:48

Howard · Accepted Answer · 2011-05-22 11:44:20Z

4

This sound very much like a rounding issue. I'd suppose that e.g. 100*0.29 (as a floating point number) is rounded downwards (i.e. truncated) and thus yields 28 instead of 29. Try rounding the numbers by yourself (i.e. a up/down rounding) before using them as an array index.

Update: Verified my conjecture by testing it, even the numbers are as described above - see here.

edited May 22, 2011 at 11:44

answered May 22, 2011 at 11:31

Howard

39.3k9 gold badges68 silver badges85 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user764704 Over a year ago

It has worked. I suspected it may be something about python handling numbers in a weird way, but didn't know what to do. Thank you very much!

xgrg · Accepted Answer · 2011-05-22 13:03:46Z

4

You may findrint() useful, from numpy. It rounds a value to its nearest integer (see numpy.rint() doc). Have you tried the following :

for i in range(len(my_data2)):
    trans_pr[rint(my_data2[i,1]), rint(my_data2[i,0])] = \
         trans_pr[rint(my_data2[i,1]), rint(my_data2[i,0])] + 1

edited May 22, 2011 at 13:03

answered May 22, 2011 at 12:29

xgrg

414 bronze badges

2 Comments

Katriel Over a year ago

Sorry, wasn't clear. rint is not a Python function that I know of; it's not in the builtin namespace or in math.

xgrg Over a year ago

You are right, it is not in the builtin namespace nor in math but from numpy. Since it is imported at the beginning, I assumed it could be used. Now adding that detail in my answer, thanks.

Collectives™ on Stack Overflow

Python bug when creating matrices

2 Answers 2

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related