2

I have a doubt about SVD. in the literature that i had read, it's written that we have to convert our input matrix into covariance matrix first, and then SVD function from matlab (SVD) is used.

But, in Mathworks website we can use SVD function directly to the input matrix (no need to convert it into covariance matrix)..

   [U,S,V]=svd(inImageD);

  1. Which one is the true??

  2. And if we want to do dimensionality reduction, we have to project our data into eigen vector.. But where is the eigen vector generated by SVD function.. I know that S is the eigen value.. But what is U and S??

  3. To reduce our data dimensional, do we need to substract the input matrix with its mean and then multiply it with eigen vector?? or we can just multiply our input matrix with the eigen vector (no need to substract it first with its mean)..

EDIT

Suppose if I want to do classification using SIFT as the features and SVM as the classifier.

I have 10 images for training and I arrange them in a different row..

So 1st row for 1st images, 2nd row for second images and so on...

Feat=[1 2 5 6 7   >> Images1
      2 9 0 6 5   >> Images2
      3 4 7 8 2   >> Images3
      2 3 6 3 1   >> Images4
      ..
      .
      so on. . ]

To do dimensionality reduction (from my 10x5 matrix), we have yo do A*EigenVector

And from what U had explained (@Sam Roberts), I can compute it by using EIGS function from the covariance matrix (instead of using SVD function).

And as I arrange the feat of images in different row, so I need to do A'*A So it becomes:

 Matrix=A'*A
 MAT_Cov=Cov(Matrix)
 [EigVector EigValue] = eigs (MAT_Cov);

is that right??

Improve this question
4
  • 1
    take a look en.wikipedia.org/wiki/… Commented May 12, 2014 at 1:48
  • My answer did not mention the function eigs at all. It described the function eig, which is different. I also did not suggest that you use either eig or eigs - I just described how EVD can be seen as equivalent to SVD. If I were in your position, I would not use EVD, I would use SVD. And I wouldn't use the command svd for that, I would pca or princomp. Commented May 25, 2014 at 10:31
  • Just to be clear, eigs is typically used for sparse matrices, whereas eig is used for full matrices. Commented May 25, 2014 at 10:36
  • I become confuse now,, Can u explain by using an example. For ex: I have feature matrix A=[1 2 3 4 5 6;9 7 6 5 5 7; 9 8 0 1 2 3;4 5 6 2 1 0]; There I have 3 images with 6 features per each images.. How to do dimensionality reduction?? Commented May 25, 2014 at 14:57

1 Answer 1

Reset to default
4

Eigenvector decomposition (EVD) and singular value decomposition (SVD) are closely related.

Let's say you have some data a = rand(3,4);. Note that this not a square matrix - it represents a dataset of observations (rows) and variables (columns).

Do the following:

[u1,s1,v1] = svd(a);
[u2,s2,v2] = svd(a');
[e1,d1] = eig(a*a');
[e2,d2] = eig(a'*a);

Now note a few things.

  1. Up to the sign (+/-), which is arbitrary, u1 is the same as v2. Up to a sign and an ordering of the columns, they are also equal to e1. (Note that there may be some very very tiny numerical differences as well, due to slight differences in the svd and eig algorithms).
  2. Similarly, u2 is the same as v1 and e2.
  3. s1 equals s2, and apart from some extra columns and rows of zeros, both also equal sqrt(d1) and sqrt(d2). Again, there may be some very tiny numerical differences as well just due to algorithmic issues (they'll be on the order of something to the -10 or so).

Note also that a*a' is basically the covariances of the rows, and a'*a is basically the covariances of the columns (that's not quite true - a would need to be centred first by subtracting the column or row mean for them to be equal, and there might be a multiplicative constant difference as well, but it's basically pretty similar).

Now to answer your questions, I assume that what you're really trying to do is PCA. You can do PCA either by taking the original data matrix and applying SVD, or by taking its covariance matrix and applying EVD. Note that Statistics Toolbox has two functions for PCA - pca (in older versions princomp) and pcacov.

Both do essentially the same thing, but from different starting points, because of the above equivalences between SVD and EVD.

Strictly speaking, u1, v1, u2 and v2 above are not eigenvectors, they are singular vectors - and s1 and s2 are singular values. They are singular vectors/values of the matrix a. e1 and d1 are the eigenvectors and eigenvalues of a*a' (not a), and e2 and d2 are the eigenvectors and eigenvalues of a'*a (not a). a does not have any eigenvectors - only square matrices have eigenvectors.

Centring by subtracting the mean is a separate issue - you would typically do that prior to PCA, but there are situations where you wouldn't want to. You might also want to normalise by dividing by the standard deviation but again, you wouldn't always want to - it depends what the data represents and what question you're trying to answer.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Sorry, I didnt get any notif abt this before, So I didnt know that u had answered it.. And thanks that you answered it in good way.. But I just wanna make it clear by using an example,, Suppose if I want to do classification using SIFT as the features and SVM as the classifier. I have 10 images for training..
Sorry, I didnt get any notif abt this before, So I didnt know that u had answered it.. And thanks that you answered it in good way.. But I just wanna make it clear by using an example,, As it is too long, I wrote them in my question in EDIT section,, Please have a look at it,, And im going to accept ur answer for it.. Just need to make it clear

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.