Group Abstract Group Abstract

Message Boards Message Boards

1
|
9.4K Views
|
21 Replies
|
19 Total Likes
View groups...
Share
Share this post:

How can I get transform coefficients for Principal Component Analysis?

Posted 2 years ago

Mathematica provides the function PrincipalComponentAnalysis which transforms data to be expressed in terms of its principal components. How do I get the coefficients mapping the values of the original variables to their values expressed via the principal components.
I have seen this question asked elsewhere and not received any very satisfactory answer. I feel there ought to be a simple command or function that will do it, whereas the proposed solutions involve cumbersome manual manipulations.
Is it really the case that Mathematica cannot do this directly and, if so, what is the simplest workaround?
To clarify what I am looking for, suppose my original data is expressed in terms of two variables x and y, and I have two data points, (x1, y1) and (x2, y2). PrincipalComponentsAnalysis re-expresses the data in terms of variables u and v, which are linear combinations of x and y, thus returning points (u1,v1) and (u2,v2) such that

u1=a x1 + b y1

v1 = c x1 + d y1

u2 = a x2 + b y2

v2= c x2 +d y2

How do I find the a, b, c and d?

POSTED BY: Marc Widdowson
21 Replies

Hi Marc, some time ago, I wrote a package for PCA "à la française". Perhaps it would be useful to you?

Attachments:
POSTED BY: Claude Mante
Posted 2 years ago

Yes, that sounds like it would be useful to me. Thank you.

Having now had the chance to try the suggestion of @Sangdon Lee, I have found that SingularValueDecomposition works well subject to the following caveats:

  1. To get the best result from SVD, it is necessary first to subtract the mean of each variable from its respective values, i.e. to translate the values so that they have zero mean. Otherwise, SVD does not return the same result as PCA and is not as successful in concentrating the variance into a few components.

  2. With SVD, the signs of the transformed variables are in general different from those returned by PCA though the absolute values are the same. The sign is essentially arbitrary and this does not affect any conclusions from the analysis.

It seems that PrincipalComponents is a convenience method that automatically adjusts the variables to have zero mean before applying SVD. I now imagine that the reason there is no option to get a vector with the loadings from PrincipalComponents is because the transformed variables are not just a linear combination of the original variables but are related to them by a shift and then a linear combination. It would nevertheless be good if the PrincipalComponents documentation could include some discussion of this and of how to obtain the relationship between the original variables and the transformed variables if that is what one is interested in.

POSTED BY: Marc Widdowson
POSTED BY: Daniel Lichtblau
POSTED BY: Claude Mante
Posted 2 years ago

@Sangdon Lee

Thank you very much.

POSTED BY: Marc Widdowson
Posted 2 years ago

PCA is identical with SVD (Singular Value Decomposition). Therefore,

X=U*S*V'=T*V'= PC Scores * PC Loadings'.  (' is for Transpose).

The PrincipalComponentAnalysis function displays the PC Scores (T=U.S) only and does not provide the "V". Thus use the SingularValueDecomposition function. The "V" is the loadings you want to find.

For example,

  X = N[{{1, 2, 3}, {4, 5, 6}, {7, 8, 9}, {10, 11, 12}}]

  {U, S, V} = SingularValueDecomposition[X],

  U.S.Transpose[V] reproduces the original matrix (X).  

By the way, I wish that Wolfram would develop functions for PARAFAC.

I found Mathematica functions for Factor Analysis with Varimax rotation and PLS (partial least square) (don't remember the location. search in the Mathematica webpage). Anton Antonov developed ICA (independent component analysis): https://resources.wolframcloud.com/FunctionRepository/resources/IndependentComponentAnalysis/

POSTED BY: Sangdon Lee

I assume the original question involved

DimensionReduction[...,Method->"PrincipalComponentsAnalysis"]

and not the function PrincipalComponents. In which case this isn't correct, because, for whatever reason, the dimension reduction method does not correspond to the usual definition of PCA . The SVD usage, as shown, is however appropriate to the setting Method->"LatentSemanticAnalysis" of DimensionReduction and the like. Here is a quick example to illustrate.

mat = {{1., 2, 5.}, {3., -1., 2.}, {5, -1, 2}};
dimredPCA = DimensionReduction[mat, 2, Method -> "PrincipalComponentsAnalysis"];
dimredPCA[mat, "ReducedVectors"]

(* Out[107]= {{-2.34313, 0.0987804}, {0.83005, -0.55769}, {1.51308,  0.458909}} *)

We will not recover these using PCA.

{uu, ww, vv} = SingularValueDecomposition[mat, 2];
uu . ww

(* Out[110]= {{-4.16306, -3.55907}, {-3.55612, 1.06304}, {-5.00475,  2.20517}} *)

We do recover them with LSA though.

dimredLSA = DimensionReduction[mat, 2, Method -> "LatentSemanticAnalysis"];
dimredLSA[mat, "ReducedVectors"]

(* Out[112]= {{-4.16306, 3.55907}, {-3.55612, -1.06304}, {-5.00475, -2.20517}} *)

Let's get back to the other possibility, the function PrincipalComponents.

PrincipalComponents[mat]

(* Out[481]= {{3.45902, 0.187592, 0.}, {-1.10879, -0.877829, 0.}, {-2.35023, 0.690237, 0.}} *)

It turns out this is essentially the same as what comes from the resource function MultidimensionalScaling, bearing in mind that resulting columns are only unique up to sign.

ResourceFunction["MultidimensionalScaling"][mat, 3]

(* Out[482]= {{-3.45902, 0.187592, 0.}, {1.10879, -0.877829, 0.}, {2.35023, 0.690237, 0.}} *)

Who knew? Certainly not the author of RF[MDS]. As a further note, this is essentially different from using Method->"MultidimensionalScaling" in DimensionReduction. Another note is that both PrincipalComponents and ResourceFunction["MultidimensionalScaling"] use what is usually termed "Principal Coordinate Analysis" (read second word carefully). This is as seen in the Wikipedia article for MDS at

https://en.wikipedia.org/wiki/Multidimensional_scaling

but I will remark that I've seen the same distinction in other places as well.

POSTED BY: Daniel Lichtblau
Posted 2 years ago

Hi Daniel,

I believe you and I are discussing different aspects. Your comment pertains to the PC scores (T=US) (e.g., uu.ww in your comments), while my comment is about the PC loadings (V). The initial question was regarding the PrincipalComponentAnalysis function, which only displays the T and does not provide the V. The V shows how the original variables are transformed (as a linear combination of the original variables) and maybe interpreted afterward. The T values represent the outcomes of the transformation (the reduced vector): T=XV, where X=TV' and XV=TV'V=T, as V'V=I.

X=USV’=Left singular vector*Singular values*right singular vector’ = TV’ = PC scores*loadings.  

SVD is related to EVD as follows to clarify terminologies between SVD and EVD:

A=X’X=(USV’)’(USV’)=VS^2V’=eigenvectors*eigenvalues*eigenvectors’, because U’U=I and V’V=I. 

By the way, many dimension reduction methods apply SVD after deriving various “similarity” matrices. X’X represents a covariance or correlation matrix in PCA, depending on normalization or standardization applied to column variables. Experiences regarding the applications of PCA, MDS, and CA have reported producing similar results.

  • PCA: X --> covariance or correlation matrix (X’X) --> apply SVD on the covariance matrix.
  • Multidimensional Scaling (MDS) : X--> distance matrix (various distance matrices) --> apply SVD on the distance matrix
  • Correspondence analysis (CA): X-->profile matrix -->apply SVD on the profile matrix

By the ways, thanks for the detail information about the DimensionReduction function. I am not well familiar with this function and your comments are useful to me.

POSTED BY: Sangdon Lee
POSTED BY: Daniel Lichtblau
Posted 2 years ago

Hi Daniel,

I posted another question on PCA after reading your comments carefully because I consider PCA as one of the most important methods.

https://community.wolfram.com/groups/-/m/t/2949003?p_p_auth=6pSCpXdg

POSTED BY: Sangdon Lee

Sangdon,

I upvoted your new post and also upvoted this one (which I should have done days ago). I agree the naming conventions, lack of full documentation and lack of means to work with DimensionReduction results all are weaknesses.

POSTED BY: Daniel Lichtblau
Posted 2 years ago

Yes, mea culpa. In the original post, I referred to a PrincipalComponentsAnalysis function when I should have said PrincipalComponents function. Sorry for the confusion. It does seem that this is kind of an omission in WL. When you're interested in doing a PCA with Mathematica, you naturally reach for the PrincipalComponents function and yet it won't give you the loadings, which is, very often, what is of most interest, i.e. to identify which of your original variables is responsible for most of the variation in the data. While you can, it seems, extract that information with other methods as described by @Sangdon Lee , you don't really want to be reading through MSE or Wolfram Community threads to try and understand exactly how PCA works so that you can apply other more general purpose functions to your problem.

POSTED BY: Marc Widdowson

Marc, Do you have a simple example of input and desired output? It's still not clear to me what you want, and whether it pertains to the usual definition of Principal Components Analysis, or to the actual implementation of PrincipalComponents.

POSTED BY: Daniel Lichtblau
Posted 2 years ago
POSTED BY: Marc Widdowson

Marc,

I think Sangdon Lee may have provided a bit more than what was originally requested. Per my prior notes, it turns out that ``PrincipalComponents``` is really implementing something else under the hood. So his recipe not only gives the other values of interest but also shows how to do the PCA you actually want.

From your description I think you want the singular values and perhaps also the conversion matrix one would apply to new data to attain the same projection.

POSTED BY: Daniel Lichtblau
Posted 2 years ago

You're right. Thank you very much.

POSTED BY: Marc Widdowson

Thanks for referring to my implementation of Independent Component Analysis (ICA)!

I recently "pacletized" the ICA and Non-Negative Matrix Factorization (NNMF) into the paclet "DimensionReducers".

(So, I can have a more modular implementation of my LSA monad.)

POSTED BY: Anton Antonov

Thanks a lot for sharing! I frequently use your nice MonadicQuantileRegression workflow.

POSTED BY: Claude Mante

Good to hear! It inspires me "finish" the documentation of the LSA monad...

POSTED BY: Anton Antonov
Posted 2 years ago
POSTED BY: Marc Widdowson

There is this MSE post that might be of use.

POSTED BY: Daniel Lichtblau
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard