Yes, that sounds like it would be useful to me. Thank you.

Having now had the chance to try the suggestion of @Sangdon Lee, I have found that SingularValueDecomposition works well subject to the following caveats:

1. To get the best result from SVD, it is necessary first to subtract the mean of each variable from its respective values, i.e. to translate the values so that they have zero mean. Otherwise, SVD does not return the same result as PCA and is not as successful in concentrating the variance into a few components.

2. With SVD, the signs of the transformed variables are in general different from those returned by PCA though the absolute values are the same. The sign is essentially arbitrary and this does not affect any conclusions from the analysis.

It seems that PrincipalComponents is a convenience method that automatically adjusts the variables to have zero mean before applying SVD. I now imagine that the reason there is no option to get a vector with the loadings from PrincipalComponents is because the transformed variables are not just a linear combination of the original variables but are related to them by a shift and then a linear combination. It would nevertheless be good if the PrincipalComponents documentation could include some discussion of this and of how to obtain the relationship between the original variables and the transformed variables if that is what one is interested in.

Hi Daniel!

There is a lot of variants of MDS, and the one implemented here is Principal Coordinates Analysis. So, It is exactly identical to PCA (up to the sign of eigenvectors), but since you cannot have an access to the loadings of the variable, results are less understandable than those from PCA. So, PCA is best, excepted if one use a special proximity measure (the Rao distance between distributions, the ManhattanDistance between vectors, etc). In such cases, the original table of data is no more considered, but there is NO guarantee to fit an Euclidean structure to the data. For instance, the Torgerson matrix may have large negative eigenvalues, which shows that data don't have an euclidean image - as an example, the Rao's distance is Riemannian.

PCA is identical with SVD (Singular Value Decomposition). Therefore,

X=U*S*V'=T*V'= PC Scores * PC Loadings'.  (' is for Transpose).

The PrincipalComponentAnalysis function displays the PC Scores (T=U.S) only and does not provide the "V". Thus use the SingularValueDecomposition function. The "V" is the loadings you want to find.

For example,

  X = N[{{1, 2, 3}, {4, 5, 6}, {7, 8, 9}, {10, 11, 12}}]

  {U, S, V} = SingularValueDecomposition[X],

  U.S.Transpose[V] reproduces the original matrix (X).  

By the way, I wish that Wolfram would develop functions for PARAFAC.

I found Mathematica functions for Factor Analysis with Varimax rotation and PLS (partial least square) (don't remember the location. search in the Mathematica webpage). Anton Antonov developed ICA (independent component analysis): https://resources.wolframcloud.com/FunctionRepository/resources/IndependentComponentAnalysis/

Hi Daniel,

I believe you and I are discussing different aspects. Your comment pertains to the PC scores (T=US) (e.g., uu.ww in your comments), while my comment is about the PC loadings (V). The initial question was regarding the PrincipalComponentAnalysis function, which only displays the T and does not provide the V. The V shows how the original variables are transformed (as a linear combination of the original variables) and maybe interpreted afterward. The T values represent the outcomes of the transformation (the reduced vector): T=XV, where X=TV' and XV=TV'V=T, as V'V=I.

X=USV’=Left singular vector*Singular values*right singular vector’ = TV’ = PC scores*loadings.  

SVD is related to EVD as follows to clarify terminologies between SVD and EVD:

A=X’X=(USV’)’(USV’)=VS^2V’=eigenvectors*eigenvalues*eigenvectors’, because U’U=I and V’V=I. 

By the way, many dimension reduction methods apply SVD after deriving various “similarity” matrices. X’X represents a covariance or correlation matrix in PCA, depending on normalization or standardization applied to column variables. Experiences regarding the applications of PCA, MDS, and CA have reported producing similar results.

• PCA: X --> covariance or correlation matrix (X’X) --> apply SVD on the covariance matrix.
• Multidimensional Scaling (MDS) : X--> distance matrix (various distance matrices) --> apply SVD on the distance matrix
• Correspondence analysis (CA): X-->profile matrix -->apply SVD on the profile matrix

By the ways, thanks for the detail information about the DimensionReduction function. I am not well familiar with this function and your comments are useful to me.

Hi Daniel,

I posted another question on PCA after reading your comments carefully because I consider PCA as one of the most important methods.

https://community.wolfram.com/groups/-/m/t/2949003?p_p_auth=6pSCpXdg

Yes, mea culpa. In the original post, I referred to a PrincipalComponentsAnalysis function when I should have said PrincipalComponents function. Sorry for the confusion. It does seem that this is kind of an omission in WL. When you're interested in doing a PCA with Mathematica, you naturally reach for the PrincipalComponents function and yet it won't give you the loadings, which is, very often, what is of most interest, i.e. to identify which of your original variables is responsible for most of the variation in the data. While you can, it seems, extract that information with other methods as described by @Sangdon Lee , you don't really want to be reading through MSE or Wolfram Community threads to try and understand exactly how PCA works so that you can apply other more general purpose functions to your problem.

Again, I apologise for causing confusion by my inaccuracy in using the wrong name for the function PrincipalComponents.

Here is the situation.

The data characterises a range of societies in terms of 15 sociological attributes, each of which can have the value either +1, 0, or -1. So each society is a 15-dimensional point. It is suspected that some of these attributes or dimensions are correlated and this is the kind of thing that PCA ought to be able to pick up.

There are thousands of datapoints, and just for illustration the first five are as follows: {{0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., -1., 0., 0., 0.}, {0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., -1., 0., 0., 0.}, {0., 0., 0., -1., 0., 0., 0., 0., 0., 0., 0., -1., 0., 0., 0.}, {0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., -1., 0., 0., 0.}, {0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., -1., 0., 0., 0.}}

When I supply the full dataset to PrincipalComponents, I get back a list of new 15-dimensional points in the transformed dimensions, which appear in order of the amount of variance they contain. Here are the first five points of the transformed data: {{0.628139, 0.080849, 0.141136, 0.215542, 0.123124, 0.475692, 0.250839, 0.0970342, 0.369091, 0.187105, 0.232599, 0.16679, 0.123272, 0.0682626, 0.096034}, {0.628139, 0.080849, 0.141136, 0.215542, 0.123124, 0.475692, 0.250839, 0.0970342, 0.369091, 0.187105, 0.232599, 0.16679, 0.123272, 0.0682626, 0.096034}, {0.326903, -0.0281672, 0.16276, 0.355153, 0.102935, 0.592857, 0.242404, -0.149108, 0.177386, 0.461905, -0.269042, -0.0351931, 0.341595, -0.0184339, 0.681469}, {0.628139, 0.080849, 0.141136, 0.215542, 0.123124, 0.475692, 0.250839, 0.0970342, 0.369091, 0.187105, 0.232599, 0.16679, 0.123272, 0.0682626, 0.096034}, {0.628139, 0.080849, 0.141136, 0.215542, 0.123124, 0.475692, 0.250839, 0.0970342, 0.369091, 0.187105, 0.232599, 0.16679, 0.123272, 0.0682626, 0.096034}}

I can calculate the variance along each dimension just by applying Variance to the list returned by PrincipalComponents, and I find that the first dimension (first principal component) has about 65% of the total variance.

Now I know that that each dimension of the transformed data is a linear combination of the original variables / dimensions / attributes. So if we write pc1 for the value on the first principal component (0.628139 in the case of the first datapoint above) and v1...v15 for my original variable values ({0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., -1., 0., 0., 0.} in the case of the first datapoint above), I know that

pc1 = a1 v1 + a2 v2 + a3 v3 + ... + a15 v15

where a1...a15 are coefficients describing the transformation. It is these coefficients that I want to find. This is because, if the coefficient is large for a particular variable, it means that that variable contributes strongly to the first principal component, and that is of interest because it means that that variable is particularly good at capturing the distinctive character of each society.

Of course, there are also coefficients mapping the variables onto the other principal components, e.g.

pc2 = b1 v1 + b2 v2 + b3 v3 + ... + b15 v15

pc3 = c1 v1 + c2 v2 + c3 v3 + ... + c15 v15

etc.

which are of subsidiary interest.

In my particular case, because my original data is so sparse and the values are restricted to 1, 0, and -1, I can pretty much work out the coefficients with pencil and paper. However, I felt that this is the sort of thing that Mathematica ought to be able to return very easily, to check my own working, and I thought it would just be a case of supplying some option to PrincipalComponents. It was when I couldn't find any way to do that, and the fact that the MSE threads did not seem to give any simple, definitive answer, that caused me to post here.

I believe @Sangdon Lee has understood what I was asking for and given me the solution.

Thank you very much for your interest and assistance.

You're right. Thank you very much.

Thanks a lot for sharing! I frequently use your nice MonadicQuantileRegression workflow.

Thank you very much. I have seen that post and some similar ones. Unfortunately, that post illustrates the problem in that it talks about manually reproducing the effect of the PrincipalComponentAnalysis function rather than pointing to any built-in Mathematica functionality, and in any case it doesn't answer my specific question about getting the coefficients. I know that I can work out the coefficients from first principles. However, I would have thought that there ought to be a function in Mathematica for doing this. The closest solution I have seen is in this MSE post, which suggests using the KarhunenLoeveDecomposition function because that does return the transformed axes as vectors expressed in the original basis, though it also requires some manual manipulation of the data to get it zero-centred before applying the function.

I became interested in this because a friend was doing a PCA on some data in R and I thought I would like to reproduce what he did in Mathematica. At first I thought great, when I saw that there is a PrincipalComponentAnalysis function, but then I could not find any simple way to get the coefficients, which my friend gets easily and automatically in R. I am surprised that my options are either to use a different, more general purpose function, KarhunenLoeveDecomposition, which is not ideal for this application, or to program it manually from first principles. Nevertheless, it seems that must be the case.