I want to standardize training data and testing data. Both of them are matrix. But I have to make testing data minus the mean of training data and then devide the standard deviation of training data.
I found a similar question below, but the format of my data is matrix. So, how to write it?
https://mathematica.stackexchange.com/questions/212351/how-can-i-normalize-a-dataset-relative-to-data-already-given-previously
Well, it would be a lot easier if we could see some code that you tried. It would help to know the exact functions you’re trying to use.
But, taking a stab in the dark here, a matrix is just a two dimensional array. So, specifically, it’s a List of Lists where all the child Lists are the same length.
{{1, 1, 1, 1, 1}, {1, 1, 1, 1, 1}, {1, 1, 1, 1, 1}, {1, 1, 1, 1, 1}}
Is that even close to what you’re asking?
ClearAll[standardizeTo]
standardizeTo[oldData_, newData_] :=
Module[
{oldMean, oldStdev},
{oldMean, oldStdev} = Through[{Mean, StandardDeviation}[oldData]];
oldStdev Standardize[newData] + oldMean
]
old = {{5,1,1,1,2,1,3,1,1,2},{5,4,4,5,7,10,3,2,1,2},{3,1,1,1,2,2,3,1,1,2},{6,8,8,1,3,4,3,7,1,2},
{4,1,1,3,2,1,3,1,1,2},{8,10,10,8,7,10,9,7,1,4},{1,1,1,1,2,10,3,1,1,2},{2,1,2,1,2,1,3,1,1,2},{2,1,1,1,2,1,1,1,5,2},
{4,2,1,1,2,1,2,1,1,2},{1,1,1,1,1,1,3,1,1,2},{2,1,1,1,2,1,2,1,1,2}};
new = {{4,8,6,4,3,4,10,6,1,4},{4,8,8,5,4,5,10,4,1,4}};
restandardized = standardizeTo[old, new];
I can’t use the function directly.
I’m not sure what exactly the semantics of your situation are, but flattening your matrices into lists would at least give you something that “works” (as in produces an answer) with Standardize:
restandardized = standardizeTo[Flatten@old, Flatten@new]
So, if your question was “how do I turn a matrix into a flat list?”, then Flatten is your answer. If that’s still not your question, I apologize that I’m not a domain expert for Standardize. Maybe some more “flavor” would help.
Thank you for your answer. But the result needs 10 mean and 10 standard devieation. You can image that every column is a feature and every feature has it own mean and deviation.
From the documentation on Standardize:
Based on that, and on the error message you received, it seems pretty clear that the problem is that you have columns in new whose values are equal, and so StandardDeviation returns 0. So, maybe Standardize is not doing what you want for 2D data, in which case, you’ll need to transform your data somehow. Or maybe your data is malformed, so you’ll need to figure out what’s wrong with it.
Anyway, it doesn’t seem like the problem is with the shape of your data but with the values of the data itself. But then, again, I don’t really know the semantics of your situation, so I may be spouting nonsense.
