Message Boards Message Boards

0
|
2846 Views
|
6 Replies
|
0 Total Likes
View groups...
Share
Share this post:

Normalizing training and testing data in a matrix

Posted 2 years ago

I want to standardize training data and testing data. Both of them are matrix. But I have to make testing data minus the mean of training data and then devide the standard deviation of training data. I found a similar question below, but the format of my data is matrix. So, how to write it?

https://mathematica.stackexchange.com/questions/212351/how-can-i-normalize-a-dataset-relative-to-data-already-given-previously

POSTED BY: Pei Hsuan Shen
6 Replies
Posted 2 years ago

From the documentation on Standardize:

enter image description here

Based on that, and on the error message you received, it seems pretty clear that the problem is that you have columns in new whose values are equal, and so StandardDeviation returns 0. So, maybe Standardize is not doing what you want for 2D data, in which case, you'll need to transform your data somehow. Or maybe your data is malformed, so you'll need to figure out what's wrong with it.

Anyway, it doesn't seem like the problem is with the shape of your data but with the values of the data itself. But then, again, I don't really know the semantics of your situation, so I may be spouting nonsense.

POSTED BY: Eric Rimbey
Posted 2 years ago

I'm not sure what exactly the semantics of your situation are, but flattening your matrices into lists would at least give you something that "works" (as in produces an answer) with Standardize:

restandardized = standardizeTo[Flatten@old, Flatten@new]

So, if your question was "how do I turn a matrix into a flat list?", then Flatten is your answer. If that's still not your question, I apologize that I'm not a domain expert for Standardize. Maybe some more "flavor" would help.

POSTED BY: Eric Rimbey
Posted 2 years ago

Thank you for your answer. But the result needs 10 mean and 10 standard devieation. You can image that every column is a feature and every feature has it own mean and deviation.

POSTED BY: Pei Hsuan Shen
Posted 2 years ago

Well, it would be a lot easier if we could see some code that you tried. It would help to know the exact functions you're trying to use.

But, taking a stab in the dark here, a matrix is just a two dimensional array. So, specifically, it's a List of Lists where all the child Lists are the same length.

{{1, 1, 1, 1, 1}, {1, 1, 1, 1, 1}, {1, 1, 1, 1, 1}, {1, 1, 1, 1, 1}}

Is that even close to what you're asking?

POSTED BY: Eric Rimbey
Posted 2 years ago

enter image description here

ClearAll[standardizeTo]
standardizeTo[oldData_, newData_] :=
 Module[
   {oldMean, oldStdev},
   {oldMean, oldStdev} = Through[{Mean, StandardDeviation}[oldData]];
   oldStdev Standardize[newData] + oldMean
 ]

old = {{5,1,1,1,2,1,3,1,1,2},{5,4,4,5,7,10,3,2,1,2},{3,1,1,1,2,2,3,1,1,2},{6,8,8,1,3,4,3,7,1,2},
{4,1,1,3,2,1,3,1,1,2},{8,10,10,8,7,10,9,7,1,4},{1,1,1,1,2,10,3,1,1,2},{2,1,2,1,2,1,3,1,1,2},{2,1,1,1,2,1,1,1,5,2},
{4,2,1,1,2,1,2,1,1,2},{1,1,1,1,1,1,3,1,1,2},{2,1,1,1,2,1,2,1,1,2}};
new = {{4,8,6,4,3,4,10,6,1,4},{4,8,8,5,4,5,10,4,1,4}};

restandardized = standardizeTo[old, new];
POSTED BY: Pei Hsuan Shen
Posted 2 years ago

I can't use the function directly.

POSTED BY: Pei Hsuan Shen
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract