Group Abstract Group Abstract

Message Boards Message Boards

0
|
4.5K Views
|
6 Replies
|
0 Total Likes
View groups...
Share
Share this post:

Normalizing training and testing data in a matrix

Pei Hsuan Shen
Posted 4 years ago

I want to standardize training data and testing data. Both of them are matrix. But I have to make testing data minus the mean of training data and then devide the standard deviation of training data. I found a similar question below, but the format of my data is matrix. So, how to write it?

https://mathematica.stackexchange.com/questions/212351/how-can-i-normalize-a-dataset-relative-to-data-already-given-previously
POSTED BY: Pei Hsuan Shen
6 Replies
Sort By:
Eric Rimbey
Eric Rimbey
Posted 4 years ago
POSTED BY: Eric Rimbey
Eric Rimbey
Eric Rimbey
Posted 4 years ago

I'm not sure what exactly the semantics of your situation are, but flattening your matrices into lists would at least give you something that "works" (as in produces an answer) with Standardize:

restandardized = standardizeTo[Flatten@old, Flatten@new]

So, if your question was "how do I turn a matrix into a flat list?", then Flatten is your answer. If that's still not your question, I apologize that I'm not a domain expert for Standardize. Maybe some more "flavor" would help.
POSTED BY: Eric Rimbey
Pei Hsuan Shen
Pei Hsuan Shen
Posted 4 years ago

Thank you for your answer. But the result needs 10 mean and 10 standard devieation. You can image that every column is a feature and every feature has it own mean and deviation.

POSTED BY: Pei Hsuan Shen
Eric Rimbey
Eric Rimbey
Posted 4 years ago

Well, it would be a lot easier if we could see some code that you tried. It would help to know the exact functions you're trying to use.

But, taking a stab in the dark here, a matrix is just a two dimensional array. So, specifically, it's a List of Lists where all the child Lists are the same length.

{{1, 1, 1, 1, 1}, {1, 1, 1, 1, 1}, {1, 1, 1, 1, 1}, {1, 1, 1, 1, 1}}

Is that even close to what you're asking?
POSTED BY: Eric Rimbey
Pei Hsuan Shen
Pei Hsuan Shen
Posted 4 years ago

enter image description here

ClearAll[standardizeTo]
standardizeTo[oldData_, newData_] :=
 Module[
   {oldMean, oldStdev},
   {oldMean, oldStdev} = Through[{Mean, StandardDeviation}[oldData]];
   oldStdev Standardize[newData] + oldMean
 ]

old = {{5,1,1,1,2,1,3,1,1,2},{5,4,4,5,7,10,3,2,1,2},{3,1,1,1,2,2,3,1,1,2},{6,8,8,1,3,4,3,7,1,2},
{4,1,1,3,2,1,3,1,1,2},{8,10,10,8,7,10,9,7,1,4},{1,1,1,1,2,10,3,1,1,2},{2,1,2,1,2,1,3,1,1,2},{2,1,1,1,2,1,1,1,5,2},
{4,2,1,1,2,1,2,1,1,2},{1,1,1,1,1,1,3,1,1,2},{2,1,1,1,2,1,2,1,1,2}};
new = {{4,8,6,4,3,4,10,6,1,4},{4,8,8,5,4,5,10,4,1,4}};

restandardized = standardizeTo[old, new];
POSTED BY: Pei Hsuan Shen
Pei Hsuan Shen
Pei Hsuan Shen
Posted 4 years ago

I can't use the function directly.
POSTED BY: Pei Hsuan Shen
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard