How do I ReplacePart conditionally?

Questions
1

I have a four column matrix with missing data, each represented by “x”

ImportString["12\t27\t19\t44
11\tx\t13\t43
x\t56\t23\t60
15\t37\t19\t52
10\t31\t21\tx
18\t40\t22\t43
15\t50\t28\t56
21\tx\tx\t46
20\t39\t25\t51
x\t36\t23\t56
x\t27\t21\t39
14\t32\t20\t46
22\t40\tx\t49
x\t31\tx\t44
x\t18\t21\tx
16\t23\t21\t34
27\t26\t23\t59
17\tx\tx\t35
13\t21\t20\t48
11\t51\t18\t54
", "TSV"];

I need to replace each “x” with a Random Variate from a distribution that is specific to each column (the parameters are determined by mean and variance in actual observed data in each column)

It seems that, for the first column,

If[# == "x", 
  ReplacePart[
   data[[All,1]], # -> RandomVariate[NormalDistribution[0, 1]]]] &

should work, but it doesn’t.
Once that issue is solved the four different parameter sets for the four distributions needs to be threaded over the four columns.

2

Here’s a possible approach:

(*Find where all the "x" are. Since the replacement will depend on \
position, we can use these positions later.*)
Position[data, "x"]

(*You'll need some function that uses the position to determine a \
replacement value. This is just a placeholder for this example*)
ColumnBasedFn

(*Here's a function that uses the position to do a replacement on the \
data*)
TheReplacer[data_, position_] := 
 ReplacePart[data, position -> ColumnBasedFn[Last@position]]

(*And finally, we apply the replacer with our positions. There are \
many ways to do this, I just happened to choose Fold.*)
Fold[TheReplacer, data, Position[data, "x"]]

This produces:

{{12, 27, 19, 44}, {11, ColumnBasedFn[2], 13, 43}, {ColumnBasedFn[1], 
  56, 23, 60}, {15, 37, 19, 52}, {10, 31, 21, ColumnBasedFn[4]}, {18, 
  40, 22, 43}, {15, 50, 28, 56}, {21, ColumnBasedFn[2], 
  ColumnBasedFn[3], 46}, {20, 39, 25, 51}, {ColumnBasedFn[1], 36, 23, 
  56}, {ColumnBasedFn[1], 27, 21, 39}, {14, 32, 20, 46}, {22, 40, 
  ColumnBasedFn[3], 49}, {ColumnBasedFn[1], 31, ColumnBasedFn[3], 
  44}, {ColumnBasedFn[1], 18, 21, ColumnBasedFn[4]}, {16, 23, 21, 
  34}, {27, 26, 23, 59}, {17, ColumnBasedFn[2], ColumnBasedFn[3], 
  35}, {13, 21, 20, 48}, {11, 51, 18, 54}}
3

Here’s another approach:

(*This time we look at the actual value, so we use a function that \
has a special case for "x" and doesn't disturb any other values*)
AnotherReplacer["x", position_] := ColumnBasedFn[Last@position];
AnotherReplacer[value_, _] := value

(*MapIndexed will provide the position info we need.*)
MapIndexed[AnotherReplacer, data, {-1}]

This produces (again)

{{12, 27, 19, 44}, {11, ColumnBasedFn[2], 13, 43}, {ColumnBasedFn[1], 
  56, 23, 60}, {15, 37, 19, 52}, {10, 31, 21, ColumnBasedFn[4]}, {18, 
  40, 22, 43}, {15, 50, 28, 56}, {21, ColumnBasedFn[2], 
  ColumnBasedFn[3], 46}, {20, 39, 25, 51}, {ColumnBasedFn[1], 36, 23, 
  56}, {ColumnBasedFn[1], 27, 21, 39}, {14, 32, 20, 46}, {22, 40, 
  ColumnBasedFn[3], 49}, {ColumnBasedFn[1], 31, ColumnBasedFn[3], 
  44}, {ColumnBasedFn[1], 18, 21, ColumnBasedFn[4]}, {16, 23, 21, 
  34}, {27, 26, 23, 59}, {17, ColumnBasedFn[2], ColumnBasedFn[3], 
  35}, {13, 21, 20, 48}, {11, 51, 18, 54}}
4

@Eric - That works. Thanks for your help.

But rather than x I now have “ColumnBasedFn[x]” where I once had “x”

I feel like I am very close. Attached notebook shows the current situation (earlier file had a problem and has been replaced with one that opens cleanly)

ReplacePartHelp2.nb (27.4 KB)

5

Right. So now you just need to provide a definition for ColumnBasedFn (and probably change the name :slight_smile: ). In your first post above, you were using a NormalDistribution to generate a random value, but I realize that that was just an illustration and that in your real situation the function would be different for each column. Just to illustrate, consider this:

ColumnBasedFn[n_] := RandomVariate[UniformDistribution[{n, n + 1}]]

If you added that and ran my code examples above, you’d get all of your "x"s replaced with a random number from within a range defined by the column number.

I noticed that at the bottom of your notebook you’re doing this replacement (I assume your colFct is equivalent to my ColumnBasedFn):

Transpose[dataMI1][[#]] /. 
 colFct[#] -> 
  RandomVariate[NormalDistribution[mnstd[[#, 1]], mnstd[[#, 2]]]]

But you don’t need to do a replacement, you just need to provide a definition for colFct. Once the colFct expressions are inserted into your data (via either of the methods I demonstrated above), they will get evaluated using whatever definition you provide for colFct. Based on what I see, maybe something like this would work:

colFct[n_] := RandomVariate[NormalDistribution[mnstd[[n, 1]], mnstd[[n, 2]]]]

But I actually haven’t carefully analyzed what you’re doing, so this is probably not quite right.

6

Brilliant. It took me 5 minutes to finish it off. Notebook attached in case you are interested. I particularly appreciate your exposition explaining the code. Often help here (still good of course) just sends the code. But, as the saying goes “If you give a man a fish…”

Thanks. I will probably be back. This is part of a larger project to work out Wolfram code for Full Information Likelihood which I do not believe exists at present.

ReplacePartHelp2.nb (22.1 KB)

7

Hi Roger

data//Transpose//
  Map[With[{r = DeleteCases[#, "x"]},
     # /. "x" -> RandomVariate[NormalDistribution[Mean@r, Variance@r]]]&]//
 Transpose

Robert

8

Tighter code, but all the rvs in a particular column must be differently, even though from the same dist.

9

Hi Roger

ok, just replace → by :>

data// Transpose // 
   Map[With[{r = DeleteCases[#, "x"]}, # /. 
       "x" :> RandomVariate[NormalDistribution[Mean@r, Variance@r]]] &] // 
  Transpose // TableForm

Further you have to take care, that there remain at least two non “x” elements in each column.

Robert

10

Hi Roger,

I think you consequently should use StandardDeviation not Variance.

data// Transpose // 
   Map[With[{r = DeleteCases[#, "x"]}, # /. 
       "x" :> RandomVariate[NormalDistribution[Mean@r, StandardDeviation@r]]] &] // 
  Transpose // TableForm

Robert

11

That fixed it. Thanks
Your last comment is a weakness in my system so far. I have, for simplicity in order to get my head around the problem, assumed at least two non-missing elements in each row. Had not thought about the same affecting columns. Datasets with 19 of the 20 obs missing probably would be eliminated for other reasons before getting this far?

12

I had already made that change since that is the second argument in NormalDistribution[ ] that the documentation calls for.

13

Hi Roger

you could handle the problem of insufficient number of non “x” elements like this:

data // Transpose // 
   Map[With[{r = DeleteCases[#, "x"]}, # /. 
       "x" :> If[Length@r >= 2, RandomVariate[NormalDistribution[Mean@r, StandardDeviation@r]], 
         If[Length@r == 1, r[[1]], 0]]] &] // Transpose // TableForm

If a column has just one non “x” element left, return it as rv if it has no elements return 0 as rv.

Robert