Message Boards Message Boards

0
|
3245 Views
|
14 Replies
|
5 Total Likes
View groups...
Share
Share this post:

Remove elements of a matrix from a larger one?

I have two Matrices with sizes 2086,2 & 196,2 ; the smaller matrix objects are some place in the bigger matrix and I want to Remove the objects which are in the smaller matrix from the bigger one; the smaller matrix objects are a table of Spike data I found from my data and the bigger matrix is the matrix of all data I'm working with. I think I can produce a zeroes matrix with equal size with the all data (2086*2) matrix and replace My spike Matrix objects in their true position in zeroes matrix and then subtract the modified zeroes matrix from all data matrix. how can i get to my purpose? I'm so new in coding in Mathematica , so I'm sorry if I couldn't explain my intention clearly. Thank you for helping me.

POSTED BY: Amirhosein Hsa
14 Replies

Hello Amirhosein,

I noticed that the function sList does not work properly. To see this run the code above with n = 5 and look at

sList[#, n] & /@ pos;
%//MatrixForm

You see that in the 3rd row the position of the spike at 12 is missing, that is correct, but there are not 5 positions at the left and right side as it was required. Therefore I wrote an alternative. Use this, but make sure that you define

bL = Length[bigmat];
pL = Length[pos];  

as soon as bigmat and pos are known. The new function is

Clear[sList1]
sList1[kk_, nk_] := Module[{nn},
  nn = nk;
  Which[
   kk == 1,
   t1 = Complement[Table[j, {j, 2, 2 + nn + pL}], pos];
   t1L = Length[t1];
   If[t1L > nn, t1 = Drop[t1, nn - t1L]];
   t1,
   kk == bL,
   t1 = Complement[Table[j, {j, bL - 1 - nn - pL, bL - 1}], pos];
   t1L = Length[t1];
   If[t1L > nn, t1 = Drop[t1, nn - t1L]];
   t1,
   1 < kk <= nn,
   t1 = Complement[Table[Max[j, 1], {j, kk - 1 - nn - pL, kk - 1}], 
     pos];
   t1L = Length[t1];
   t2 = Complement[Table[j, {j, kk + 1, kk + 1 + nn + pL}], pos];
   t2L = Length[t2];
   If[t1L + t2L > 2 nn, t2 = Drop[t2, 2 nn - t1L - t2L]];
   Join[{t1, t2}] // Flatten,
   bL - nn <= kk < bL,
   t1 = Complement[Table[j, {j, kk - 1 - nn - pL, kk - 1}], pos];
   t1L = Length[t1];
   t2 = Complement[Table[Min[j, bL], {j, kk + 1, kk + 1 + nn + pL}], 
     pos];
   t2L = Length[t2];
   t1 = If[t1L + t2L > 2 nn, t1 = Drop[t1, t1L + t2L - 2 nn]];
   Join[{t1, t2}] // Flatten,
   True,
   t1 = Complement[Table[j, {j, kk - 1 - nn - pL, kk - 1}], pos];
   t1L = Length[t1];
   t2 = Complement[Table[j, {j, kk + 1, kk + 1 + nn + pL}], pos];
   t2L = Length[t2];
   If[t1L + t2L > 2 nn, t1 = Drop[t1, t1L - nn]; 
    t2 = Drop[t2, nn - t2L]];
   Join[{t1, t2}] // Flatten
   ]
  ]

With this try

pos
sList1[#, 5] & /@ pos;
% // MatrixForm
POSTED BY: Hans Dolhaine

And if you insist to have exactly nk (in your case nk = 12 ) points on both sides of a spike, which could not be possible if a spike is near the "edges" of your data, you should use this code

Clear[sList];
sList[j_, nk_] := Module[{nn},
  nn = nk;
  jj = j;
  pp = Select[Complement[Table[n, {n, jj - nn, jj + nn}], pos],  And[# > 0, # <= Length[bigmat]] &];
  While[
   Length[pp] < 2 nk,
   nn = nn + 1;
   pp = Select[Complement[Table[n, {n, jj - nn, jj + nn}], pos], 
     And[# > 0, # <= Length[bigmat]] &]
   ]; 
If[Length[pp] > 2 nk,
   If[nk > j, pp = Drop[pp, -1], pp = Drop[pp, 1]]
   ]; pp]

spikepos = {5, 8, 12, 40, 70, 95};
xx = Range[100];
yy = Table[
   If[Or @@ Thread[spikepos == j], 50, RandomReal[{-1, 1}]], {j, 1, 100}];
bigmat = Transpose[{xx, yy}];
ListLinePlot[bigmat, PlotRange -> All]
(*now construct smallmat*)
smallmat = bigmat[[#]] & /@ spikepos

(*with smallmat given,find the positions of peaks in bigmat*)
pos = Flatten[Position[bigmat, #] & /@ smallmat]
(*now get the datapoints around these positions*)
n = 5;(*this should be replaced by 12 as you wanted*)
sublists = sList[#, n] & /@ pos
flists = Interpolation[#] & /@ (bigmat[[#]] & /@ sublists)
xvals = bigmat[[#, 1]] & /@ pos
replacements = Table[{xvals[[i]], flists[[i]][xvals[[i]]]}, {i, 1, Length[pos]}]
newbigmat =  ReplacePart[bigmat, Transpose[{pos, replacements}] /. {x_, y_} -> x -> y];
ListLinePlot[newbigmat]
POSTED BY: Hans Dolhaine

Hi again Mr. Dolhaine. I can’t thank you enough! This code is exactly the one that I needed for my work. Thank you again for all your help.

POSTED BY: Amirhosein Hsa

I had the idea that it could be that another spike lies in the range of plus/minus 12 of your spike-position. in this case you should use the following

    spikepos = {5, 8, 40, 70, 95};
    xx = Range[100];
    yy = Table[
       If[Or @@ Thread[spikepos == j], 50, RandomReal[{-1, 1}]], {j, 1, 100}];
    bigmat = Transpose[{xx, yy}];
    ListLinePlot[bigmat, PlotRange -> All]
    (*now construct smallmat*)
    smallmat = bigmat[[#]] & /@ spikepos

    (*with smallmat given,find the positions of peaks in bigmat*)
    pos = Flatten[Position[bigmat, #] & /@ smallmat]
    (*now get the datapoints around these positions*)
    n = 12;(*this should be replaced by 12 as you wanted*)
    pos1 =  Table[j, {j, # - n, # + n}] & /@ pos;
    % // MatrixForm
    pos2 = Function[x, Select[x, And[# > 0, # <= Length[bigmat]] &]] /@  pos1;
    pos2 // MatrixForm
    pos3 = Complement[#, pos] & /@ pos2;
    sublists = bigmat[[#]] & /@ pos3
    flists = Interpolation[#] & /@ sublists
    xvals = bigmat[[#, 1]] & /@ pos
    replacements = Table[{xvals[[i]], flists[[i]][xvals[[i]]]}, {i, 1, Length[pos]}]
    newbigmat =  ReplacePart[bigmat, Transpose[{pos, replacements}] /. {x_, y_} -> x -> y];
    ListLinePlot[newbigmat, PlotRange -> All]
POSTED BY: Hans Dolhaine

Hello Amirhosein,

I have duplicate points in my data

What does that mean? Besides, it would be useful to see your original data.

I think it's better to not consider the value of point which we want to replace

I think I don't do that. Have a careful look at my code.

I find out that using a 3rd Order (Cubic) polynomial of best fit through 12 points on either side of the spikes then interpolate across the spike

Ok. Here is a code using n = 2 points on both sides of the peak. This makes it easier to look at the code and the results. Just change n to 12 for your purposes. Is this code that what you want?

spikepos = {22, 40, 70};
xx = Range[100];
yy = Table[
   If[Or @@ Thread[spikepos == j], 50, RandomReal[{-1, 1}]], {j, 1, 
    100}];
bigmat = Transpose[{xx, yy}];
ListLinePlot[bigmat, PlotRange -> All]
(*now construct smallmat*)
smallmat = bigmat[[#]] & /@ spikepos

(*with smallmat given, find the positions of peaks in bigmat *)
pos = Flatten[Position[bigmat, #] & /@ smallmat]
(* now get the datapoints around these positions*)
n = 2;  (* this should be replaced by 12 as you wanted  *)
sublists =
 Delete[#, n + 1] & /@ 
  Flatten[bigmat[[Max[1, # - n] ;; Min[Length[bigmat], # + n]]] & /@ 
    pos, {1}]
flists = Interpolation[#] & /@ sublists
xvals = bigmat[[#, 1]] & /@ pos
replacements = 
 Table[{xvals[[i]], flists[[i]][xvals[[i]]]}, {i, 1, Length[pos]}]
newbigmat = 
  ReplacePart[bigmat, 
   Transpose[{pos, replacements}] /. {x_, y_} -> x -> y];
ListLinePlot[newbigmat]
POSTED BY: Hans Dolhaine

A bit more instructive would be to use

xx = Range[10];
yy = Table[
   If[Or @@ Thread[{2, 5, 9} == j], 300, 
    j^2 + RandomReal[{-10, 10}]], {j, 1, 10}];
bigmat = Transpose[{xx, yy}]
POSTED BY: Hans Dolhaine

For numerical data tehre are some minor modifications necessary:

(* Original data *)

xx = Range[10];
yy = Table[j^2 + RandomReal[{-10, 10}], {j, 1, 10}];
bigmat = Transpose[{xx, yy}]
smallmat = bigmat[[#]] & /@ {2, 5, 9}
pl1 = ListLinePlot[bigmat, PlotRange -> {-10, 110}, 
  AxesOrigin -> {0, 0}, PlotStyle -> Black]

(*  "subtract" smallmat from bigmat  *)

cc = Complement[bigmat, smallmat]
cc1 = Complement[bigmat, smallmat, 
  SameTest -> (Sqrt[(#1 - #2).(#1 - #2)] < .0001 &)]

(*method average around spike*)

index = Function[{x}, Position[bigmat, x][[1, 1]]] /@ smallmat
sublists = 
 DeleteCases[bigmat[[Max[1, # - 2] ;; Min[Length[bigmat], # + 2]]], 
    bigmat[[#]]] & /@ index
meanvals = 
  Function[{x}, Total[Transpose[x][[2]]]/Length[x]] /@ sublists;
(*add the x-vals*)
corrvals = Transpose[{bigmat[[#]][[1]] & /@ index, meanvals}]
rule = Table[smallmat[[i]] -> corrvals[[i]], {i, 1, Length[smallmat]}]
bigmatnew = bigmat /. rule
pl2 = ListLinePlot[bigmatnew, PlotRange -> {-10, 110}, 
  AxesOrigin -> {0, 0}, PlotStyle -> Red]

(* method with interpolation *)

index = Function[x, Position[bigmat, x][[1, 1]]] /@ smallmat
index1 = {Max[1, # - 1], #, Min[Length[bigmat], # + 1]} & /@ index
val = Map[bigmat[[#]] &, index1, {2}]
nval = Function[{a, b, c},
   {b[[1]], 
    a[[2]] (b[[1]] - c[[1]])/(a[[1]] - c[[1]]) + 
     c[[2]] (b[[1]] - a[[1]])/(c[[1]] - a[[1]])}] @@@ val
rule = Table[smallmat[[i]] -> nval[[i]], {i, 1, Length[smallmat]}]
bigmatnew = bigmat /. rule
pl3 = ListLinePlot[bigmatnew, PlotRange -> {-10, 110}, 
  AxesOrigin -> {0, 0}, PlotStyle -> Blue]

(* show them all *)

Show[pl1, pl2, pl3]
POSTED BY: Hans Dolhaine
Posted 3 years ago

Hi sir, Thank you for your help. I used both methods you mentioned for replacement and none of them gave satisfactory results. About the interpolation method, I have duplicate points in my data, and in these specific places in my data, the method couldn't find a good replacement value for my data. And as you said the average method was not good either.
I checked about the best replacement method for my data and I find out that using a 3rd Order (Cubic) polynomial of best fit through 12 points on either side of the spikes then interpolate across the spike with this polynomial is the most satisfactory method. Could you please help me to write the code?

POSTED BY: Updating Name

about the average method you wrote, I think it's better to not consider the value of point which we want to replace in the calculation of average so I modified it to just consider the points around the specific point we want to replace. again thank you for your help.

bigmat = Table[{x[i], y[i]}, {i, 10}];
smallmat = {{2, 2}, {5, 5}, {9, 9}} /. {a_, b_} -> {x[a], y[b]};
(*positions of spikes*)
index = Function[{x}, Position[bigmat, x][[1, 1]]] /@ smallmat;    
(*values around these positions*)
    sublists1=DeleteCases[bigmat[[Max[1,#-2];;Min[Length[bigmat],#-1]]],{x[#],y[#]}]&/@index ;
    sublists2=DeleteCases[bigmat[[Max[1,#+1];;Min[Length[bigmat],#+2]]],{x[#],y[#]}]&/@index ;
    (*calculate the means*)
    meanvals1=Function[{x},Total[Transpose[x][[2]]]/Length[x]]/@sublists1 ;
    meanvals2=Function[{x},Total[Transpose[x][[2]]]/Length[x]]/@sublists2 ;
    (*add the x-vals*)
    corrvals=Transpose[{bigmat[[#]][[1]]& /@index,(meanvals1+meanvals2)/2}] ;
    rule=Table[smallmat[[i]]->corrvals[[i]],{i,1,Length[smallmat]}] ;
    bigmatnew=bigmat/.rule 
POSTED BY: Amirhosein Hsa

Do you mean something like this?

bigmat = Table[{i, i}, {i, 10}]
smallmat = {{3, 3}, {5, 5}, {9, 9}}
Complement[bigmat, smallmat]
POSTED BY: Hans Dolhaine

Hi sir, Thank You very much for your response, yes I think the Function I need is the function you said, Complement. I had a problem when I used this function that is I checked and the intersection of two matrices size is equal to smaller matrix means that every object in smaller one is an object of bigger one, but this Complement function gives me 1812 result points instead of the difference of this two matrices namely 2086-169 = 1917 points; is there any problem I didn't get it or no?? Again thank you for your help.

POSTED BY: Amirhosein Hsa
Posted 3 years ago

OK.

I assume your matrices are x,y values. May be you don't want to just throw away your spikes, but substitute them by appropriate means of surrounding values (here 2 to the left and 2 to the right, at least far from the edges). Look at this

bigmat = Table[{x[i], y[i]}, {i, 10}]
smallmat = {{2, 2}, {5, 5}, {9, 9}} /. {a_, b_} -> {x[a], y[b]}
(*positions of spikes *)
index = Function[{x}, Position[bigmat, x][[1, 1]]] /@ smallmat
(*values around these positions*)
sublists = DeleteCases[
    bigmat[[Max[1, # - 2] ;; Min[Length[bigmat], # + 2]]],
    {x[#], y[#]}] & /@ index
(* calculate the means*)
meanvals = 
 Function[{x}, Total[Transpose[x][[2]]]/Length[x]] /@ sublists
(* add the x-vals*)
corrvals = Transpose[{x[#] & /@ index, meanvals}]
rule = Table[smallmat[[i]] -> corrvals[[i]], {i, 1, Length[smallmat]}]
bigmatnew = bigmat /. rule
POSTED BY: Updating Name

Hi, Thank you very much for your awesome response, it will gonna help me so much.

POSTED BY: Amirhosein Hsa

Thank you.

It occurred to me that it is perhaps better to use a linear interpolation between two points neighbouring the spikes instead to take the mean of several points around the spike? What do you think?

Have a look at the following code

bigmat = Table[{x[i], y[i]}, {i, 10}]
smallmat = {{2, 2}, {5, 5}, {9, 9}} /. {a_, b_} -> {x[a], y[b]}
(*positions of spikes *)
index = Function[{x}, Position[bigmat, x][[1, 1]]] /@ smallmat
index1 = {Max[1, # - 1], #, Min[Length[bigmat], # + 1]} & /@ index
(*values around these positions*)
val = Map[bigmat[[#]] &, index1, {2}]
(* do the interpolation *)
nval = Function[{a, b, c},
   {b[[1]],  a[[2]] (b[[1]] - c[[1]])/(a[[1]] - c[[1]]) + c[[2]] (b[[1]] - a[[1]])/(c[[1]] - a[[1]])}] @@@ val
rule = Table[smallmat[[i]] -> nval[[i]], {i, 1, Length[smallmat]}]
bigmatnew = bigmat /. rule
% // Simplify

You should test it and decide which method is more appropriate.

And in fact I don't understand your issue with Complement:

Length[Complement[bigmat, smallmat]]
Length[bigmat]
Length[smallmat]

Are there duplicates in your data?

POSTED BY: Hans Dolhaine
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract