Use graphics inside TogglerBar?

Hi,
In my current work im using a table to produce a set of points in a listplot

PointSet[t_]:= Table[{1 /2 (i + j),  (j - i)}, {i, -(1/ 2) (t - 1), (t - 1)/2}, {j, -(1/2) (t - 1), (t - 1)/2}]

Now i wanted to exclude some points so im using.

With[{t = 5}, {TogglerBar[Dynamic[x], 
   Flatten[PointSet[t], 1], 
   Appearance -> "Horizontal" -> {t, t}], Dynamic[x]}]

It worked but when i try to used them inside a Manipulate e.g

Manipulate[{ToggleBar[Dynamic[x],PointSet[t],Plot[f[x],{x,-10,10}]}]

the value of x is updating when i press something in the togglebar but the argument in function f isn’t updating so the function keep equal.
I try putting all in dynamic module and other stuff but it didn’t work.
For last, i want that the listplot of Point Set as a togglebar inside manipulate controls , so the user see a point distribution insted of a bunch of coordinates.

Thank you for any help.