Sure

onesCount // Select[Length@#[[2]] > 0 && #[[2, 2]] == rows - 1 &] // Map[First]
(* {1, 2, 62, 82, 96} *)

With some helper functions

countsOf[n_, matrix_] := 
 Table[{i, Flatten@Tally[Select[matrix[[All, i]], # == n &]]}, {i, 1, Last@Dimensions@matrix}]

withCount[n_] := 
 Select[Length@#[[2]] > 0 && #[[2, 2]] == n &] /* Map[First]

Index of columns in colmat that have 10 2's

countsOf[2, colmat] // withCount[10]
(* {97, 100, 110, 129, 139, 143, 149, 150, 153, 174, 205} *)

Yeah I got it now. I understood your comment. Thanks a million:) One last question please if we could re-shape the output to be only 1, 2, 62, 82, 96 without the number of ones in each column. couldn't thank you more. You really saved my day.

I don't want to know the position of ones in terms of (row, column)

Here is one way

{rows, columns} = Dimensions@colmat

(* Create list of column number and number of ones in the column *)
onesCount = 
 Table[{i, Flatten@Tally[Select[colmat[[All, i]], # == 1 &]]}, {i, 1, columns}]

(* Select the ones that have a count equal to the number of rows *)
onesCount // Select[Length@#[[2]] > 0 && #[[2, 2]] == rows &]

(* {} *}

(* Columns that have one value not equal to one *)
onesCount // Select[Length@#[[2]] > 0 && #[[2, 2]] == rows - 1 &]

(* {{1, {1, 43}}, {2, {1, 43}}, {62, {1, 43}}, {82, {1, 43}}, {96, {1, 43}}} *)