Your determinant is nonzero but rather small (10^-7). However, it seems that the problem is symbolic. When Mathematica solves for the second derivatives, unfortunately the expression contains Cot[\[Alpha][0]] and Cot[\[Beta][0]], which are infinite at your initial conditions. One way to avoid the problem is to manually invert the matrix and thus bypass the built-in solver:
eqDiff = {0.098` Cos[\[Beta][t]] Sin[\[Alpha][t]] - \[Alpha][t] -
0.0014074074074074073` Sin[2 \[Beta][t]] Derivative[1][\[Alpha]][
t] Derivative[1][\[Beta]][t] -
1/200 Cos[\[Beta][t]] Derivative[1][\[Beta]][t] Derivative[
1][\[Phi]][t] +
0.010000000000000002` Cos[\[Alpha][t]] Cos[\[Beta][t]] (
Yc^\[Prime]\[Prime])[t] -
0.0035925925925925925` Cos[\[Beta][
t]]^2 (\[Alpha]^\[Prime]\[Prime])[t] -
1/200 Sin[\[Beta][t]]^2 (\[Alpha]^\[Prime]\[Prime])[t] -
1/200 Sin[\[Beta][t]] (\[Phi]^\[Prime]\[Prime])[t] == 0,
0.098` Cos[\[Alpha][t]] Sin[\[Beta][t]] - \[Beta][t] +
0.0014074074074074073` Cos[\[Beta][t]] Sin[\[Beta][
t]] Derivative[1][\[Alpha]][t]^2 +
1/200 Cos[\[Beta][t]] Derivative[1][\[Alpha]][t] Derivative[
1][\[Phi]][t] -
0.010000000000000002` Cos[\[Beta][t]] (Xc^\[Prime]\[Prime])[t] -
0.010000000000000002` Sin[\[Alpha][t]] Sin[\[Beta][t]] (
Yc^\[Prime]\[Prime])[t] -
0.0035925925925925925` (\[Beta]^\[Prime]\[Prime])[t] == 0,
1/200 (-Cos[\[Beta][t]] Derivative[1][\[Alpha]][t] Derivative[
1][\[Beta]][t] -
Sin[\[Beta][t]] (\[Alpha]^\[Prime]\[Prime])[
t] - (\[Phi]^\[Prime]\[Prime])[t]) == 0,
3.` + 10 (t UnitStep[t] -
Sin[t] UnitStep[t] + (2 \[Pi] - t) UnitStep[-2 \[Pi] + t] +
Sin[t] UnitStep[-2 \[Pi] + t]) - 10 Xc[t] +
0.010000000000000002` Sin[\[Beta][t]] Derivative[1][\[Beta]][
t]^2 - 1.1` (Xc^\[Prime]\[Prime])[t] -
0.010000000000000002` Cos[\[Beta][t]] (\[Beta]^\[Prime]\[Prime])[
t] == 0, -Yc[t] - (Yc^\[Prime]\[Prime])[t] -
0.1` (0.1` Cos[\[Beta][t]] Sin[\[Alpha][t]] Derivative[
1][\[Alpha]][t]^2 +
0.2` Cos[\[Alpha][t]] Sin[\[Beta][t]] Derivative[1][\[Alpha]][
t] Derivative[1][\[Beta]][t] +
0.1` Cos[\[Beta][t]] Sin[\[Alpha][t]] Derivative[1][\[Beta]][
t]^2 + (Yc^\[Prime]\[Prime])[t] -
0.1` Cos[\[Alpha][t]] Cos[\[Beta][
t]] (\[Alpha]^\[Prime]\[Prime])[t] +
0.1` Sin[\[Alpha][t]] Sin[\[Beta][
t]] (\[Beta]^\[Prime]\[Prime])[t]) == 0};
inCond = {Xc[0] == 0.3`, Derivative[1][Xc][0] == 0, \[Beta][0] == 0,
Derivative[1][\[Beta]][0] == 0, \[Phi][0] == 0,
Derivative[1][\[Phi]][0] == 200, \[Alpha][0] == 0,
Derivative[1][\[Alpha]][0] == 0, Yc[0] == 0,
Derivative[1][Yc][0] == 0};
vars = {\[Alpha], \[Beta], \[Phi], Yc, Xc};
vars2 = D[Through[vars[t]], t, t];
m = Map[Coefficient[Map[First, eqDiff], #] &, vars2];
b = Map[First, eqDiff] /. Thread[vars2 -> 0];
linSol1 = Inverse[m].(-b);
NDSolve[{vars2 == linSol1, inCond}, vars, {t, 0, 100}]
The built-in linear solvers have the problem:
linSol2 = LinearSolve[m, -b];
linSol3 = Solve[eqDiff, vars2];
{linSol1, linSol2, linSol3} /. t -> 0 /. Solve[inCond]
