The syntax X = X[\[Xi]_] := f[\[Xi]] does not do what you expect. The same for (Y^i)[\[Xi]]. I have tried to clean up your code until I had to go:

ClearAll["Global`*"]
eq1 = f''[\[Xi]] == (c/(c^2 - 1))*f'[\[Xi]] - (\[Alpha]/(c^2 - 1))*
     f[\[Xi]] - (\[Beta]/(c^2 - 1))*f[\[Xi]]^3;
repl = {X' -> Y, X'' -> Y'};
eq3 = eq1 /. f -> X /. repl
sm = Sum[a[i][X[\[Xi]]]*Y[\[Xi]]^i, {i, 0, m}];
qq = (h[X[\[Xi]]] + g[X[\[Xi]]]*Y[\[Xi]])*sm
ss = D[sm, \[Xi]]
eq6 = (qq - ss) /. m -> 1 /. repl
assumptions = {a[1] -> Function[1], g -> Function[0], 
   h -> Function[x, A*x + B]};
eq6 /. assumptions /. Solve[eq3, Y'[\[Xi]]][[1]]