Hi,

there are some issues with it. First the command is DSolve not Dsolve. Capitalisation is important in Mathematica. Second, you have Three functions that you declare as A,B and C. One of the differential equations contains an F' and F, but none contains a C. So I changed the function to:

DSolve[{A'[t] + ik A[t] - (m + 1)^(.5) B[t] == 0, B'[t] + (m + 1)^(.5) A[t] + m^(.5) F[t] == 0, F'[t] - ik F[t] - m^(.5) B[t] == 0, A[0] == 0, B[0] == 0, F[0] == 0}, {A[t], B[t], F[t]}, t] 

It finds

(*{{A[t] -> 0, B[t] -> 0, F[t] -> 0}}*)

which appears to be the solution.

Best wishes,

M.