"The system is inconsistent and correct."

A more accurate statement would be "the system is inconsistent and (therefore) unsolvable". So this model needs to be rethought.

What have you tried? What is unclear about the error message?

OK.

But what do you expect? Are you sure your system (which is inconsistent as pointed out above) is correct?

Do you really mean

h'[t] == ksi/2*((1/(h[t]*h[t] + 1/4)^3/2) - 1)

or perhaps

h'[t] == ksi/2*((1/(h[t]*h[t] + 1/4)^(3/2)) - 1)

How did you arrive at your system of equations? What is the underlying physics?

But you can do the following. Your system is incosistent, but you could ask for another function u. Redefine your system and solve it numerically (NDSolve)

ksi = 1;
system = {
   h'[t] == ksi/2*((1/(h[t]*h[t] + 1/4)^3/2) - 1),
   h[0] == 0,
   u'[t] == -((1/(h[t]*h[t] + 1/4)^3/2) - 1)*h[t],
   u[0] == 0
   };
sol = NDSolve[system, {h, u}, {t, 0, 10}] // Flatten;

Get the functions h and u

h = h /. sol;
u = u /. sol;

And their derivatives

hp = Derivative[1][h ];
up = Derivative[1][u];

Calculate values of these at certain values of t

hppts = Table[{t, hp[t]}, {t, 0, 2, .05}];
uppts = Table[{t, up[t]}, {t, 0, .6, .05}];

and look whether these values fit to the right-hand sides of your differential equation(s)

Plot[ksi/2*((1/(h[t]*h[t] + 1/4)^3/2) - 1), {t, 0, 1}, 
 PlotRange ->All,  Epilog -> {Red, PointSize[.015], Point /@ hppts}]

and

Plot[-((1/(h[t]*h[t] + 1/4)^3/2) - 1)*h[t], {t, 0, .6}, 
 PlotRange -> All, Epilog -> {Red, PointSize[.015], Point /@ uppts}]

Welcome to Wolfram Community! Please make sure you know the rules: https://wolfr.am/READ-1ST

The rules explain how to format your code properly. If you do not format code, it may become corrupted and useless to other members. Please EDIT your post and make sure code blocks start on a new paragraph and look framed and colored like this.

int = Integrate[1/(x^3 - 1), x];
Map[Framed, int, Infinity]

The "pendulum" described above is sort of unphysical. The two huge masses can't stay at rest, they should be attracted by gravitation. So here is another system, three masses (restricted to a plane) but following Newtons law

n = 3;(* number of masses  *)
dim = 2;  (* dimension of space - here a plane  *)
pos = Table[x[i, j][t], {i, 1, n}, {j, 1, dim}];
mass = {10^-4, 1., 1.05};  (* Params of the system  *)
grav = 1.;  (* constant of gravity  *)

force[i_] := 
 - Sum[If[j == i,0, (grav mass[[j]])/ Sqrt[(pos[[i]] - pos[[j]]).(pos[[i]] - pos[[j]])] (pos[[i]] - pos[[j]])], {j, 1, n}]

Initial conditions

initpos = Thread /@ Thread[(pos /. t -> 0) == {{0, 5}, {-2, 0}, {2, 0}}];
initspeed = Thread /@ Thread[(D[pos, t] /. t -> 0) == {{0, .2}, {0, -.15}, {0, .15}}];  

ODE and its solution

sys = Flatten[Join[DGL, initpos, initspeed]];
sol = Flatten[NDSolve[sys, Flatten[pos /. a_[t] -> a], {t, 0, 200}]];
xx = Partition[(Flatten[pos]) /. sol, dim];

and look at it

Animate[
 Graphics[{PointSize[.05], Point /@ xx /. t -> tt}, 
  PlotRange -> {{-6, 6}, {-10, 10}}],
 {tt, 0, 50}]

It seems that the small mass finally picks up so much speed that it leaves the system

i have not reviewed the ODE properties of the 2 equations i'm not swift at reading them in mathematica yet

i suggest if there is some discussion over validity that the original poster plots (isoclines) of the two (differential equations) and see visually (though it's possible to be "tricked" by such plots)

https://reference.wolfram.com/language/howto/PlotTheResultsOfNDSolve.html

EquationTrekker

i would say if Hans Dolhaine said there is a problem then to believe him

As I consider the restricted three-body problem, three bodies form an equilateral triangle. Variable h(t) is height of this equilateral triangle and massless body is on vertex of this triangle. The differential equations system describes the dynamics of the restricted three-body problem.

The system is inconsistent and correct. I expect to plot numerical values of functions h[t] and u[t].

Thanks a lot, I am trying to work out it.

Could you show how to plot functions h[t] and u[t] please?