If such vectors a and b exist, then a.b should be -1/2 provided the context is a real vector space with an inner product (so it doesn't have to be vectors in the plane with the standard dot product). The usual trick on that sort of problem is to square both equations, use the properties of the inner/dot product to break down the left-hand sides into norms and the inner product <a,b> (or a.b), and then add/subtract the equations to eliminate the terms you don't want (in this case the norms/sizes).
I was able to translate that into code (somewhat clumsily) by treating a real vector kx as the list {k,x}, defining a function IP to be the inner product, and then setting the usual properties of the inner/dot product (including relating it to a "Size") for IP:
Clear[IP, Size]
IP[ {1, x_}, {1, y_ + z_}] := IP[{1, x}, {1, y}] + IP[{1, x}, {1, z}]
IP[ {1, x_}, {1, y_ - z_}] := IP[{1, x}, {1, y}] - IP[{1, x}, {1, z}]
IP[{1, x_ + y_}, {1, z_}] := IP[ {1, x}, {1, z}] + IP[{1, y}, {1, z}]
IP[{1, x_ - y_}, {1, z_}] := IP[ {1, x}, {1, z}] - IP[{1, y}, {1, z}]
IP[ {s_, x_}, {t_ , y_}] := s IP[ {1, x}, {t, y}] /; s != 1
IP[{s, x_}, {t_, y_}] := t IP[ {s, x}, {1, y}] /; t != 1
IP[ {1, x_}, {1, x_}] := Size[{1, x}]^2
Attributes[ IP] = {Orderless};
equation1 = IP[ {1, a + b}, {1, a + b}] == 1^2;
equation2 = IP[ {1, a - b}, {1, a - b}] == (Sqrt[3])^2;
Solve[ {equation1, equation2, Size[ {1, a}] >= 0,
Size[{1, b}] >= 0}, Reals]
So this gives the value of a.b (written as the horribly cumbersome IP[{1,a}, {1,b}]) as -1/2. Note that conditions in the solution (basically |a|^2+|b|^2=2) aren't if and only if; for example {1,b} really can't have size 0 as it would be the zero vector, in which case the original equations become |a| is both 1 and Sqrt[3] at the same time.
