Why do you believe that the result should be 0.5? A brute-force check by dividing up the unit square into 10,000^2 little squares and summing up the contributions from the appropriate squares gets me a value of 0.22992. In Mathematica 10 I get the following:
In[4]:= Integrate[x Cos[y], {x, 0, 1}, {y, 0, x^2}]
NIntegrate[x Cos[y], {x, 0, 1}, {y, 0, x^2}]
Out[4]= Sin[1/2]^2
Out[5]= 0.229849
