I took the derivative of (sin(x^2)^2 and got 4xsin(x^2)cos(x^2) then i took the derivative of that answer but it was not correct. The correct answer is 2sin(x^4) + 8x^4 cos(x^4). I'm not understanding how to get this answer.

Why don't you just type it in to get an idea?

In[1]:= DSolve[D[f[x], x] == Sin[x^2], f, x]
Out[1]= {{f -> Function[{x}, C[1] + Sqrt[\[Pi]/2] FresnelS[Sqrt[2/\[Pi]] x]]}}

In[5]:= Clear[h, f]
f[x_] := Sqrt[\[Pi]/2] FresnelS[Sqrt[2/\[Pi]] x]
h[x_] := f[x^2]

In[8]:= D[h[x], x, x]
Out[8]= 8 x^4 Cos[x^4] + 2 Sin[x^4]

So Mathematica agrees with the answer your book presents. That's good, but not good enough: For sure, if the solution is given, you are proposed to find the solution path: Try to find the solution without solving the differential equation.

What happens if you don't know f itself, but only it's first derivative?

In[9]:= Clear[h, f]
h[x] := f[x^2]

In[11]:= D[h[x], x, x]
Out[11]= 2 Derivative[1][f][x^2] + 4 x^2 (f^\[Prime]\[Prime])[x^2]

What do you do?

• use the derivative in the first summand, take care about the dependency from x - easy, you get one summand of the final solution here
• take the derivative of the derivative (to get the second one), again take care about the dependency from x

That's the way your teacher intends you to go; all steps can by done inside Mathematica. Go ahead!