Again, I believe that most of your problems are simple syntax errors. For example, Mathematica does not use square brackets for grouping, but for function arguments, etc. Use (nested) parentheses instead. For details see the following link.
http://reference.wolfram.com/language/howto/UseBracketsAndBracesCorrectly.html
Also, commas are only used to separate expressions in lists, or function arguments. The comma at the end of the "U=" line is causing a problem. Semicolons are used to separate expressions on the same line, and to supress output. The D[f,x] function differentiates f with respect to its second argument, etc.
You might save yourself some time by scanning through something like Stephen Wolfram's recent introductory book on the Mathematica language, and/or the tutorials in the Mathematica Documentation Center.
I fixed some of the errors, and this may be closer to what you want.
Clear[r, R, y, Y, k, A, B, U, Ur, Uy]
k = ((4*R*r)/((y - Y)^2 + (r + R)^2))^1/2;
A[k_] := EllipticK[k];
B[k_] := EllipticE[k];
U = (R*r)^1/2/(2*Pi*k) ((2 - k^2)*A[k] - 2*B[k]);
Ur = (1/r) D[U, y]
Uy = (-1/r) D[U, r]