Functions in the Wolfram Language, like Sin and Cos, all start with capital letters, and arguments are held within square braces. Knowing this, you would want to use the following to get the answer you'd expect:
D[\[Rho]*Sin[\[Theta]]*Cos[\[Phi]], \[Rho]]
There's a decent list of resources available for learning the language here, if you are interested.
