that is because you do not follow the standard syntax. The different types of parenthesis mean different things in Mathematica. The square brackets mean input to a function. Your request in Mathematica notation would be like this:
Integrate[x^3/(e^x - 1), x]
This said, there is of course Mathematica's linguistic input. If you type in a double equal sign first and then exactly what you typed you get this:
Alternatively you could start with a single equal sign and use your input. You then get:
You see that Mathematica suggests the standard notation, or its interpretation of your input in standard syntax. You might want to go through some of the videos in the Support and Learning section or watch these videos by Richard Gaylord. If you are rather the reading type of person you might like this book by Paul Wellin, which is an outstanding introduction to Mathematica.
Hope this helps,