Thank you both for your comments. However, both of you give different advices on how to localize the variable. But the variable is already localized by being a control variable inside Manipulate. I don't see why it needs to be localized again. SaveDefinition may save something, but it does not change the value of the outside x. I tried it. So the outside x is not the variable Manipulate works with. How come then it uses it's value? I still think this should not happen this way.
However I found a solution. Instead of giving the starting value of x in the controller definition (which as I see it should override everything, but apparently it does not), I initialize x in the initialization part of Manipulate. The following code works for me.
Manipulate[x, {x, 0, 1}, SaveDefinitions -> True, Initialization :> (x = 0.5)]
The value of the outside x is not changed with this version of the code either.