In SystemModeler the functions have lowercase names, and you use () instead of [], below is a modified version of your code that should do the trick:
block my_mod1 "find remainder after division"
Modelica.Blocks.Interfaces.RealInput u "input signal connector" ;
Modelica.Blocks.Interfaces.RealOutput y "output signal connector" ;
equation
y = mod(u, 360);
end my_mod1;