Perhaps try the following

1. Open System Preferences, and select Security and Privacy.
2. Select “allow applications downloaded from anywhere”.
3. Launch Mathematica by double clicking on the program’s icon, wait until it has completed the process and open an empty document. Then, close the program by selecting Quit from the Mathematica menu
4. Again open System Preferences, and select Security and Privacy.
5. Select “allow applications downloaded from MacApp Store and identified developers”.
6. Launch Mathematica by double clicking on the program’s icon

No issues with Mathematica 10.4 and OS X 10.11.4 on either of my computers. As noted elsewhere in the community, there is an issue with the OpenGL rendering for some 3-D graphics, but that happened with the previous iteration of OS X.

Try 'resetting' Mathematica by starting it holding down the option and shift keys. As I recall, this was an issue some time in the past.