I would first try to figure out what part of Mathematica is becoming too slow: The kernel, the front-end or a combination?
The kernel may be slow down when using too much RAM, or you code may be in need of optimization.
The front-end may become slow due having some cells that are just too large. Rasterizing plots could help. There are tools to help you identify huge cells in you notebook. Too many open notebooks also slow it down.
The combination of kernel / front-end will become slow when you have dynamic content that is being updated too frequently.
