It’s certainly possible (http://blog.wolfram.com/2013/08/13/a-visit-to-disneys-magic-kingdom/) to analyze movies frame by frame. According to that blog post, each movie took a day to process. But I think that Mathematica likes to load everything into memory where possible, so movies are going to be difficult to work with unless you divide them into smaller pieces. Try loading a few frames at a time, as Michael suggested.
It’s also possible that the MOV file has problems - many things go into a MOV container…
I have done something similar to this but rather than importing the .mov file I had better luck exporting each frame of the movie to a folder of still images and than running the analysis on the stills.