Ok - so with a little bit of changes I can get this to work 99% of the time with 4 facial states dynamically.

-Smile -Frown -Excited -Scowl

Check out the live demo at:

https://www.youtube.com/watch?v=a39LL5wV_A4&feature=youtu.be

All I did was implement validations sets of equal length of the training sets - both being 50 - and set the FeatureExtractor to be "FaceFeatures". This is new to Mathematica 11. I am really excited about how well this works.

c1 = Classify[
  Join[smileData[[1 ;; 50]], neutralData[[1 ;; 50]], 
   scowlData[[1 ;; 50]], excitedData[[51 ;; 100]]], 
  Method -> "NeuralNetwork", PerformanceGoal -> "Quality", 
  FeatureExtractor -> "FaceFeatures", 
  ValidationSet -> 
   Join[smileData[[51 ;; 100]], neutralData[[51 ;; 100]], 
    scowlData[[51 ;; 100]], excitedData[[51 ;; 100]]]]