Dear Terrance, this behavior is strange because the automation is supposed to discard models that give bad probabilities like this, but it can happen in small dataset. Most likely the "RandomForest" method is selected here (you can obtain such information by doing ClassifierInformation[c] ), and this method is not great to give good probabilities.

You can try to improve the model quality by doing

Classify[..., PerformanceGoal ->"Quality"] 

or setting up a time goal, e.g.

Classify[..., TimeGoal -> 10]

or choosing a method yourself, e.g.

Classify[..., Method->"LogisticRegression"]

Finally, you can try to add more data, this should help the automation to do the right thing by testing on larger test sets.

Thanks, Etienne