I do have experience with importing large Excel, SQL, or text data sets in Mathematica and using classification or regression methods over that data.

I have described one such activity, similar to the scenario you outlined, in this blog post: http://mathematicaforprediction.wordpress.com/2014/03/30/classification-and-association-rules-for-census-income-data/ .

And here is a blog post showing analysis using Mathematica's weather data access functions: http://mathematicaforprediction.wordpress.com/2014/01/13/estimation-of-conditional-density-distributions/ .

I think your frustration and complaint are valid, but you also have them because you have used SAS or STATA, which are domain specific languages, not general systems for mathematical computations and visualization like Mathematica.

Thanks, Pat.

Here are several data reading and importing steps (using the "Adult" dataset) that I hope you will find concrete and useful.

In[4]:= lines = Import["~/Data sets/adult/adult.data"];
lines = Select[lines, Length[#] > 3 &];
Dimensions[lines]

Out[6]= {32561, 15}

In[7]:= linesTest = Import["~/Data sets/adult/adult.test"];
linesTest = Select[linesTest, Length[#] > 3 &];
Dimensions[linesTest]

Out[9]= {16281, 15}

In[10]:= columnNames = 
 StringSplit[
  "age,workclass,fnlwgt,education,education-num,marital-status,occupation,relationship,race,sex,capital-gain,capital-loss,hours-per-week,native-country", ","]

Out[10]= {"age", "workclass", "fnlwgt", "education", "education-num",  "marital-status", "occupation", "relationship", "race", "sex", "capital-gain", "capital-loss", "hours-per-week", "native-country"}

In[11]:= AppendTo[columnNames, "income"]

Out[11]= {"age", "workclass", "fnlwgt", "education", "education-num",  "marital-status", "occupation", "relationship", "race", "sex", "capital-gain", "capital-loss", "hours-per-week", "native-country", "income"}

In[12]:= Magnify[
 TableForm[lines[[1 ;; 12]], 
  TableHeadings -> {Automatic, 
    Style[#, Blue, FontFamily -> "Times"] & /@ columnNames}], 0.9]

Is this post in a direction you would like this discussion to go?

I have attached a notebook to this response that goes through the steps of building a regression model with LinearModelFit and using it for classification. One important question is how to separate the regression model values so we can obtain the best possible classification rates. In the notebook this is done using ROC. (See http://en.wikipedia.org/wiki/Receiver_operating_characteristic .)

LinearModelFit has several signatures. For the data we have I think the most convenient one is LinearModelFit[{m,v}] .

In order to keep the exposition simple in the notebook the regression is done with the two numerical columns "education-num" and "hours-per-week". With the replacement rules {"<=50K"->0,">50K"->1} we convert the data column "income" into a vector of 0's and 1's.

In the attached notebook we call positive the income values ">50K" and negative the income values "<=50K".

The result of LinearModelFit is a function based on the training set of data. We can plot a histogram of values from the regression model, and then we pick a threshold above which the model values are considered to be 1's (and hence ">=50K").

In the attached notebook the first example of using the result of LinearModelFit is extended with a more systematic approach of determining the best threshold to separate the regression model values. The ROC functions Positive Predictive Value (PPV), Negative Predictive Value (NPV), True Positive Value (TPV), accuracy (ACC), and specificity (SPC).

Attachments:

Looking at the table of values at the end of your notebook, instead of

DeleteCases[tstmdl, "na", Infinity]

you should use

DeleteCases[tstmdl, "\"na\"", Infinity]

This though would break the shape of your data, so you might be better of using

tstmdl /. "\"na\"" -> 0

Both commands I mentioned in my previous post worked on the data in the notebook you provided.

I did not think the code worked because I issued a dimensions command which identified the same dimension. However, I now realize what you mean by messing up the table. So yes, both codes work. I also tried the second version but replacing '0' with Missing[na]. That also worked and I was hoping that if I then issued a Mean command, the program would give the mean based upon non-missing variables. This didn't happen so presumably the program is not seeing Missing[na] as a missing value. So I need to look into this more. Thanks again.

You're correct, which is why I'm spending as much time as I am trying to understand mathematica's language and functionalities.

I have attached an update of the notebook that describes the use of LinearModelFit with ROC. In the updated notebook one more column, "age", of the "Adult" dataset is used to do the regression, and some minor additional comments and links are included.

Attachments: