Here's code that works well for my purposes and handles the missing values by eliminating observations with missing values.

sample := Flatten[ Import[paht\sample.xlsx", "xlsx", HeaderLines -> 1], 1] ; Dimensions[sample] TableForm[sample[[1 ;; 10]]]

missingdata[entry] := Not[MatchQ[entry, {_?NumberQ}]] subsetdta = DeleteCases[sample, _?missingdata] ;

regdata = subsetdta[[All, {7, 20, 3, 4, 2}]]; data = regdata ; lm = LinearModelFit[data, {x1, x2, x3, x4}, {x1, x2, x3, x4}]

Although part this was domain specific, a greater part is familiarity with the appropriate commands and functionality of mathematica. This is why I believe that a few mathematica examples similar to that above would be very helpful to users such as myself that identify the commands need in going from data importing through a multiple regression problem. Thanks for pointing me in the right direction.

It seems to me that you have certain assumptions about Mathematica's behavior based on your experience with R, S, or SAS.

For example, Mathematica does not give the special treatment of Missing[___] in the same way R and S do of NA.

As I mentioned in my first post in this discussion, R, S, and SAS are domain specific languages, their style and structure make more sense after taking a statistics class (or two). Mathematica has both (i) a powerful, general functional programming language, and (ii) functionalities for different scientific and mathematical sub-cultures. Because Mathematica is a general system for mathematical and technological computations, certain out of the box behavior would not fit expectations based upon R, S, or SAS experiences.

Ok. Then I must have made a mistake. I'll redo and let you know. And apologies for all the trouble. Pat

Both commands I mentioned in my previous post worked on the data in the notebook you provided.

I have attached a notebook to this response that goes through the steps of building a regression model with LinearModelFit and using it for classification. One important question is how to separate the regression model values so we can obtain the best possible classification rates. In the notebook this is done using ROC. (See http://en.wikipedia.org/wiki/Receiver_operating_characteristic .)

LinearModelFit has several signatures. For the data we have I think the most convenient one is LinearModelFit[{m,v}] .

In order to keep the exposition simple in the notebook the regression is done with the two numerical columns "education-num" and "hours-per-week". With the replacement rules {"<=50K"->0,">50K"->1} we convert the data column "income" into a vector of 0's and 1's.

In the attached notebook we call positive the income values ">50K" and negative the income values "<=50K".

The result of LinearModelFit is a function based on the training set of data. We can plot a histogram of values from the regression model, and then we pick a threshold above which the model values are considered to be 1's (and hence ">=50K").

In the attached notebook the first example of using the result of LinearModelFit is extended with a more systematic approach of determining the best threshold to separate the regression model values. The ROC functions Positive Predictive Value (PPV), Negative Predictive Value (NPV), True Positive Value (TPV), accuracy (ACC), and specificity (SPC).

Attachments:

I have attached an update of the notebook that describes the use of LinearModelFit with ROC. In the updated notebook one more column, "age", of the "Adult" dataset is used to do the regression, and some minor additional comments and links are included.

Attachments:

Thanks, Pat.

Here are several data reading and importing steps (using the "Adult" dataset) that I hope you will find concrete and useful.

In[4]:= lines = Import["~/Data sets/adult/adult.data"];
lines = Select[lines, Length[#] > 3 &];
Dimensions[lines]

Out[6]= {32561, 15}

In[7]:= linesTest = Import["~/Data sets/adult/adult.test"];
linesTest = Select[linesTest, Length[#] > 3 &];
Dimensions[linesTest]

Out[9]= {16281, 15}

In[10]:= columnNames = 
 StringSplit[
  "age,workclass,fnlwgt,education,education-num,marital-status,occupation,relationship,race,sex,capital-gain,capital-loss,hours-per-week,native-country", ","]

Out[10]= {"age", "workclass", "fnlwgt", "education", "education-num",  "marital-status", "occupation", "relationship", "race", "sex", "capital-gain", "capital-loss", "hours-per-week", "native-country"}

In[11]:= AppendTo[columnNames, "income"]

Out[11]= {"age", "workclass", "fnlwgt", "education", "education-num",  "marital-status", "occupation", "relationship", "race", "sex", "capital-gain", "capital-loss", "hours-per-week", "native-country", "income"}

In[12]:= Magnify[
 TableForm[lines[[1 ;; 12]], 
  TableHeadings -> {Automatic, 
    Style[#, Blue, FontFamily -> "Times"] & /@ columnNames}], 0.9]

Is this post in a direction you would like this discussion to go?

I do have experience with importing large Excel, SQL, or text data sets in Mathematica and using classification or regression methods over that data.

I have described one such activity, similar to the scenario you outlined, in this blog post: http://mathematicaforprediction.wordpress.com/2014/03/30/classification-and-association-rules-for-census-income-data/ .

And here is a blog post showing analysis using Mathematica's weather data access functions: http://mathematicaforprediction.wordpress.com/2014/01/13/estimation-of-conditional-density-distributions/ .

I think your frustration and complaint are valid, but you also have them because you have used SAS or STATA, which are domain specific languages, not general systems for mathematical computations and visualization like Mathematica.