I have attached a notebook to this response that goes through the steps of building a regression model with LinearModelFit and using it for classification. One important question is how to separate the regression model values so we can obtain the best possible classification rates. In the notebook this is done using ROC. (See http://en.wikipedia.org/wiki/Receiver_operating_characteristic .)

LinearModelFit has several signatures. For the data we have I think the most convenient one is LinearModelFit[{m,v}] .

In order to keep the exposition simple in the notebook the regression is done with the two numerical columns "education-num" and "hours-per-week". With the replacement rules {"<=50K"->0,">50K"->1} we convert the data column "income" into a vector of 0's and 1's.

In the attached notebook we call positive the income values ">50K" and negative the income values "<=50K".

The result of LinearModelFit is a function based on the training set of data. We can plot a histogram of values from the regression model, and then we pick a threshold above which the model values are considered to be 1's (and hence ">=50K").

In the attached notebook the first example of using the result of LinearModelFit is extended with a more systematic approach of determining the best threshold to separate the regression model values. The ROC functions Positive Predictive Value (PPV), Negative Predictive Value (NPV), True Positive Value (TPV), accuracy (ACC), and specificity (SPC).

Attachments:

Anton, thanks for your reply and the references. I am familiar with the examples similar to the weather data and the census example is much closer to what I have in mind. I will be working through this. I understand that the software programs I mentioned are domain specific and that Mathematica is closer to Gauss or Matlab. Yet even for the domain specific software, one must understand the specifics of how data are read in etc. Once inside the 'box' with one's data, it then becomes much easier to exploit the functionality of the program. Pat

Anton, thanks for the additional code and output, again very helpful in answering my original question. Also, I tried your suggestion on the delete cases but that didn't work. I'm attaching the output. The code is by no means elegant but this is a work in progress. Pat

Attachments:

Thanks. Unfortunately, neither of these works. I'm currently exploring SemanticImport which imports the data correctly and recognizes the headers. But again having trouble with the missing values. If I give the command

testdata = SemanticImport["path\test.xlsx"], the data import correctly including variable names. But when I issue the command (or some slight variations),

testdata = SemanticImport["path\test.xlsx" , "MissingDataRules" -> <|"wage" -> {"na" -> Missing["xx"]}, "lwage" -> {"na" -> Missing["rr"]}|>]

trying to replace na with a mathematica missing value (Missing [xx] and Missing [rr]), the program returns a $Failed. Still stuck in a missing value limbo. Pat

Ok. Then I must have made a mistake. I'll redo and let you know. And apologies for all the trouble. Pat

It seems to me that you have certain assumptions about Mathematica's behavior based on your experience with R, S, or SAS.

For example, Mathematica does not give the special treatment of Missing[___] in the same way R and S do of NA.

As I mentioned in my first post in this discussion, R, S, and SAS are domain specific languages, their style and structure make more sense after taking a statistics class (or two). Mathematica has both (i) a powerful, general functional programming language, and (ii) functionalities for different scientific and mathematical sub-cultures. Because Mathematica is a general system for mathematical and technological computations, certain out of the box behavior would not fit expectations based upon R, S, or SAS experiences.

Here's code that works well for my purposes and handles the missing values by eliminating observations with missing values.

sample := Flatten[ Import[paht\sample.xlsx", "xlsx", HeaderLines -> 1], 1] ; Dimensions[sample] TableForm[sample[[1 ;; 10]]]

missingdata[entry] := Not[MatchQ[entry, {_?NumberQ}]] subsetdta = DeleteCases[sample, _?missingdata] ;

regdata = subsetdta[[All, {7, 20, 3, 4, 2}]]; data = regdata ; lm = LinearModelFit[data, {x1, x2, x3, x4}, {x1, x2, x3, x4}]

Although part this was domain specific, a greater part is familiarity with the appropriate commands and functionality of mathematica. This is why I believe that a few mathematica examples similar to that above would be very helpful to users such as myself that identify the commands need in going from data importing through a multiple regression problem. Thanks for pointing me in the right direction.