Gregory: I believe you might be using the starter function I responded to a while back. Below is slight modification just change hard coded year to get most recent year, so this returns the most recent raw text of the DEF14A filing from a list range 1994 to current year for a given ticker

processSECDEF14A[ticker_] := 
  Module[{cik, paddedCIK, urlfullpath, searchResults, textOnlylinks, 
    top1linkfromList, formDEF14A}, cik = FinancialData[ticker, "CIK"];
   paddedCIK = IntegerString[ToExpression[cik], 10, 10];
   urlfullpath = 
    "http://www.sec.gov/cgi-bin/srch-edgar?text=CIK%3D" <> paddedCIK <>
      "+TYPE%3DDEF&first=1994&last=" <> 
     DateString[DateList[], "Year"] <> "";
   searchResults = Import[urlfullpath, "Hyperlinks"];
   textOnlylinks = 
    Select[searchResults, Function[StringMatchQ[#, "*.txt"] == True]];
   top1linkfromList = First[textOnlylinks];
   formDEF14A = Import[top1linkfromList, "Plaintext"];
   Return[formDEF14A];];

I tried this for HEWLETT PACKARD CO

hpDEF14A = processSECDEF14A["HPQ"];

The raw data is as previously stated, SGML,HTML, JPG, PDF combo document. In order find the beneficial ownerships (section16a) there will be some data mining necessary. The raw data HTML segments are created by many different edgar filing service providers (software). But all of them are convoluted HTML so I would suggest working with both the raw file and a cleaner version. The following simple statement will remove all HTML element attributes:

hpDEF14ANoAttribs = StringReplace[hpDEF14A, 
 RegularExpression["(<\\w+)[^>]*(>)"] -> "$1$2"];

You can perform some further cleaning such as removing font elements or any other cleaning necessary. The trick as you are trying to accomplish is to get the HTML Table that contains the beneficial ownership list. Footnote would have to be dealt with later. But using:

ImportString["Table Fragment Here", {"HTML", "Data"}];

I'm still thinking about a way to grab the required table. These HTML files are so complicated that I don't think "FullData" would return a clean data set. StringPosition may come in handy but figuring out the proper algorithm is more critical to find a solution. Something like find html table that contains the word "beneficial", etc

I'm not able to take a deep look at the file right now, but since (X)HTML is a subset of XML, parsing your HTML as XML using Mathematica's XML processing capabilities might make handling the data easier.

Additionally, Mathematica can directly import XHTML, but I'm not sure how easy that is to work with.

Hi Hans, Nice to hear from you, and sorry for my delay in responding. I learned a lot from the code you gave me, and I modified it, not so much to do something different, but to match my own coding style. I agree with what you say about the challenges of extracting the data. The SEC is not always consistent in the way the data is entered, so there will be a lot of work in refining the code to grab compensation tables and ownership tables.

Below is my work-in-progress. Following the code you gave me, this function grabs all of the DEF14A text links by searching for CIK. It then returns a number of results. The ownership table results are still incomplete. If you play with it, run it without displaying the results because the output will be big.

I'm going to try your suggestion about using ImportString.

Regards,

Gregory

processSECDEF14A[cik_] := 
 Module[{paddedCIK, urlFullPath, searchResults, textLinks, numLinks, 
   formDEF14A, formDEF14AHTML, formDEF14ATables,

   companyInfoVarNames,
   companyInfoStartPos,
   companyInfoEndPos,
   companyInfoRaw,
   companyInfoArray,
   companyInfoForThisFiling,
   companyInfoVarPos,
   companyInfoVarData,

   resultsTable, textStartpos, theTables, ownershipTablePos},

  (*Search is by central index key*)

  paddedCIK = IntegerString[ToExpression[cik], 10, 10];
  urlFullPath = 
   "http://www.sec.gov/cgi-bin/srch-edgar?text=CIK%3D" <> paddedCIK <>
     "+TYPE%3DDEF&first=1994&last=2014";

  (*Grab all hyperlinks from search results page*)

  searchResults = Import[urlFullPath, "Hyperlinks"];
  textLinks = 
   DeleteDuplicates[
    Select[searchResults, Function[StringMatchQ[#, "*.txt"] == True]]];
  numLinks = Length[textLinks];

  (*EXTRACT DATA AND PROCESS*)

  (*Names of variables about the filing companies*)

  companyInfoVarNames = {"CENTRAL INDEX KEY", "FILED AS OF DATE", 
    "ACCESSION NUMBER", "CONFORMED SUBMISSION TYPE", 
    "PUBLIC DOCUMENT COUNT", "CONFORMED PERIOD OF REPORT", 
    "DATE AS OF CHANGE", "EFFECTIVENESS DATE", 
    "COMPANY CONFORMED NAME", "STANDARD INDUSTRIAL CLASSIFICATION", 
    "IRS NUMBER", "STATE OF INCORPORATION", "FISCAL YEAR END", 
    "FORM TYPE", "SEC ACT", "SEC FILE NUMBER", "DATE OF NAME CHANGE", 
    "FORMER CONFORMED NAME", "DATE OF NAME CHANGE", "STREET 1", 
    "STREET 2", "CITY", "STATE", "ZIP", "BUSINESS PHONE"};

  (* resultsTable contains tables of information extracted from all \
of the filings. 
  It is created by looping through all of the text links. *)

  resultsTable = Table[
    (*Grab the HTML for DEF14A*)
    (*formDEF14A=Import[theTextLink,
    "Plaintext"]; Not used currently *)

    formDEF14A = Import[theTextLink];(*Default import*)

    formDEF14ATables = 
     Import[theTextLink, {"HTML", 
       "Data"}];(*Import while attempting to grab tables*)

    (*---Company info---*)

    companyInfoStartPos = 
     Flatten[StringPosition[formDEF14A, "<SEC-HEADER>", 1, 
        Overlaps -> False]][[1]];
    companyInfoEndPos = 
     Flatten[StringPosition[formDEF14A, "</SEC-HEADER>", 1, 
        Overlaps -> False]][[1]];
    companyInfoRaw = 
     StringTake[
      formDEF14A, {companyInfoStartPos, companyInfoEndPos}];
    (*Split company info into an array with variable names and data*)

        companyInfoArray = 
     Select[StringTrim /@ 
       StringSplit[
        Select[ReadList[StringToStream[companyInfoRaw], String], 
         StringMatchQ[#, ___ ~~ ":" ~~ ___] && ! 
            StringMatchQ[#, "<" ~~ ___] &], ":"], 
      Length[#] == 2 && #[[2]] =!= "" &];
    (*companyInfoForThisFiling is a list or row for each filing*)

    companyInfoForThisFiling = Table[
      companyInfoVarPos = 
       Position[companyInfoArray, 
        thisCompanyInfoVar];(*Position of the variable in the array*)


      companyInfoVarData = 
       If[companyInfoVarPos =!= {}, 
        companyInfoArray[[companyInfoVarPos[[1, 1]], 
          companyInfoVarPos[[1, 2]] + 1]], 
        "NA"];(*Data extracted for that variable*)

      companyInfoVarData = Which[
        thisCompanyInfoVar =!= "STANDARD INDUSTRIAL CLASSIFICATION", 
        companyInfoVarData,
        thisCompanyInfoVar == "STANDARD INDUSTRIAL CLASSIFICATION" && 
         StringCases[companyInfoVarData, 
           "[" ~~ __ ~~ "]"] =!= {}, {StringTrim[
          StringReplace[companyInfoVarData, "[" ~~ __ ~~ "]" -> ""]], 
         StringCases[companyInfoVarData, 
          "[" ~~ sic__ ~~ "]" -> sic]},
        thisCompanyInfoVar == "STANDARD INDUSTRIAL CLASSIFICATION" && 
         StringCases[companyInfoVarData, 
           "[" ~~ __ ~~ "]"] == {}, {"NA", "NA"}
        ];
      companyInfoVarData,
      {thisCompanyInfoVar, 
       companyInfoVarNames}];(*End of processing the company info for \
this filing*)

    (*---Ownership info---*)
    (*Work in progress*)

    ownershipTablePos = 
     StringPosition[formDEF14A, 
      "SECURITY OWNERSHIP OF CERTAIN BENEFICIAL OWNERS AND \
MANAGEMENT"];

    (*Each of the numLinks rows of resultsTable contains the \
following*)
    {
     formDEF14A,(*The default import of DEF14A in case it needs to be \
examined*)

     companyInfoArray,(*Raw array of company info for debugging; 
     can be removed later*)

     Flatten[companyInfoForThisFiling],(*A neat table of company \
information*)

     ownershipTablePos,(*A stab at the position of the ownership \
table if it exists [in progress]*)

     formDEF14ATables(*A stab a grab any data that is tabular*)
     },
    {theTextLink, textLinks}];(*End resultsTable*)

  (*RESULTS RETURNED BY THE FUNCTION*)
  {
   resultsTable[[All, 1]](*formDEF14A*),
   resultsTable[[All, 2]](*companyInfoArray for debugging*),
   resultsTable[[All, 3]](*Company info table*),
   resultsTable[[All, 4]](*Ownership table position*),
   resultsTable[[All, 5]](*form DEF14A imported as HTML data [
   tables]*),
   numLinks (*Number of text links that were found on the search \
results page*),
   textLinks (*Text links that that were found on the search results \
page*)
   }
  ]