Gregory:

I was looking at your previous code where you process the company information segment. I believe that you may need to rewrite your code; particular this segment

   companyInfoArray = 
     Select[StringTrim /@ 
       StringSplit[
        Select[ReadList[StringToStream[companyInfoRaw], String], 
         StringMatchQ[#, ___ ~~ ":" ~~ ___] && ! 
            StringMatchQ[#, "<" ~~ ___] &], ":"], 
      Length[#] == 2 && #[[2]] =!= "" &];

The StringToStream[] is not ever Close[] (close stream). So set the stream (give it a handle) then Close[], so to something like this:

companyInfoStream = StringToStream[companyInfoRaw];
companyInfoArray = 
  Select[StringTrim /@ 
    StringSplit[
     Select[ReadList[companyInfoStream, String], 
      StringMatchQ[#, ___ ~~ ":" ~~ ___] && ! 
         StringMatchQ[#, "<" ~~ ___] &], ":"], 
   Length[#] == 2 && #[[2]] =!= "" &];
Close[companyInfoStream];

At least it may help with memory management. I would start there. Hans

Greg:

If this is the same SEC Form DEF14A data processing then I have a few questions: Are you persisting the *.txt files to your file system? If so URLFetch is working well (would you say)? Then you go back and process each file using Import[] from local file store, Is this correct? Do you save your CSV files in the same directory as your source files (I hope not)?

If this is the SEC Index files, those file types are zipped (compress) so any auto decompress that Mathematica has to do is going to be memory intensive.

If this is the former situation and you are OK with URLFetch and persisting to file system, then I would not go back to Import[] when there are other functions that give you more control such as OpenRead[] and Close[] stream. For HTML document fragments to pull out tables the automated choice is to use ImportString[HTMLFragment, "Data"] , note that the older DEF14A files (year<2001) are more text files then HTML. I would advise that you stop batching, just process each file one at a time, use a bit of error tracking mark files that fail processing, and return to the errored list if any. 14,000 files ~ 4 hours at 1 second processing per file.

I also assumed that your were going to wrap the entire process in a Mathematica package (before you do that, work out the kinks).

Can you share more of your code with us? Without those specifics, it is very hard to diagnose what could be going wrong.