A while ago, I wrote a package that might help you with this problem: lazyLists
https://github.com/ssmit1986/lazyLists
The idea of a lazy list is that it only generates elements on request. The full documentation is in the example notebook in the repo, but here's an example of how to deal with your particular problem:
Load the package and get all words:
Needs["lazyLists`"]
words = WordList["CommonWords", Language -> "French"];
Partition the words into chunks of length 100 (the Hold
wrapper is used to indicate that the list is stored in the symbol words
and works as a kind of pointer; it prevents the whole list from being copied over all the time):
partitionedWords = lazyPartition[Hold[words], 100];
Map WordFrequencyData
over the list. The {fun, Listable}
notation tells the package that the function can be applied to an entire chuck with needing to Map
the function over the chunck. I convert the output of WordFrequencyData
into a list with Normal
, because the the package doesn't work with Associations
:
wordFreq = Map[
{Normal @ WordFrequencyData[#, Language -> "French"] &, Listable},
partitionedWords
];
Get the frequency data for the first 1000:
result = Take[wordFreq, 1000];
Most[result]
Most
returns the 1000 elements that have been extracted. Last
can be used to get the "tail" of the list and continue evaluating where you left off. E.g., get the next 10 results:
Most@Take[Last[result], 10]
You can also use Take[..., All]
to go through the whole word list.
Hope this helps!