Hello everyone,
I am writing scripts to download data from public websites. I use
Import[URL, "Plaintext"]
or
Import[URL, {"HTML","Data"}]
The scripts work fine as long as I don't make too many calls to the function that is doing the importing. When the import function is housed in a loop, I can make about 1,000 downloads, at which point the loop has been running for half an hour or longer, and then it either stalls or I get an error from Mathematica telling me that I need to check my internet connectivity because the session has timed out. I guess that means that I have overstayed my welcome at the host server.
I could break up my downloads into fewer calls. But is there anything else I can do to a avoid the session timeout and grab all the data I need in one swoop?
Any tips would be much appreciated,
Gregory