# Asynchronous Import (MultipleDownload function)

**URL:** <https://community.wolfram.com/t/asynchronous-import-multipledownload-function/10468>\
**Category:** Ideas\
**Tags:** computer-science, software-development, mathematica, import-and-export, computational-linguistics\
**Created:** [May 3, 2017, 4:08pm UTC](https://community.wolfram.com/t/asynchronous-import-multipledownload-function/10468 "2017-05-03T16:08:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![thalesfdfernandes](https://yyz1.discourse-cdn.com/wolfram/user_avatar/community.wolfram.com/thalesfdfernandes/32/24924_2.png) [@thalesfdfernandes](https://community.wolfram.com/u/thalesfdfernandes)\
**Post date:** [May 3, 2017, 4:08pm UTC](https://community.wolfram.com/t/asynchronous-import-multipledownload-function/10468/1 "2017-05-03T16:08:43Z")

</div>

Suppose you want to download multiples files from the internet, perhaps data from a dictionary or something like that. Using multiples `Import` would be extremely slow since the download would be synchronous (one at a time).

You can use Mathematica function `URLSubmit` asynchronous request (multiples at a time). The only “drawback” is that you need to wait for the request to be done (you can still run part of your code while it is downloading).

With the simple code bellow, you can download multiple files with a given extension and parse the result, returning a list of data.

```
Options@MultipleDownload = {TimeConstraint -> 30};
MultipleDownload[urls:{__String}, fileType_String, parseFun_, OptionsPattern[]] := Module[{L = {}},
	Function[{url}, URLSubmit[url, HandlerFunctionsKeys -> {"Body"},
		HandlerFunctions -> <|"BodyReceived" -> (AppendTo[L, parseFun@ImportString[First@#1["Body"], fileType]] &)|>]
	] /@ urls;
	TimeConstrained[While[Length@L < Length@urls,
		Pause[1];
		PrintTemporary[{Length@L, Length@urls}]
	], OptionValue@TimeConstraint];
	
	L
]

```

Where it basicallyes make a call to URLSubmit, fetch the data, parse it and append to a temporary list L. Then wait until the list is full or we have run out of time. It does no error handling, so use with care.

Real world usage would be (getting the pronunciation respelling from [dictionary.com](http://dictionary.com)):

```
words = {"pig", "people", "math", "car", "book"};
urls = StringTemplate["http://www.dictionary.com/browse/`1`"] /@ words;
data = MultipleDownload[urls, "XMLObject",
	Function[{xml},
      First@Cases[xml, a:XMLElement["span", {"class" -> "pron spellpron"}, {L__}] :>
      StringTrim@ImportString[ExportString[a, "XML"], "HTML"], \[Infinity]]
    ]
]
(* Output *)
{"[kahr]", "[pig]", "[pee -p uh l]", "[math]", "[b oo k]"}

```

PS: The function posted is only for academic or personal usage. No content from the original website should ever be used for monetary gain or distributed.

---

<div class="post-metadata">

**Author:** ![vincentvirgilio](https://avatars.discourse-cdn.com/v4/letter/v/97f17d/32.png) [@vincentvirgilio](https://community.wolfram.com/u/vincentvirgilio)\
**Post date:** [May 3, 2017, 4:45pm UTC](https://community.wolfram.com/t/asynchronous-import-multipledownload-function/10468/2 "2017-05-03T16:45:12Z")

</div>

I’ve used URLSaveAsynchronous to do similar. Then create a rudimentary dashboard of active jobs with the below.

Dynamic[Refresh[AsynchronousTasks[]//Column,UpdateInterval-\>1]]

---

<div class="post-metadata">

**Author:** ![thalesfdfernandes](https://yyz1.discourse-cdn.com/wolfram/user_avatar/community.wolfram.com/thalesfdfernandes/32/24924_2.png) [@thalesfdfernandes](https://community.wolfram.com/u/thalesfdfernandes)\
**Post date:** [May 3, 2017, 5:24pm UTC](https://community.wolfram.com/t/asynchronous-import-multipledownload-function/10468/3 "2017-05-03T17:24:52Z")

</div>

> I’ve used URLSaveAsynchronous to do similar. Then create a rudimentary dashboard of active jobs with the below.

My first go at this “problem” was to use it too. But I needed to download hundreds or even thousands of HTML only to get minimal data from it. So, to save it and parse every single file was a waste of space, thus I decided to just parse the data on memory and save it later.