Message Boards

GROUPS:

Staff Picks Biological Sciences Data Science Medical Sciences Curated Data Import and Export Wolfram Language Geographic Information System

Robert Rimmer

[Notebook] Web Scraper: New York Times Coronavirus Data

Robert Rimmer

Posted 7 months ago

2350 Views

3 Replies

11 Total Likes

Follow this post

enter image description here

This notebook scrapes state by state current coronavirus data from the New York Times web page shown in WebExecute[...]. This page is more frequently updated than the GitHub database.

Note: this notebook may have to be revised if web page is changed. Select the desired browser below.

session=StartWebSession[(*"Chrome"*)"Firefox"(*,Visible->False*)];

In[]:=

WebExecute["OpenPage""https://www.nytimes.com/interactive/2020/us/coronavirus-us-cases.html"];

In[]:=

If you don’t have a subscription you may have to scroll down the page to have the ‘Show more’ button appear or you can just click the button manually to get the whole table, rather than find the button and click it with the procedures below.

Chrome XPath
//*[@id=”coronavirus-us-cases”]/div/div/div[3]/div/button

Firefox XPath
/html/body/div[1]/main/article/section/div/div/div[3]/div/button

Clear[element];If[session["Browser"]"Chrome",element=WebExecute["LocateElements""XPath""//*[@id=\"coronavirus-us-cases\"]/div/div/div[3]/div/button"]];If[session["Browser"]"Firefox",element=WebExecute["LocateElements""XPath""/html/body/div[1]/main/article/section/div/div/div[3]/div/button"]];Pause[1];WebExecute["ClickElement"element[[1]]]

In[]:=

Success

✓

Message: ClickElement was successful.

Element: fc2b2a36-5028-494a-a541-a8a801037b06



Out[]=

After the full table is open this can be executed, otherwise only the abbreviated table will be loaded.

table=WebExecute["LocateElements""Tag""table"]

In[]:=

WebElementObject

Browser: Firefox
ElementId: 9ed4fbd2-42f6-45c8-86a4-dc258905b132

,WebElementObject

Browser: Firefox
ElementId: 8fee143d-3f59-4a66-9b4e-f13dac9417be



Out[]=

tableText=WebExecute["ElementText"#]&/@table;

In[]:=

tableText[[1]]

In[]:=

STATE CONFIRMED CASES DEATHSNew York 142 0Washington 141 20California 112 2Massachusetts 28 0Texas 24 0Nebraska 16 0Oregon 13 0Florida 13 2Georgia 11 0New Jersey 11 0Colorado 9 0Illinois 7 0Pennsylvania 7 0South Carolina 6 0Arizona 5 0Maryland 5 0New Hampshire 4 0Nevada 4 0Kentucky 4 0Rhode Island 3 0Tennessee 3 0Virginia 3 0Iowa 3 0Ohio 3 0Utah 2 0North Carolina 2 0Indiana 2 0Minnesota 2 0Hawaii 2 0Wisconsin 1 0Oklahoma 1 0Kansas 1 0District of Columbia 1 0Missouri 1 0Vermont 1 0Connecticut 1 0Louisiana 1 0

Out[]=

tableText[[2]]

In[]:=

HOW VIRUS WAS CONTRACTED CONFIRMED CASESUnknown 258Cluster connected to a community in New Rochelle, N.Y. 54Nursing facility in Kirkland, Wash. 51Diamond Princess cruise ship 43Personal contact in U.S. 39Travel overseas 32Travel in Italy 28Business conference in Boston 22Travel in Egypt 21Grand Princess cruise in February 18

Out[]=

DeleteObject[session]

In[]:=

dataStrs=StringSplit[tableText[[1]],"\n"];data0=StringCases[#,{LetterCharacter..,DigitCharacter..,DigitCharacter..}]&/@dataStrs;data=If[Length[#]>3,{StringJoin@Riffle[Take[#,Length[#]-2]," "],ToExpression@#[[-2]],ToExpression@#[[-1]]},{#[[1]],ToExpression@#[[-2]],ToExpression@#[[-1]]}]&/@data0;nytDataset=AssociationThread[{"State","Cases","Deaths"},#]&/@Rest[data]//Dataset

In[]:=

State	Cases	Deaths
New York	142	0
Washington	141	20
California	112	2
Massachusetts	28	0
Texas	24	0
Nebraska	16	0
Oregon	13	0
Florida	13	2
Georgia	11	0
New Jersey	11	0
Colorado	9	0
Illinois	7	0
Pennsylvania	7	0
South Carolina	6	0
Arizona	5	0
Maryland	5	0
New Hampshire	4	0
Nevada	4	0
Kentucky	4	0
Rhode Island	3	0
showing 1–20 of 37

Out[]=

nytDataset[Total,{"Cases","Deaths"}]

In[]:=

Cases	595
Deaths	24

Out[]=

chartelements=Map[PieChart[{#[[1]],#[[2]]}]&,Values@Normal@nytDataset[All,{"Cases","Deaths"}]];

In[]:=

GeoBubbleChart[Map[If[#[[3]]>0||#[[2]]>15,Callout[Interpreter["USState"][#[[1]]]#[[2]],Column[{Style[#[[1]],Bold],"Cases:",#[[2]],"Deaths:",#[[3]]}],Right],Interpreter["USState"][#[[1]]]#[[2]]]&,Values@Normal@nytDataset[All,{"State","Cases","Deaths"}]],ChartElementschartelements,ImageSize900,GeoRange{{24.,50.},{-130.,-60.}}]

POSTED BY: Robert Rimmer

Answer

3 Replies

Sort By:

Rohit Namjoshi

Posted 7 months ago

There is also this site which is updated frequently.

POSTED BY: Rohit Namjoshi

Answer

Jesse Dohmann

Jesse Dohmann, WOLFRAM

Posted 7 months ago

Very cool use of `WebExecute`! I love seeing it used for web scraping/data visualization applications.

POSTED BY: Jesse Dohmann

Answer

Moderation Team

Moderation Team, WOLFRAM

Posted 7 months ago

-- you have earned *Featured Contributor Badge* Your exceptional post has been selected for our editorial column *Staff Picks* http://wolfr.am/StaffPicks and Your Profile is now distinguished by a *Featured Contributor Badge* and is displayed on the Featured Contributor Board. Thank you, keep it coming, and consider contributing to the The Notebook Archive!

POSTED BY: Moderation Team

Answer

Reply to this discussion

Reply Preview

Attachments

Remove Add a file to this post

Follow this discussion

or Discard

Group Abstract

Feedback

Enable JavaScript to interact with content and submit forms on Wolfram websites. Learn how »