Message Boards Message Boards

Scrape dynamic data from a webpage that has JS content changing over time?

Posted 5 years ago

Hi all

Does anyone know how to scrape data from a webpage that has javascript content whose value is dynamic and changes over time (such as pages containing stock prices)? If I try to use

Import[url,"Data"]

the javascript data cannot be imported.

POSTED BY: bob tan

Sometimes the web sites use JSON files, so I look for ".json" file path in the source of the website: enter image description here

In[23]:= ClearAll["Global`*"]

In[24]:= url = 
  "https://www.masslottery.com/data/json/games/lottery/17_0.json";

In[25]:= data = Import[url, {"Data"}];

In[26]:= table = data[[8, 2]]

Out[26]= {{"draw_id" -> "19048", "draw_date" -> "05/28/2019", 
  "draw_date_name" -> "Tuesday", "winning_num" -> "5-4-4-3", 
  "payout1" -> "$5,929", "payout2" -> "$830", "payout3" -> "$71", 
  "payout4" -> "$7", "payout5" -> "$494", "payout6" -> "$277", 
  "payout7" -> "$277", "draw_type" -> "0", "session" -> "Evening", 
  "next_draw_date" -> "05/29/2019", 
  "next_draw_date_name" -> "Wednesday", 
  "create_datetime" -> "2019-05-28 19:57:16", 
  "modify_datetime" -> "2019-05-28 19:57:16", 
  "video" -> "https://youtu.be/bYVdYiP-o6E", 
  "video_id" -> "bYVdYiP-o6E"}, {"draw_id" -> "19047", 
  "draw_date" -> "05/28/2019", "draw_date_name" -> "Tuesday", 
  "winning_num" -> "3-8-0-1", "payout1" -> "$5,023", 
  "payout2" -> "$703", "payout3" -> "$60", "payout4" -> "$6", 
  "payout5" -> "$209", "payout6" -> "$117", "payout7" -> "$117", 
  "draw_type" -> "1", "session" -> "Mid-Day", 
  "next_draw_date" -> "05/29/2019", 
  "next_draw_date_name" -> "Wednesday", 
  "create_datetime" -> "2019-05-28 12:56:39", 
  "modify_datetime" -> "2019-05-28 12:56:39", 
  "video" -> "https://youtu.be/OV4d8hAAvZs", 
  "video_id" -> "OV4d8hAAvZs"}, {"draw_id" -> "19046", 
  "draw_date" -> "05/27/2019", "draw_date_name" -> "Monday", 
  "winning_num" -> "6-5-9-0", "payout1" -> "$4,928", 
  "payout2" -> "$690", "payout3" -> "$59", "payout4" -> "$6", 
  "payout5" -> "$205", "payout6" -> "$115", "payout7" -> "$115", 
  "draw_type" -> "0", "session" -> "Evening", 
  "next_draw_date" -> "05/28/2019", 
  "next_draw_date_name" -> "Tuesday", 
  "create_datetime" -> "2019-05-27 19:54:22", 
  "modify_datetime" -> "2019-05-27 19:54:22", 
  "video" -> "https://youtu.be/5v-hcG7QOKk", 
  "video_id" -> "5v-hcG7QOKk"}, {"draw_id" -> "19045", 
  "draw_date" -> "05/27/2019", "draw_date_name" -> "Monday", 
  "winning_num" -> "5-4-1-0", "payout1" -> "$5,318", 
  "payout2" -> "$745", "payout3" -> "$64", "payout4" -> "$6", 
  "payout5" -> "$222", "payout6" -> "$124", "payout7" -> "$124", 
  "draw_type" -> "1", "session" -> "Mid-Day", 
  "next_draw_date" -> "05/28/2019", 
  "next_draw_date_name" -> "Tuesday", 
  "create_datetime" -> "2019-05-27 13:02:45", 
  "modify_datetime" -> "2019-05-27 13:02:45", 
  "video" -> "https://youtu.be/uAE8cJgzKXc", 
  "video_id" -> "uAE8cJgzKXc"}, {"draw_id" -> "19044", 
  "draw_date" -> "05/26/2019", "draw_date_name" -> "Sunday", 
  "winning_num" -> "4-4-2-3", "payout1" -> "$4,398", 
  "payout2" -> "$616", "payout3" -> "$53", "payout4" -> "$5", 
  "payout5" -> "$366", "payout6" -> "$205", "payout7" -> "$103", 
  "draw_type" -> "0", "session" -> "Evening", 
  "next_draw_date" -> "05/27/2019", "next_draw_date_name" -> "Monday",
   "create_datetime" -> "2019-05-26 19:58:19", 
  "modify_datetime" -> "2019-05-26 19:58:19", 
  "video" -> "https://youtu.be/NfycxZlCvPM", 
  "video_id" -> "NfycxZlCvPM"}, {"draw_id" -> "19043", 
  "draw_date" -> "05/26/2019", "draw_date_name" -> "Sunday", 
  "winning_num" -> "1-4-9-1", "payout1" -> "$4,985", 
  "payout2" -> "$698", "payout3" -> "$60", "payout4" -> "$6", 
  "payout5" -> "$415", "payout6" -> "$116", "payout7" -> "$116", 
  "draw_type" -> "1", "session" -> "Mid-Day", 
  "next_draw_date" -> "05/27/2019", "next_draw_date_name" -> "Monday",
   "create_datetime" -> "2019-05-26 12:56:15", 
  "modify_datetime" -> "2019-05-26 12:56:15", 
  "video" -> "https://youtu.be/TE0K5-RZD_4", 
  "video_id" -> "TE0K5-RZD_4"}, {"draw_id" -> "19042", 
  "draw_date" -> "05/25/2019", "draw_date_name" -> "Saturday", 
  "winning_num" -> "0-8-0-6", "payout1" -> "$4,839", 
  "payout2" -> "$677", "payout3" -> "$58", "payout4" -> "$6", 
  "payout5" -> "$403", "payout6" -> "$226", "payout7" -> "$113", 
  "draw_type" -> "0", "session" -> "Evening", 
  "next_draw_date" -> "05/26/2019", "next_draw_date_name" -> "Sunday",
   "create_datetime" -> "2019-05-25 19:56:40", 
  "modify_datetime" -> "2019-05-25 19:56:40", 
  "video" -> "https://youtu.be/ZxZhNk4vpCA", 
  "video_id" -> "ZxZhNk4vpCA"}, {"draw_id" -> "19041", 
  "draw_date" -> "05/25/2019", "draw_date_name" -> "Saturday", 
  "winning_num" -> "8-0-9-6", "payout1" -> "$5,105", 
  "payout2" -> "$715", "payout3" -> "$61", "payout4" -> "$6", 
  "payout5" -> "$213", "payout6" -> "$119", "payout7" -> "$119", 
  "draw_type" -> "1", "session" -> "Mid-Day", 
  "next_draw_date" -> "05/26/2019", "next_draw_date_name" -> "Sunday",
   "create_datetime" -> "2019-05-25 13:06:24", 
  "modify_datetime" -> "2019-05-25 18:39:38", 
  "video" -> "https://youtu.be/AS2qCbSVlwc", 
  "video_id" -> "AS2qCbSVlwc"}, {"draw_id" -> "19040", 
  "draw_date" -> "05/24/2019", "draw_date_name" -> "Friday", 
  "winning_num" -> "5-7-7-6", "payout1" -> "$5,274", 
  "payout2" -> "$738", "payout3" -> "$63", "payout4" -> "$6", 
  "payout5" -> "$439", "payout6" -> "$246", "payout7" -> "$246", 
  "draw_type" -> "0", "session" -> "Evening", 
  "next_draw_date" -> "05/25/2019", 
  "next_draw_date_name" -> "Saturday", 
  "create_datetime" -> "2019-05-24 19:56:48", 
  "modify_datetime" -> "2019-05-24 19:56:48", 
  "video" -> "https://youtu.be/ZzqSyXTUaBk", 
  "video_id" -> "ZzqSyXTUaBk"}, {"draw_id" -> "19039", 
  "draw_date" -> "05/24/2019", "draw_date_name" -> "Friday", 
  "winning_num" -> "4-6-6-9", "payout1" -> "$5,242", 
  "payout2" -> "$734", "payout3" -> "$63", "payout4" -> "$6", 
  "payout5" -> "$437", "payout6" -> "$245", "payout7" -> "$245", 
  "draw_type" -> "1", "session" -> "Mid-Day", 
  "next_draw_date" -> "05/25/2019", 
  "next_draw_date_name" -> "Saturday", 
  "create_datetime" -> "2019-05-24 12:59:30", 
  "modify_datetime" -> "2019-05-24 12:59:30", 
  "video" -> "https://youtu.be/mJEi8Rf_F44", 
  "video_id" -> "mJEi8Rf_F44"}, {"draw_id" -> "19038", 
  "draw_date" -> "05/23/2019", "draw_date_name" -> "Thursday", 
  "winning_num" -> "2-0-8-8", "payout1" -> "$5,109", 
  "payout2" -> "$715", "payout3" -> "$61", "payout4" -> "$6", 
  "payout5" -> "$426", "payout6" -> "$119", "payout7" -> "$238", 
  "draw_type" -> "0", "session" -> "Evening", 
  "next_draw_date" -> "05/24/2019", "next_draw_date_name" -> "Friday",
   "create_datetime" -> "2019-05-23 19:55:09", 
  "modify_datetime" -> "2019-05-23 19:55:09", 
  "video" -> "https://youtu.be/ty76fEmN4ZQ", 
  "video_id" -> "ty76fEmN4ZQ"}, {"draw_id" -> "19037", 
  "draw_date" -> "05/23/2019", "draw_date_name" -> "Thursday", 
  "winning_num" -> "5-6-2-3", "payout1" -> "$5,584", 
  "payout2" -> "$782", "payout3" -> "$67", "payout4" -> "$7", 
  "payout5" -> "$233", "payout6" -> "$130", "payout7" -> "$130", 
  "draw_type" -> "1", "session" -> "Mid-Day", 
  "next_draw_date" -> "05/24/2019", "next_draw_date_name" -> "Friday",
   "create_datetime" -> "2019-05-23 12:59:08", 
  "modify_datetime" -> "2019-05-23 12:59:08", 
  "video" -> "https://youtu.be/nmFpAqFzjxM", 
  "video_id" -> "nmFpAqFzjxM"}, {"draw_id" -> "19036", 
  "draw_date" -> "05/22/2019", "draw_date_name" -> "Wednesday", 
  "winning_num" -> "5-5-5-3", "payout1" -> "$4,798", 
  "payout2" -> "$672", "payout3" -> "$58", "payout4" -> "$6", 
  "payout5" -> "$1,199", "payout6" -> "$672", "payout7" -> "$224", 
  "draw_type" -> "0", "session" -> "Evening", 
  "next_draw_date" -> "05/23/2019", 
  "next_draw_date_name" -> "Thursday", 
  "create_datetime" -> "2019-05-22 20:02:35", 
  "modify_datetime" -> "2019-05-22 20:02:35", 
  "video" -> "https://youtu.be/Af7Gjuv-_fQ", 
  "video_id" -> "Af7Gjuv-_fQ"}, {"draw_id" -> "19035", 
  "draw_date" -> "05/22/2019", "draw_date_name" -> "Wednesday", 
  "winning_num" -> "3-4-6-3", "payout1" -> "$4,999", 
  "payout2" -> "$700", "payout3" -> "$60", "payout4" -> "$6", 
  "payout5" -> "$417", "payout6" -> "$117", "payout7" -> "$117", 
  "draw_type" -> "1", "session" -> "Mid-Day", 
  "next_draw_date" -> "05/23/2019", 
  "next_draw_date_name" -> "Thursday", 
  "create_datetime" -> "2019-05-22 12:54:30", 
  "modify_datetime" -> "2019-05-22 12:55:25", 
  "video" -> "https://youtu.be/Ixx_8cMV8ME", 
  "video_id" -> "Ixx_8cMV8ME"}, {"draw_id" -> "19034", 
  "draw_date" -> "05/21/2019", "draw_date_name" -> "Tuesday", 
  "winning_num" -> "5-8-6-8", "payout1" -> "$4,441", 
  "payout2" -> "$622", "payout3" -> "$53", "payout4" -> "$5", 
  "payout5" -> "$370", "payout6" -> "$104", "payout7" -> "$207", 
  "draw_type" -> "0", "session" -> "Evening", 
  "next_draw_date" -> "05/22/2019", 
  "next_draw_date_name" -> "Wednesday", 
  "create_datetime" -> "2019-05-21 19:58:30", 
  "modify_datetime" -> "2019-05-21 19:58:30", 
  "video" -> "https://youtu.be/2DIenRnIBF8", 
  "video_id" -> "2DIenRnIBF8"}, {"draw_id" -> "19033", 
  "draw_date" -> "05/21/2019", "draw_date_name" -> "Tuesday", 
  "winning_num" -> "4-3-9-9", "payout1" -> "$5,817", 
  "payout2" -> "$814", "payout3" -> "$70", "payout4" -> "$7", 
  "payout5" -> "$485", "payout6" -> "$136", "payout7" -> "$271", 
  "draw_type" -> "1", "session" -> "Mid-Day", 
  "next_draw_date" -> "05/22/2019", 
  "next_draw_date_name" -> "Wednesday", 
  "create_datetime" -> "2019-05-21 12:57:29", 
  "modify_datetime" -> "2019-05-21 13:02:15", 
  "video" -> "https://youtu.be/lQvz7gEfQ3Q", 
  "video_id" -> "lQvz7gEfQ3Q"}}
POSTED BY: Kay Herbert
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract