Message Boards Message Boards

0
|
9719 Views
|
6 Replies
|
5 Total Likes
View groups...
Share
Share this post:

How can I send keyboard commands via Mathematica to a given URL?

Posted 9 years ago

I am trying to extract job listings from a website that presents many hundreds of jobs available; this listing is limited to 10 per page and it is possible to step through the long list with keyboard commands such as 'Alt+Shift+P'. I have tried the Import[url,"Data"] which only returns the 10 listing on the currently supplied webpage. Would like to be able to send commands under Mathematica control to the website to step through this longer list and retrieve the entire listing of jobs. The attachment is a relevant snippet of the page HTML. I apologize if this is too off-topic for this discussion group; not really sure which group is the 'correct' one.

Attachments:
POSTED BY: Coach KLC
6 Replies

Unfortunately, you can't send keyboard commands because Mathematica is only able to fetch the raw contents of the page. Would you be able to post a link to the website? It's possible there's some backend XMLHttpRequest API that's fetching data every time you hit Alt+Shift+P.

And don't worry, it's not off-topic at all!

POSTED BY: Jesse Friedman
Posted 9 years ago

@Jessie Friedman -- Thanks for your reply; sorry to take so long to get back to you. Here is the link to the web page which exhibits the behaviour I described. In the page source you can find the references to the key commands:

<input type="submit" name="__Previous" value=" < " id="__Previous" disabled="disabled" class="ButtonClassNavigation" style="cursor:default;" />  <input type="submit" name="__Next" value=" > " id="__Next" accesskey="n" title="Alt+n" class="ButtonClassNavigation" />  <input type="submit" name="__Last" value=">>" id="__Last" accesskey="l" title="Alt+l" class="ButtonClassNavigation" /></form> Again thanks for your response! K

POSTED BY: Coach KLC

No problem about the delay. It looks like you didn't post the right text, though.

POSTED BY: Jesse Friedman
Posted 9 years ago

Right! Here is the url to the webpage in question: https://re11.ultipro.com/KIM1000/JobBoard/ListJobs.aspx?__VT=ExtCan

POSTED BY: Coach KLC

We can append the URL parameter __RecordsPerPage to the URL to get all the jobs on one page: https://re11.ultipro.com/KIM1000/JobBoard/ListJobs.aspx?__VT=ExtCan&__RecordsPerPage=1000

And here we can make a Dataset of all the jobs:

maxrecords = 1000;
pagedata = 
  Import["https://re11.ultipro.com/KIM1000/JobBoard/ListJobs.aspx?__VT=ExtCan&__RecordsPerPage=" <>
    ToString[maxrecords], "Data"][[1, 2]];
jobs = Dataset[AssociationThread[pagedata[[1]], #] & /@ pagedata[[2 ;;]]]

The Dataset output of the above code

If a given page has more than 1000 records, just change maxrecords to something like 10000.

POSTED BY: Jesse Friedman
Posted 9 years ago

Thank you very much Jesse. Back to work on this after a prolonged hiatus. I really appreciate your time and expertise on this effort. coach

POSTED BY: Coach KLC
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract