Group Abstract Group Abstract

Message Boards Message Boards

Gather and analyse data from internet using WebSearch and WebAccess?

Ian Marius Peters, MIT
Posted 6 years ago

Hi everyone,

I'm trying to use Mathematica to automatically gather and analyze data from the internet. It's a pet project but one that I hope to become quite useful some day. I've made decent progress but then ran into some issues. At the moment, I'm trying to solve two issues, and would appreciate any help or suggestions how to resolve them:

  1. WebSearch stopped working. I've been using WebSearch to find websites of solar energy related topics. It worked great at the beginning, but after two days of using it, the command stopped working. Now it executes as if it was an unknown command (i.e. just returns the written text WebSearch["topic"]. Does anyone have an idea what happened there? Neither restarting the computer nor quitting the kernel helped.

  2. This might be a stupid question, but I have banged my head against it for too long. How do I reset the saved ServiceConnect connection to GoogleCustomSearch? I want the program to prompt the API again, but have so far been unsuccessful (even after updating from 11.2 to 11.3 it didn't prompt the API again). ServiceDisconnect doesn't do the trick, as it doesn't delete saved connection. Anyone any ideas?

Thanks so much!
POSTED BY: Ian Marius Peters
5 Replies
Sort By:

Neat! I'm looking forward to having that available.
POSTED BY: Ian Marius Peters

@Ian Marius Peters

You can delete the saved connections using below commands( those are not documented though): In[11]:= savedConnections = ServiceConnections`SavedConnections["GoogleCustomSearch"]

Out[11]= { ServiceObject[ "GoogleCustomSearch", "ID" -> "connection-6774d4b6c2755d7139e6292d4ee03963"]}

In[12]:= ServiceConnections`DeleteConnection[#] & /@ savedConnections

Out[12]= {Null}

In[13]:= ServiceConnections`SavedConnections["GoogleCustomSearch"]

Out[13]= {}

POSTED BY: Damanjit Singh

In the soon-to-be-release version 12 of the Wolfram Language we are including another option to interact with web pages, namely the function WebExecute. Here is a sneak preview of the documentation:

enter image description here

It is based on a repository of mine on GitHub (https://github.com/arnoudbuzing/webtools) and lets you control a Chrome or Firefox web browser. It can complement the WebSearch function in the sense that you can navigate to a page, take screen captures of a page, click on links, type input fields, submit forms, and execute javascript to return text, data, or images from a web page of interest.
POSTED BY: Arnoud Buzing
Rohit Namjoshi
Rohit Namjoshi
Posted 6 years ago

From the "Details and Options" section of the documentation for WebSearch:

WebSearch requires internet connectivity and uses external services that require Service Credits.

Perhaps you have exhausted all of your credits and need to purchase more.

From the documentation for ServiceConnect:

ServiceConnect["service","New"] forces a new connection to be created to a particular service.
POSTED BY: Rohit Namjoshi

Thanks, Rohit. I wasn't aware of the Service Credits (most have over-read that).
POSTED BY: Ian Marius Peters
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Tag limit exceeded
Note: Only the first five people you tag will receive an email notification; the other tagged names will appear as links to their profiles.
Reply Preview
Attachments
Remove
or Discard