Gather and analyse data from internet using WebSearch and WebAccess?

Questions
, , ,
1

Hi everyone,

I’m trying to use Mathematica to automatically gather and analyze data from the internet. It’s a pet project but one that I hope to become quite useful some day. I’ve made decent progress but then ran into some issues. At the moment, I’m trying to solve two issues, and would appreciate any help or suggestions how to resolve them:

  1. WebSearch stopped working. I’ve been using WebSearch to find websites of solar energy related topics. It worked great at the beginning, but after two days of using it, the command stopped working. Now it executes as if it was an unknown command (i.e. just returns the written text WebSearch[“topic”]. Does anyone have an idea what happened there? Neither restarting the computer nor quitting the kernel helped.

  2. This might be a stupid question, but I have banged my head against it for too long. How do I reset the saved ServiceConnect connection to GoogleCustomSearch? I want the program to prompt the API again, but have so far been unsuccessful (even after updating from 11.2 to 11.3 it didn’t prompt the API again). ServiceDisconnect doesn’t do the trick, as it doesn’t delete saved connection. Anyone any ideas?

Thanks so much!

2

From the “Details and Options” section of the documentation for WebSearch:

WebSearch requires internet connectivity and uses external services that require Service Credits.

Perhaps you have exhausted all of your credits and need to purchase more.

From the documentation for ServiceConnect:

ServiceConnect[“service”,“New”] forces a new connection to be created
to a particular service.

3

Thanks, Rohit. I wasn’t aware of the Service Credits (most have over-read that).

4

In the soon-to-be-release version 12 of the Wolfram Language we are including another option to interact with web pages, namely the function WebExecute. Here is a sneak preview of the documentation:

Annotation_OP_7_CP_.png
Annotation_OP_7_CP_.png491×503 13.2 KB

It is based on a repository of mine on GitHub (GitHub - arnoudbuzing/webtools: A Wolfram Language package which automates interactions with web browsers · GitHub) and lets you control a Chrome or Firefox web browser. It can complement the WebSearch function in the sense that you can navigate to a page, take screen captures of a page, click on links, type input fields, submit forms, and execute javascript to return text, data, or images from a web page of interest.

5

@Ian Marius Peters

You can delete the saved connections using below commands( those are not documented though):

In[11]:= savedConnections =
ServiceConnections`SavedConnections[“GoogleCustomSearch”]

Out[11]= {
ServiceObject[
“GoogleCustomSearch”,
“ID” → “connection-6774d4b6c2755d7139e6292d4ee03963”]}

In[12]:= ServiceConnections`DeleteConnection[#] & /@ savedConnections

Out[12]= {Null}

In[13]:= ServiceConnections`SavedConnections[“GoogleCustomSearch”]

Out[13]= {}

6

Neat! I’m looking forward to having that available.