Message Boards Message Boards

Using Basic JSON REST Services in Mathematica

Posted 7 years ago

I recently had a job interview where I was given the following problem and I decided to tackle it with Mathematica vs. Groovy; it seemed very simple to accomplish using Mathematica: (Hopefully they take kindly to it)

  • There’s a “fake” REST service available for testing at: http://jsonplaceholder.typicode.com/
  • Write some code in the Groovy programming language that uses the
    “posts” API published at http://jsonplaceholder.typicode.com/posts,
    which reports back some fake blog posts including a userid, post id, title, and body.
  • Retrieve the records from the API and generate a report that lists only the users who wrote posts where the post title starts with the letter “s” along with the number of such posts for each of those users.
  • Use the user's name in the report, not the numerical user ID.
  • Return to us the report that's generated, along with the (working) code that generated it.

First, let's import our posts from the website as RawJSON format and generate a Dataset from it which is much more convenient to work with:

    posts = Import["http://jsonplaceholder.typicode.com/posts", "RawJSON"];
    postsDataSet = Dataset[posts]

enter image description here

Next, let us filter out all of the posts whose bodies of text start with the string character "s":

    filterPostsDataSet = postsDataSet[Select[StringTake[#title, 1] == "s" &]]

enter image description here

We are left with 9 entries and all we need to do is keep the userId values associated with these:

userIDs = filterPostsDataSet[All, "userId"]

enter image description here

Now we need to retrieve the user's information attached to these Id values; although not specifically stated I assumed these could be found at http://jsonplaceholder.typicode.com/users. Thus, we simply import these values and, like before, wrap them in a Dataset.

users = Import["http://jsonplaceholder.typicode.com/users", "RawJSON"];
usersDataSet = Dataset[users]

enter image description here Although I feel there is a more elegant way - I am going to make a list of just the users names from this Dataset to work with using the following:

userResults = 
 Table[usersDataSet[Select[#id == userIDs[[i]] &]][[1]]["name"], {i, 1, Length[userIDs]}]

Additionally, I want to keep a list of the unique user names since some might be duplicated in the previous list:

uniqueUsers = DeleteDuplicates[userResults]

Finally, I will create a final list which is essentially my required report, which will be a list of pairs. Each pair will contain the unique user name as well as the count of how many times that unique user name appeared in the original list. I put this in TableForm just for aesthetic value. This can then be exported to the desired format later on if necessary.

uniqueUserPosts = Table[{uniqueUsers[[i]], Count[userResults, uniqueUsers[[i]]]}, {i, 1, Length[uniqueUsers]}] // TableForm

enter image description here

In conclusion, I think that in the future using Mathematica in working with REST API's could and should be allowed if not encouraged as a proper scripting languages in many commercial and industrial settings. So - what do you guys think?? Is this better or worse than what could be done in Groovy or other scripting languages? Also, I feel that I could condense and optimize my code in areas and would love to hear feedback on that.

POSTED BY: William Duhe
12 Replies
Posted 7 years ago

Cool! I've completed a programming test for a job using WL before. Regarding the part where you felt there was a more elegant way, I would suggest using JoinAcross. So if posts and users are your imported datasets then you can do:

Select[posts, StringStartsQ[#title, "s"] &] // 
 JoinAcross[#, users, "userId" -> "id"][Counts, "name"] &
POSTED BY: Michael Hale

Hi William,

Thank you for your post.

We currently support many RESTful APIs in Mathematica. We are working on expanding this as a more general framework for users to connect to any kind of external RESTful API.

Any feedback is always much appreciated. Thanks!

Posted 7 years ago

No need to restrict yourselves to RESTful APIs, I think (although they will be by far the most useful). You can also use the ServiceConnect framework to link to random, poorly designed websites that don't support any real API, e.g. A ServiceConnection for Wolfram Community

POSTED BY: b3m2a1 ​ 

Thanks! Will look at latest downloads on my Mac. I may be using V10. Henk

POSTED BY: Henk Roodt
Posted 7 years ago

Let me know when you move forward and what your goals with the data are - I have some experience visualizing Geographical data in neat ways - which gets a lot more powerful in Mathematica 11!

POSTED BY: William Duhe

Yes, I think that is a neat plan. I for one will gain a lot. I am keen to operate directly with the database and do analytics and graphs.

I have duplicated your example and it clearly works. However, if I read from my own .json local files, I cannot get things to work.

A real noob, so here is what is in the file: [ { "cap": "Algiers", "imp": "31", "iso": "DZ", "lat": "28", "long": "3", "name": "Algeria", "region": "Africa" }, { "cap": "Cairo", "imp": "54", "iso": "EG", "lat": "27", "long": "30", "name": "Egypt", "region": "Africa" }, { "cap": "Addis Ababa", "imp": "62", "iso": "ET", "lat": "8", "long": "-5", "name": "Ethiopia", "region": "Africa" } ]

==> First off, I do not understand why the square brackets are required, because without them Mathematica gives an error.

mySt = Import["/Volumes/DATA_2TB 1/mathematicaAPPS/tes.json"];

stuffs = Dataset[mySt]

Delivers this: enter image description here

SO: I do not seem to get the dataset right, my import does not work, and when I use the RawJSON parameter I get another error.

Any ideas very welcome.

I think a track with basic things, from reading .json files from disk into the nice datasets to eventually operating on a live data set will be awesome. Small steps, sort of thing. And then analytics and graphs etc. I can see it being visited quite a bit.

Henk

POSTED BY: Henk Roodt
Posted 7 years ago

Can you attach the raw data file to the post for my analysis - without altering it at all from it's original format? I am having trouble understanding why it won't import correctly.

EDIT::

Actually if you use the file you put in the post exactly it imports fine on my machine shown bellow: enter image description here

Are you using the latest version of Mathematica - being 11.2?

POSTED BY: William Duhe

Here it is

Attachments:
POSTED BY: Henk Roodt
Posted 7 years ago

Yeah this file seems to load fine for me using the code I showed above. Again, you might want to update your kernel to the latest version. Also, you need the square brackets because they are part of the official RawJSON format =)

POSTED BY: William Duhe

Wow! Right, I think the way you have explained things is really, really good. I am only starting out after many years not using Mathematica and I am keen to do some data analytics on Firebase data. This will already help me I think. Thanks!!

POSTED BY: Henk Roodt
Posted 7 years ago

Thanks for your feedback Henk! I am really glad it could offer you some insight. If you need any help working with Firebase I would happy to point you in the right direction. It may even be fun to make another community post about how to work with Firebase specifically.

POSTED BY: William Duhe

enter image description here - Congratulations! This post is now a Staff Pick as distinguished by a badge on your profile! Thank you, keep it coming!

POSTED BY: EDITORIAL BOARD
Reply to this discussion
Community posts can be styled and formatted using the Markdown syntax.
Reply Preview
Attachments
Remove
or Discard

Group Abstract Group Abstract