Message Boards Message Boards

Using Basic JSON REST Services in Mathematica

GROUPS:

I recently had a job interview where I was given the following problem and I decided to tackle it with Mathematica vs. Groovy; it seemed very simple to accomplish using Mathematica: (Hopefully they take kindly to it)

  • There’s a “fake” REST service available for testing at: http://jsonplaceholder.typicode.com/
  • Write some code in the Groovy programming language that uses the
    “posts” API published at http://jsonplaceholder.typicode.com/posts,
    which reports back some fake blog posts including a userid, post id, title, and body.
  • Retrieve the records from the API and generate a report that lists only the users who wrote posts where the post title starts with the letter “s” along with the number of such posts for each of those users.
  • Use the user's name in the report, not the numerical user ID.
  • Return to us the report that's generated, along with the (working) code that generated it.

First, let's import our posts from the website as RawJSON format and generate a Dataset from it which is much more convenient to work with:

    posts = Import["http://jsonplaceholder.typicode.com/posts", "RawJSON"];
    postsDataSet = Dataset[posts]

enter image description here

Next, let us filter out all of the posts whose bodies of text start with the string character "s":

    filterPostsDataSet = postsDataSet[Select[StringTake[#title, 1] == "s" &]]

enter image description here

We are left with 9 entries and all we need to do is keep the userId values associated with these:

userIDs = filterPostsDataSet[All, "userId"]

enter image description here

Now we need to retrieve the user's information attached to these Id values; although not specifically stated I assumed these could be found at http://jsonplaceholder.typicode.com/users. Thus, we simply import these values and, like before, wrap them in a Dataset.

users = Import["http://jsonplaceholder.typicode.com/users", "RawJSON"];
usersDataSet = Dataset[users]

enter image description here Although I feel there is a more elegant way - I am going to make a list of just the users names from this Dataset to work with using the following:

userResults = 
 Table[usersDataSet[Select[#id == userIDs[[i]] &]][[1]]["name"], {i, 1, Length[userIDs]}]

Additionally, I want to keep a list of the unique user names since some might be duplicated in the previous list:

uniqueUsers = DeleteDuplicates[userResults]

Finally, I will create a final list which is essentially my required report, which will be a list of pairs. Each pair will contain the unique user name as well as the count of how many times that unique user name appeared in the original list. I put this in TableForm just for aesthetic value. This can then be exported to the desired format later on if necessary.

uniqueUserPosts = Table[{uniqueUsers[[i]], Count[userResults, uniqueUsers[[i]]]}, {i, 1, Length[uniqueUsers]}] // TableForm

enter image description here

In conclusion, I think that in the future using Mathematica in working with REST API's could and should be allowed if not encouraged as a proper scripting languages in many commercial and industrial settings. So - what do you guys think?? Is this better or worse than what could be done in Groovy or other scripting languages? Also, I feel that I could condense and optimize my code in areas and would love to hear feedback on that.

POSTED BY: William Duhe
Answer
12 days ago

enter image description here - Congratulations! This post is now a Staff Pick as distinguished by a badge on your profile! Thank you, keep it coming!

POSTED BY: Moderation Team
Answer
10 days ago

Wow! Right, I think the way you have explained things is really, really good. I am only starting out after many years not using Mathematica and I am keen to do some data analytics on Firebase data. This will already help me I think. Thanks!!

POSTED BY: Henk Roodt
Answer
6 days ago

Thanks for your feedback Henk! I am really glad it could offer you some insight. If you need any help working with Firebase I would happy to point you in the right direction. It may even be fun to make another community post about how to work with Firebase specifically.

POSTED BY: William Duhe
Answer
6 days ago

Yes, I think that is a neat plan. I for one will gain a lot. I am keen to operate directly with the database and do analytics and graphs.

I have duplicated your example and it clearly works. However, if I read from my own .json local files, I cannot get things to work.

A real noob, so here is what is in the file: [ { "cap": "Algiers", "imp": "31", "iso": "DZ", "lat": "28", "long": "3", "name": "Algeria", "region": "Africa" }, { "cap": "Cairo", "imp": "54", "iso": "EG", "lat": "27", "long": "30", "name": "Egypt", "region": "Africa" }, { "cap": "Addis Ababa", "imp": "62", "iso": "ET", "lat": "8", "long": "-5", "name": "Ethiopia", "region": "Africa" } ]

==> First off, I do not understand why the square brackets are required, because without them Mathematica gives an error.

mySt = Import["/Volumes/DATA_2TB 1/mathematicaAPPS/tes.json"];

stuffs = Dataset[mySt]

Delivers this: enter image description here

SO: I do not seem to get the dataset right, my import does not work, and when I use the RawJSON parameter I get another error.

Any ideas very welcome.

I think a track with basic things, from reading .json files from disk into the nice datasets to eventually operating on a live data set will be awesome. Small steps, sort of thing. And then analytics and graphs etc. I can see it being visited quite a bit.

Henk

POSTED BY: Henk Roodt
Answer
6 days ago

Can you attach the raw data file to the post for my analysis - without altering it at all from it's original format? I am having trouble understanding why it won't import correctly.

EDIT::

Actually if you use the file you put in the post exactly it imports fine on my machine shown bellow: enter image description here

Are you using the latest version of Mathematica - being 11.2?

POSTED BY: William Duhe
Answer
6 days ago

Here it is

Attachments:
POSTED BY: Henk Roodt
Answer
6 days ago

Yeah this file seems to load fine for me using the code I showed above. Again, you might want to update your kernel to the latest version. Also, you need the square brackets because they are part of the official RawJSON format =)

POSTED BY: William Duhe
Answer
6 days ago

Thanks! Will look at latest downloads on my Mac. I may be using V10. Henk

POSTED BY: Henk Roodt
Answer
6 days ago

Let me know when you move forward and what your goals with the data are - I have some experience visualizing Geographical data in neat ways - which gets a lot more powerful in Mathematica 11!

POSTED BY: William Duhe
Answer
6 days ago

Group Abstract Group Abstract