# Help in getting started with data reduction

**URL:** <https://community.wolfram.com/t/help-in-getting-started-with-data-reduction/4211>\
**Category:** Questions\
**Tags:** mathematica, statistics-and-probability\
**Created:** [December 3, 2014, 6:13am UTC](https://community.wolfram.com/t/help-in-getting-started-with-data-reduction/4211 "2014-12-03T06:13:49Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![danielgross](https://avatars.discourse-cdn.com/v4/letter/d/2acd7d/32.png) [@danielgross](https://community.wolfram.com/u/danielgross)\
**Post date:** [December 3, 2014, 6:13am UTC](https://community.wolfram.com/t/help-in-getting-started-with-data-reduction/4211/1 "2014-12-03T06:13:49Z")

</div>

Hi,

I am looking for someone to help me get started in visualizing some data collection. The data is stored in a nested JSON file. At this stage i know how to import JSON into a rather large associative array, and now i need to know how to extract, arrange, group, sum data to present data insights using 2D and 3D graphs.

I am a programmer, and have some appreciation for functional programming, but am very new to mathematica and need some help to get started. What i need is several explained examples tailored to my needs and i think i can then continue on my own.

thanks,

Dan

---

<div class="post-metadata">

**Author:** ![dreiss](https://yyz1.discourse-cdn.com/wolfram/user_avatar/community.wolfram.com/dreiss/32/29094_2.png) [@dreiss](https://community.wolfram.com/u/dreiss)\
**Post date:** [December 3, 2014, 5:04pm UTC](https://community.wolfram.com/t/help-in-getting-started-with-data-reduction/4211/2 "2014-12-03T17:04:40Z")

</div>

Perhaps post a small example of your JSON file along with an example of what you want to plot? Otherwise your question is extremely general…

---

<div class="post-metadata">

**Author:** ![dreiss](https://yyz1.discourse-cdn.com/wolfram/user_avatar/community.wolfram.com/dreiss/32/29094_2.png) [@dreiss](https://community.wolfram.com/u/dreiss)\
**Post date:** [December 3, 2014, 6:12pm UTC](https://community.wolfram.com/t/help-in-getting-started-with-data-reduction/4211/3 "2014-12-03T18:12:33Z")

</div>

A couple of quick comments to get you started though.

When you import from a JSON file the result is a list (perhaps nested in various ways reflecting the structure of the JSON file) of replacement rules. (FYI: be careful of calling it an associative array–even though, in essence it is–since that might confuse folks with the `Association` construct in Mathematica which is somewhat different than a list of rules.)

Since, once you’ve imported your JSON file, you now have a list of rules, you can manipulate it in any way you wish using list manipulation commands as well as various other Mathematica programming constructs (pattern matching, for example, to pull out particular items from within the list) including replacements using the rules.

Here are some tutorials:

Working with Rules:

> **[Transformation Rules and Definitions—Wolfram Documentation](https://reference.wolfram.com/language/tutorial/TransformationRulesAndDefinitions.html#18035)**
>
> When you use expr/.rules, each rule is tried in turn on each part of expr. As soon as a rule applies, the appropriate transformation is made, and the resulting part is returned. The replacement expr/.rules tries each rule just once on each part of...

At the top of this page under “Learning Resources” are links to tutorials on working with Lists:

> **[List Manipulation—Wolfram Documentation](https://reference.wolfram.com/language/guide/ListManipulation.html)**
>
> Lists are central constructs in the Wolfram Language, used to represent collections, arrays, sets, and sequences of all kinds. Lists can have any structure and size and can routinely involve even millions of elements. Well over a thousand built-in...

Looking at the documentation of the various functions for plotting will help you get started there:

> **[Graphics and Sound—Wolfram Documentation](https://reference.wolfram.com/language/tutorial/GraphicsAndSound.html#11224)**
>
> When the Wolfram Language plots a graph for you, it has to make many choices. It has to work out what the scales should be, where the function should be sampled, how the axes should be drawn, and so on. Most of the time, the Wolfram Language will...

and going through some of the on-line videos can be useful (some are better than others, so use what helps and skip what doesn’t), e.g.:

> **[Wolfram U Classes and Courses](https://www.wolfram.com/wolfram-u/courses/catalog/)**
>
> Full list of computation-based classes. Includes live interactive courses as well as video classes. Beginner through advanced topics.

And if you can provide a simple example of what you want to do that will help with being able to provide you with more targeted advice.

---

<div class="post-metadata">

**Author:** ![brucem](https://yyz1.discourse-cdn.com/wolfram/user_avatar/community.wolfram.com/brucem/32/26319_2.png) [@brucem](https://community.wolfram.com/u/brucem)\
**Post date:** [December 3, 2014, 6:13pm UTC](https://community.wolfram.com/t/help-in-getting-started-with-data-reduction/4211/4 "2014-12-03T18:13:38Z")

</div>

> **[Wolfram Video Archive](https://www.wolfram.com/broadcast/)**
>
> For current Wolfram videos, screenshots, tutorials and other video content, find us on YouTube and Wolfram U.

> **[Wolfram Video Archive: How Tos](https://www.wolfram.com/broadcast/c/?c=108)**

[Wolfram Language & System Documentation Center](http://reference.wolfram.com/language/), especially

- [Statistical Data Analysis—Wolfram Documentation](http://reference.wolfram.com/language/guide/Statistics.html)
- [Handling Arrays of Data—Wolfram Documentation](http://reference.wolfram.com/language/guide/HandlingArraysOfData.html)

(Note the links to other Guide pages and tutorials.)

Videos of on-line mini courses are at [Wolfram U Classes and Courses](http://www.wolfram.com/training/special-event/)

If you have focused, specific questions, try the Community. Show the code that you tried.

If you want a lot of project-related tutoring, consider Technical Services. [Wolfram Consulting: Computational Data Solutions for the AI Era](http://www.wolfram.com/support/technical-services/)

---

<div class="post-metadata">

**Author:** ![dreiss](https://yyz1.discourse-cdn.com/wolfram/user_avatar/community.wolfram.com/dreiss/32/29094_2.png) [@dreiss](https://community.wolfram.com/u/dreiss)\
**Post date:** [December 3, 2014, 6:15pm UTC](https://community.wolfram.com/t/help-in-getting-started-with-data-reduction/4211/5 "2014-12-03T18:15:29Z")

</div>

Bruce and I were neck-and-neck in this one 😉

---

<div class="post-metadata">

**Author:** ![danielgross](https://avatars.discourse-cdn.com/v4/letter/d/2acd7d/32.png) [@danielgross](https://community.wolfram.com/u/danielgross)\
**Post date:** [December 7, 2014, 7:38am UTC](https://community.wolfram.com/t/help-in-getting-started-with-data-reduction/4211/6 "2014-12-07T07:38:46Z")

</div>

Thank you for all the replies. Here is the structure (mappings) I need to analyze and graph:

AllUsers(user)-\>UserSessions

UserSessions(DateTime)-\>OneUserSessionUserData

OneUserSessionUserData(sessionProperty) → SessionPropertyValue

OneUserSessionScreen(Session, Screen, Property) → ScreenPropertyValue

there is a bit more, but that’s enough for now.

In words the structure means the following:

We have a collection of users, known by their names, who repeatedly play games. A user plays a game on a particular date/time, usually only game a day – this we call one game session.

For each game we collect some overall data, such as duration of the game, number of screens played. Each game is made out of several screens. for each screen we also collect data, such as time to complete a screen. Whether the screen was completed successfully or not, or was skipped.

I now need to analyze this data from various points of views. For example. I want to know how often each person played the game so far. Also, on what dates/times the game was played, and how long it took to play on each date – or any of the other properties we measure per game.

We also want to know on average how long a screen took.

Finally, we partition all users into two groups, so we want to compare between groups also.

All the data is exported into a nested JSON array as per the structure above, and when Imported into mathematica (using a URL), it generates nested Association lists.

I think that Mathematica is ideal for this task, but i don’t know yet how to get started.

I wrote a small program for now, that traverses the structure and whenever it hits on a mapping it triggers an event with data. The event routines then do the data counting and cacluation work.

I need to do something similar in mathematica: some kind of Select or Apply, and one or more Functions that are applied and perhaps also composed to yield the results my event routines now calculate.

any help how to get started would be much appreciated,

Dan

---

<div class="post-metadata">

**Author:** ![dreiss](https://yyz1.discourse-cdn.com/wolfram/user_avatar/community.wolfram.com/dreiss/32/29094_2.png) [@dreiss](https://community.wolfram.com/u/dreiss)\
**Post date:** [December 7, 2014, 7:37pm UTC](https://community.wolfram.com/t/help-in-getting-started-with-data-reduction/4211/7 "2014-12-07T19:37:07Z")

</div>

Just a first point about nomenclature: what you get when you import a **JSON** file is not a Mathematica `Association`: it is a nested list of `Rule`s. `Association` ([Association—Wolfram Documentation](http://reference.wolfram.com/language/ref/Association.html)) in Mathematica is a very specific data construct that encapsulates key-value pairs. And even though the list of rules returned from the **JSON** import is generally a nested list of rules in key-value form–it is not an **Association** in the Mathematica meaning of the **Association** function.

That said… each particular analysis that you want to do will involve writing a specific function to grab the data you desire from this list of rules. In some cases it may use the `Cases` function, but more generally it will involve using things like `Part` and `ReplaceAll`.

So here is an example (if you could post a truncated example of one of your **JSON** data files I could show a couple of examples from that.

But here is a couple of examples. Attached is a **JSON** file (a very simple one) that I found on the web. If I `Import` it I get:

In[1]:= jsonImport = Import[“/Users/dreiss/Desktop/json.json”]

```
Out[1]= {"sex" -> "M", "first" -> "John", 
 "favorites" -> {"color" -> "Blue", "sport" -> "Soccer", 
   "food" -> "Spaghetti"}, "last" -> "Doe", 
 "interests" -> {"Reading", "Mountain Biking", "Hacking"}, 
 "age" -> 39, "salary" -> 70000, "registered" -> True, 
 "skills" -> {{"tests" -> {{"name" -> "One", 
       "score" -> 90}, {"name" -> "Two", "score" -> 96}}, 
    "category" -> 
     "JavaScript"}, {"tests" -> {{"name" -> "One", 
       "score" -> 79}, {"name" -> "Two", "score" -> 84}}, 
    "category" -> 
     "CouchDB"}, {"tests" -> {{"name" -> "One", 
       "score" -> 97}, {"name" -> "Two", "score" -> 93}}, 
    "category" -> "Node.js"}}}

```

So lets find the “skills” data:

```
In[3]:= "skills" /. jsonImport

Out[3]= {{"tests" -> {{"name" -> "One", 
     "score" -> 90}, {"name" -> "Two", "score" -> 96}}, 
  "category" -> 
   "JavaScript"}, {"tests" -> {{"name" -> "One", 
     "score" -> 79}, {"name" -> "Two", "score" -> 84}}, 
  "category" -> 
   "CouchDB"}, {"tests" -> {{"name" -> "One", 
     "score" -> 97}, {"name" -> "Two", "score" -> 93}}, 
  "category" -> "Node.js"}}

```

And, instead of this let’s see the “tests” directly:

```
In[3]:= testData = "tests" /. ("skills" /. jsonImport)

Out[3]= {{{"name" -> "One", "score" -> 90}, {"name" -> "Two", 
   "score" -> 96}}, {{"name" -> "One", 
   "score" -> 79}, {"name" -> "Two", 
   "score" -> 84}}, {{"name" -> "One", 
   "score" -> 97}, {"name" -> "Two", "score" -> 93}}}

```

And, now knowing this structure we can get the values for the test with “name” that has the value “one”:

```
In[5]:= Cases[testData, ({"name" -> "One", "score" -> value_}) :> 
  value, \[Infinity]]

Out[5]= {90, 79, 97}

```

And this result can be analyzed and visualized…

---

<div class="post-metadata">

**Author:** ![danielgross](https://avatars.discourse-cdn.com/v4/letter/d/2acd7d/32.png) [@danielgross](https://community.wolfram.com/u/danielgross)\
**Post date:** [December 7, 2014, 7:44pm UTC](https://community.wolfram.com/t/help-in-getting-started-with-data-reduction/4211/8 "2014-12-07T19:44:47Z")

</div>

Mathematica is magic!

The simplicity and elegance of solutions is why I really want to become proficient in Mathematica – its an amazing tool.

What is the name of the /. operator/function. Can you write this out in a non-short-hand form, where I can see what functions are used and applied?

thank you,

Dan

---

<div class="post-metadata">

**Author:** ![dreiss](https://yyz1.discourse-cdn.com/wolfram/user_avatar/community.wolfram.com/dreiss/32/29094_2.png) [@dreiss](https://community.wolfram.com/u/dreiss)\
**Post date:** [December 7, 2014, 7:49pm UTC](https://community.wolfram.com/t/help-in-getting-started-with-data-reduction/4211/9 "2014-12-07T19:49:12Z")

</div>

Great!

The `/.` operator is a shorthand (and it is almost always used in this shorthand form) fo the `ReplaceAll` function:

> **[ReplaceAll: Apply rules to each part of an expression—Wolfram Documentation](https://reference.wolfram.com/language/ref/ReplaceAll.html)**
>
> expr /. rules or ReplaceAll\[expr, rules\] applies a rule or list of rules in an attempt to transform each subpart of an expression expr. ReplaceAll\[rules\] represents an operator form of ReplaceAll that can be applied to an expression.

So a comparative example might be:

```
In[1]:= b /. {a -> 1, b -> 2, c -> 3}

Out[1]= 2

b /. {a -> 1, b -> 2, c -> 3}

In[2]:= ReplaceAll[b, {a -> 1, b -> 2, c -> 3}]

Out[2]= 2

```

By the way, since I suspect that you have considerable experience with other programming languages, you it may help you to go though the following rapid tutorial to get an overview of things

> **[Wolfram Language Tutorial: Fast Introduction for Programmers](https://www.wolfram.com/language/fast-introduction-for-programmers/en/)**
>
> Get started and up to speed on the basic principles and concepts of programming in the Wolfram Language. Introductory tutorial for programmers.

It’s rather like drinking from a firehose, but it can orient you into the overall way of thinking and to get a conceptual high-level sense of the architecture of the Wolfram language.

And a book that I often recommend is this (current through Mathematica 9 functionality):

> **[Programming with Mathematica®: An Introduction: Wellin, Paul: 9781107009462:...](https://www.amazon.com/Programming-Mathematica%C2%AE-Introduction-Paul-Wellin/dp/1107009464)**
>
> Programming with Mathematica®: An Introduction \[Wellin, Paul\] on Amazon.com. \*FREE\* shipping on qualifying offers. Programming with Mathematica®: An Introduction

---

<div class="post-metadata">

**Author:** ![danielgross](https://avatars.discourse-cdn.com/v4/letter/d/2acd7d/32.png) [@danielgross](https://community.wolfram.com/u/danielgross)\
**Post date:** [December 7, 2014, 7:59pm UTC](https://community.wolfram.com/t/help-in-getting-started-with-data-reduction/4211/10 "2014-12-07T19:59:56Z")

</div>

Thank you.

Yes. I am a programmer.

I think the key challenge in getting started is the wealth of possibilities, and the need to get into the right mindset, how, problems are solved with the language tools at disposal. Its frankly, overwhelming.

I think a key challenge i currently perceive in the documentation is that examples are “simple islands”, and its not obvious how to stitch language constructs together to solve problems that require multiple problem solving steps (apply ./, then create a “part” function, then use case to pattern match, wrapped up in one expression, kind of thinking.

I think best if there are more coarse grained end-to-end problems solving examples, that tackle typical problem and “pattern” of usage of language constructs in combination.

hope this makes sense,

Dan

p.s. I start to see the /. utility – it;s like a key, value look up – given a “key pattern” it returns a (list of fitting) value(s).
