Code puzzles: turning docs into educational games

GROUPS:

Teaching programing and assessing learning progress is often a very custom task. I wanted to create a completely automated "practically" infinite stream of random puzzles that guide a leaner towards improving programing skills. I think the major problem is content creation. To test whether the learner knows a programming concept, an exercise needs to be wisely designed. And it is better to have a randomized set of such exercises to definitely test the knowledge and exclude guesses and cheating and so on. Often creating such educational materials is very tedious, time consuming, and manual. Exactly like creating good documentation. I will explain one simple idea of using docs to make an educational game. This is just a barebone prototype to clearly follow the inner workings (try it out & share: https://wolfr.am/bughunter ). Please comment with feedback on how we can develop this idea further.

Introduction: efficient use of resources

The docs are the finest wealth and depth of information and should be explored beyond their regular usage. Manual painstaking time consuming effort of creating good programing documentation should be used to its fullest potential. An automated game play would be a novel take on docs. We can use existing code examples in docs to randomly pull pieces of code and make programing exercises automatically. Being able to read code and find bugs is, in my experience, one of the most enlightening practices. The goal of the linked above game is to find a defect of the input code (bug) and fix it. Hence, the "bug hunter". There are just 2 possible outcomes of a single game cycle, --- and after each you can "try again":

Core game code: making puzzles

Wolfram Language (WL) documentation is one of the best I've seen. It has pages and pages of examples starting from simple ones and going though the all details of the usage. Moreover the docs are written in WL itself and furthermore, WL can access docs and even has internal self-knowledge of its structure via WolframLanguageData. For instance, this is how you can show a relationship community graph for symbols related to GatherBy:

WolframLanguageData["GatherBy", "RelationshipCommunityGraph"]


We can use WolframLanguageData to access docs examples and then drop some parts of the code. The puzzle is then for the learner to find what is missing. For the sake of clarity designing a small working prototype lets limit test WL functions and corresponding docs' pages to some small number. So out of ~5000 (and we just released a new addition):

WolframLanguageData[] // Length


4838

built in symbols I just take 30

functions = {"Append", "Apply", "Array", "Cases", "Delete", "DeleteCases", "Drop", "Except",
"Flatten", "FlattenAt", "Fold", "Inner", "Insert", "Join", "ListConvolve", "Map", "MapThread",
"Nest", "Outer", "Partition", "Prepend", "ReplacePart", "Reverse", "RotateLeft", "RotateRight",

functions // Length
30


that are listed on a very old but neat animated page of some essential core-language collection. I will also add some "sugar syntax" to potential removable parts of code:

sugar = {"@@", "@", "/@", "@@@", "#", "^", "&"};


So, for instance, out of the following example in docs we could remove a small part to make a puzzle:

Here is an example of "sugar syntax" removal, which for novice programmers would be harder to solve:

Next step is to define a function that can check if a string is a built-in symbol (function, all 5000) or if it is some of sugar syntax we defined above:

ClearAll[ExampleHeads];
Select[
Cases[e,_String, Infinity],
(NameQ["System"<>#]||MemberQ[sugar,#])&&#=!="Input"&
]


Next function essentially makes a single quiz question. First it randomly picks a function from list of 30 symbols we defined. Then it goes to the doc page of that symbol to the section called "Basic Examples". It finds a random example and removes a random part out of it:

ranquiz[]:=Module[
ranexa=RandomChoice[WolframLanguageData[ranfun,"DocumentationBasicExamples"]][[-2;;-1]];
{
ranexa[[2]],
ranfun
}
]


Now we will define a few simple variables and tools.

Image variables

I keep marveling how convenient it is that Mathematica front end can make images to be part of code. This makes notebooks a great IDE:

Databin for tracking stats

It is important to have statistics of your learning game: to understand how to improve it where the education process should go. Wolfram Datadrop is an amazing tool for these purposes.

We define the databin as

bin = CreateDatabin[<|"Name" -> "BugHunter"|>]


Deploy game to the web

To make an actual application usable by everyone with internet access I will use Wolfram Development Platform and Wolfram Cloud. First I define a function that will build the "result of the game" web page. It will check is answer is wrong or right and give differently designed pages accordingly.

quiz[answer_String,check_String,fun_String]:=
(
Grid[{
Grid[{{Style["Right! You got the bug!",40,Darker@Red,FontFamily->"Chalkduster"]},{First[imgs]}}],
Grid[{{Style["Wrong! The bug got you!",40,Darker@Red,FontFamily->"Chalkduster"]},{Last[imgs]}}]
]},

{Row[
"|",
"|",
Spacer[10]
]},
{Style["===================================================="]},
{hyperlink["An Elementary Introduction to the Wolfram Language","https://www.wolfram.com/language/elementary-introduction"]},
{logo}
}]
)


This function is used inside CloudDeploy[...FormFunction[...]...] construct to actually deploy the application. FormFunction builds a query form, a web user interface to formulate a question and to get user's answer. Note for random variables to function properly Delayed is used as a wrapper for FormFunction.

 CloudDeploy[Delayed[
quizloc=ranquiz[];
FormFunction[
{{"code",None} -> "String",
{"x",None}-><|
"Input"->StringRiffle[quizloc[[3;;4]],","],
"Interpreter"->DelimitedSequence["String"],
"Control"->Function[Annotation[InputField[##],{"class"->"sr-only"},"HTMLAttrs"]]|>},
quiz[#code,#x[[1]],#x[[2]]]&,
AppearanceRules-> <|
"Title" -> Grid[{{title}},Alignment->Center],
"MetaTitle"->"BUG HUNTER",
"Description"-> Grid[{
{Style["Type the missing part of input code",15, Darker@Red,FontFamily->"Ayuthaya"]},
{Rasterize@Grid[{
{"In[1]:=",quizloc[[1]]},
{"Out[1]=",quizloc[[2]]}},Alignment->Left]}
}]
|>]],
"bughunter",
Permissions->"Public"
]


The result of the deployment is a cloud object at a URL:

CloudObject[https://www.wolframcloud.com/objects/user-3c5d3268-040e-45d5-8ac1-25476e7870da/bughunter]


with the short version:

URLShorten["https://www.wolframcloud.com/objects/user-3c5d3268-040e-45d5-8ac1-25476e7870da/bughunter", "bughunter"]


https://wolfr.am/bughunter

And we are done! You can go at the above URL and play.

Further thoughts

Here are some key points and further thoughts.

• Automation of content: NO new manual resource development, use existing code bases.
• Automation of testing: NO manual labor of grading.
• Quality of testing: NO multiple choice, NO guessing.
• Quality of grading: almost 100% exact detection of mistakes and correct solutions.
• Fight cheating: clear to identify question type "find missing code part" helps to ban help from friendly forums (such as this one).
• Almost infinite variability of examples if whole docs system is engaged.
• High range from very easy to very hard examples (exclusion of multiple functions and syntax can make this really hard).

Improvements:

• Flexible scoring system based on function usage frequencies.
• Optional placeholder as hint where the code is missing.
• Using network of related functions (see above) to move smoothly through the topical domains.
• Using functions frequency to feed easier or harder exercises based on test progress.

1 month ago
11 Replies

Reserved for analytics

1 month ago
 Frederick Wu 1 Vote The idea is fresh and new. However, I am afraid, if the kids would really love to play it. The illustrated examples appears all low level errors, which could be detected and assisted by WL grammar color system easily.In future, the kids are very very very smart. That means they would become boring very quickly either. If we want those kind of projects get to work, we should design it really playful. Maybe, we should ask and test with kids.
1 month ago
 @Frederick Wu thanks for taking a look and the comments. However, I am afraid, if the kids would really love to play it. I was actually thinking about targeting adults who are a bit above the beginner level. A few folks I tested with were adults and they enjoyed playing it. Some said that the bugs were to scary though ;-) The illustrated examples appears all low level errors, which could be detected and assisted by WL grammar color system easily. If I understood what you mean by "grammar color system" correctly, then even those examples I are showed in the post already undetected by it - there is no colors in front end highlighting these:Moreover a few testers suggested I use an optional indicator to show where exactly the code is missing, --- as a hint to help the solution. I also think those cases that will trigger "grammar color system" are quite good because learners need to have a good habit and understanding how code highlighting works to catch errors during a real programing workflow. In future, the kids are very very very smart. That means they would become boring very quickly either. If we want those kind of projects get to work, we should design it really playful. Maybe, we should ask and test with kids. Yes, if you mean young kids, they would probably need another type of game or at least some better game-play dressing for this idea. I'd love to hear your ideas who to make this work. If you come up with anything please share.
1 month ago
 Manjunath Babu 1 Vote Hello @Vitaliy Kaurov I enjoyed playing your Bug Hunter web app. The code and the thought process behind this application is brilliant. I did not know the Legacy Animations Documentation pages existed. Its a very nice and intuitive way of presenting it to users.I got a similar idea at Wolfram Summer School 2016 and built a simpler prototype. Link: http://community.wolfram.com/groups/-/m/t/886715 . Its called Infinite Coding Problems Generator in Wolfram Programming Language. I used templates and loaded a CSV file that contains questions from EIWL book. Your web app is so much fun and its giving me new ideas for improving my prototype. These educational applications has tremendous potential and spikes the learning curve in students. I like to see your app growing big like http://challenges.wolfram.com.Thank you Try my web app here: https://wolfr.am/e0t5Zn50
1 month ago
 This is some great work @Manjunath Babu! Could you explain to me briefly here, how did you handle the correctness check? Was it a verbatim-check of the code? In Wolfram Language often different versions of code yield correct result, this is why we call it multi-paradigm language. For example: data = RandomReal[1, {100, 2}]; ListPlot[data] Graphics[Point[data]] Were you taking that into account?
1 month ago
 Manjunath Babu 1 Vote Hello @Vitaliy Kaurov ,I haven't taken Random Functions into account yet.JSON Template looks something like this: [ { "Number": "11.2", "Question": "Make a single string of the whole alphabet, in upper case.", "Answer": "Function[StringJoin[Alphabet[\"Alphabet\"]]]", "Template": "Make a single string of the all Alphabet alphabets, using Function.", "Data": {"Function": ["ToUpperCase", "ToLowerCase"], "Alphabet": ["French","Russian", "Italian", "English","German"]} }, { "Number":"1.3", "Question":"Multiply the whole numbers from 1 to 5.", "Answer":"Times @@ Range[1, Number]", "Template":"Multiply the whole numbers from 1 to Number.", "Data":{"Number": "RandomInteger[{5,10}]"} }, { "Number":"4.1", "Question":"Make a bar chart of Number.", "Answer":"Function[Number]", "Template":"Make a Function of Number.", "Data":{"Function":["BarChart3D","BarChart","PieChart","PieChart3D","ListLinePlot"],"Number":"RandomInteger[50, 4]"} } ] To check for correctness, I execute the originally solution expression and execute the user provided answer expression. If these two match, then its correct.Since Wolfram Language is a symbolic language, it just compares at the lowest level of both executed expressions.However, the problem with this approach is, This prototype doesn't work with Random Function. Since Random function will store the final answer in different values at the lowest level.
1 month ago
 I think there also could be some problem with sorting functions. If the goal is to return a list of elements independent of their order, some approaches may return them in different order. Then SameQ` will not match.
1 month ago
 Arno Bosse 1 Vote This is great! I'm an adult beginner myself, with no formal training in programming. What I continue to find the most challenging aspect of learning the WL is the syntax. You have to get up to speed with the syntax, and quickly, or much of the docs, and most of the great examples shared here won't be comprehensible. This is also why, as much as I enjoyed working through the exercises in SW's new EIWL textbook, I feel that it still places too much emphasis on covering functions and not enough (and not soon enough) on syntax. I've tried out BugHunter a few times and haven't encountered an example yet that removed the 'sugar'. So I agree that allowing a user to pick different areas or themes for testing (for example, 'quiz me on syntax', 'quiz me on functions related to network analysis' etc.) would be a good choice for an initial enhancement. Thank you for making this!