My 14-year-old son is medically complex, including being a quadriplegic. He currently uses software (Grid 3) to communicate ideas and an eye tracker (Tobii) with software that allows him to “type” using his eyes.
I am looking for tools that will allow him to work out a math problem through its various steps (since he cannot hold a pencil and write on paper to show his work), and I thought Mathematica might be a tool that could help.
Has anybody successfully used Mathematica using mouse movements alone and not typing anything on the keyboard?
Hi Myles
I don’t have much experience with Tobii, but I know that it supports an on-screen keyboard with English layout. If your son can successfully click on the buttons of this keyboard using Tobii, then he probably will be able to use it to type in Wolfram notebook. If you are using Windows 10, this page can help
Hi, Ahmed. Thanks for the reply. My son wouldn’t be able to use a standard keyboard effectively, I am looking for a simplified graphical user interface to be a bridge between him and Mathematica. I understand Mathematica has a UIkit, so I’m wondering whether anybody has used this to develop a simplified interface to Mathematica online (access via web browser). Currently, my son uses a 5x4 grid (in which the screen is divided into 20 regions, with each region having a specific function that he accesses via fixing his gaze on the region and holding his gaze there).
The typical way he controls an external application (such as a web browser) is to shrink his grid to occupy only a part of the screen and allow his eyes to “become the mouse” and move/click onto the web page. Keyboard access is more tricky, which is why having a completely mouse-driven input model would be a much better idea.
I’m still new to Mathematica and the Wolfram Language. But, as I understand things, Mathematica is a general purpose tool that for most functions require entering text in some fashion.
Given your question, I am assuming that your son would not be able to use a speech-to-text interface. But for those that can use speech-to-text, I think Mathematica’s natural language interface would work well.
That said, I do believe it would be possible for someone to develop applications in Mathematica that would allow your son to work through math problems utilizing the power of Mathematica and the Wolfram language.
This is a great community that I’m sure would like to help. But first, it would help to understand what sort of math problems your son is ready to tackle.
My son is nonverbal, so would rely on eye based access to interface with his computer. He is doing math at a Grade 8 level and we are bridging him with various online tools manually. However, he will reach a point very soon in which doing things in his head becomes next to impossible and he requires some kind of tool to allow him to write things such as algebraic equations so he can reduce/solve them. Rather than just having him use a “dumb” sketchpad app, I am looking for tools that could provide a higher level of value and help him engage with his math more independently.
The Wolfram Language and Mathematica have several features that I think would make Mathematica more accessible to eye tracking software.
One place to start might be to have some custom palettes created. Palettes are on-screen windows that a user can enter things or trigger actions with a mouse. Unfortunately, the standard palettes that Mathematica ship with our too small for my aging eyes. (I have to keep buying bigger monitors every couple of years.). Fortunately, it is fairly easy to create any size custom palettes that I think can make the full power of Mathematica much more accessible. Check out How to Use Palettes.
If I am correct, custom palettes could bring the full power of Mathematica to your son. But, I must tell you, for many of us, the full power of Mathematica can feel like trying to drink from a fire hose at times.
So, while I suggest you also look into some of the other technologies and content that Wolfram has to offer students. For example Wolfram Alpha Pro for Students. Though, there are others as well. Though, I think Wolfram Alpha will require your son to still do some typing. It has a natural language interface, so you are not having to type so many symbols that can be difficult with on screen keyboards. Though, I think you indicated that on screen keyboards might not meed your needs.
Unfortunately, I do not have access to any eye tracking hardware to build you a custom palette to see if that could get you started that way. Though I wouldn’t mind trying to build you one to see if it could work. The challenge though is that palettes do not work on the web. You would have to have a desktop copy of Mathematica.
Which version of Mathematica do you have? Or are considering? After looking at How to Use Palettes, do you think more reasonably sized buttons on a palette work work for your son?
Mathematica has a Command Line Interface as well.
I am using a Macintosh. There is a terminal that can be used the way MS/PC-DOS is.
Would that be a workable interface, Windows Command Prompt? In graphical mode, palettes can be used to enter data, symbols, equations and operators more readily than solely by keyboard.
You’ve told us your son is a quadriplegic. Can a head pointer be used?
Is there enough head motion control? I suppose eye control needs a certain amount of stability to make a selection, and have the machine accept the input.
I know of people with spinal cord injury, but this sounds more like motor control issues and possible spasticity. How accomodations can be made is interesting. I have followed what can be done for those with ALS, amyotrophic lateral sclerosis. Technology has aided fighter pilots with on screen displays, to what the general public has available in automobiles as a better human machine interface.
Thanks for the reply, Stephen. My son does not have the head motor control to consistently operate head-based switches or to use a head mouse, but he does not have spasticity (in fact, his muscles are limp like wet noodles except for a very little amount of slow left-to-right neck movement). In contrast, his eyes are very fast and he has trained on how to use his gaze-based eye tracking system for several years and has become quite good at it. I think a raw command-line interface would not be feasible for him, but rather a series of graphical elements that allow for higher-level control.
I have downloaded a desktop copy of Mathematica onto my home server, this is a big software package, I’m not completely sure it would run on my son’s small resource-constrained Tobii I12+ eye tracking computer. I’m quite interested in looking into palettes - I agree, the basic palettes that ship with Mathematica are much too small (typically eye tracking targets can be no smaller than the size of a nickel, and I haven’t yet experimented with zoom-dwell-to-click control models with my son that might allow him access to smaller targets). Currently, he is using dwell-to-click against fairly large targets.
Mathematica Online might be what you are after. A web browser based way to do work in Mathematica.
I’ve perused Software & apps for communication, AAC and eye control - Tobii Dynavox Global . I saw this note: Tobii Dynavox has announced the End of Sale of the I-Series+ (I-12+ and I-15+) As of December 13th, 2019, these devices will no longer be available for sale.
Mathematica Online might be the solution as it is web based. If the Dynavox can reach the Internet then it should work. Whether it is Windows based or iOS, it should work.
Alternatively the Mathematica Front End might work on the Dynavox. The Front End would communicate with a kernel. The kernel could be on your home server. Or outside the house.
Again Mathematica Online is probably the way to go.
Are you aware of Mathematica running on Raspberry Pi? The Dynavox could be linked to a Raspberry Pi. Mathematica runs on Raspberry Pi on a free license.
I have not set up a Virtual Machine to run a Raspberry Pi. The hardware is under $100, but it has so many variations, options, versions. Windows can run on it too. Let this Forum know what ideas you need to help your son reach his computing goal.
Thanks for the replies, everybody. I really appreciate your kind interest in my son’s development.
Could you please give me some pointers on how I can run Mathematica on my resource-constrained PC (using the palettes feature), while running the kernel on my more capable home-based server, with both located on my home network? Would doing this require more than one license?
I had a meeting with Andy Giddens (Sales) and Andy Dorsett (Sales Engineer) and they explained I could set magnification settings on a palette to ensure the targets were bigger for my son. They also recommended use of the CreatePalette command. This looks like a fairly complex command, does anybody have any examples on its use (other than the very few examples in the product documentation)? I am a software developer and am a quick study.
I looked at the built-in Classroom Assistant palette and although it had some good things my son would benefit from (basic algebraic entry, expressions and equations, navigation, evaluation to name a few), only around 40% of it would be immediately useful. Rather than starting from scratch, I’d love to be able to take a copy of the code that generated this template and cut it down to include only the basics for an early learner. Any idea how to best approach this ?
I’m attaching a sample 2d exercise I created to demonstrate to my son’s tutor that Mathematica could be used to help my son with his daily work. Any idea why the Graphics[] function did not accept a Rectangle[] object as input ?
Regarding item 4, Andy D responded saying he didn’t know why Graphics[] did not accept Rectangle[] but said that the developers must have had their reasons for doing it that way. Sometimes trial and error is needed to make things work !
I hope you find a great way for Mathematica to help your son! Regarding why Graphics[] did not accept Rectangle[], could you perhaps put an example? I couldn’t find one in the notebook, but if there’s one there, you can direct us to that. Both Graphics[Rectangle[]] and Graphics[{Rectangle[]}] (where you can add more items to the list in the curly braces) should work.
It is good to see that Kyle Keane has reached out to you. I am also curious if I can help a bit. Both the Basic Math Assistant and Classroom Assistant palettes were created with the idea that a keyboard was not available and they could be made large by changing the magnification level. However, as you have noticed, they are packed with functionality that is not useful to your son is this year. How would you suggest modifying the palette(s) so they could better serve your son? You can email me directly: ericschulz at mac.com.
Attached is a notebook that creates a skeleton interface that we can use to further the discussion. Myles can you post your screenshot of your son’s computer so people can see the dimensional constraints (i.e. how many pixels by how many pixels do we have available)? It would be good if you could indicate an ideal button size in pixels as well, that will help size the buttons.
Let me know what you think of the tab+page view generated in the notebook, screenshot attached. We can of course change the sizes and padding around the buttons, but I just wanted to make sure it is generally in the right direction.
