Myles:

I'm still new to Mathematica and the Wolfram Language. But, as I understand things, Mathematica is a general purpose tool that for most functions require entering text in some fashion.

Given your question, I am assuming that your son would not be able to use a speech-to-text interface. But for those that can use speech-to-text, I think Mathematica's natural language interface would work well.

That said, I do believe it would be possible for someone to develop applications in Mathematica that would allow your son to work through math problems utilizing the power of Mathematica and the Wolfram language.

This is a great community that I'm sure would like to help. But first, it would help to understand what sort of math problems your son is ready to tackle.

Thanks very much, Kyle, for the sample draft palette framework. Thanks, Eric, for the reply also.

I've done a little more research and here's what I came up with, more details about what I'm looking for:

Questions : How can I make a workbook automatically execute a series of commands when it first loads (ie. set magnification to 150%, and re-evaluate the workbook) ?

Goals:

• Define a core Palette GUI framework that can be expanded on as my son moves from grade 8 math to more advanced math.

• If a symbol (such as !) is available instead of an underlying function (Factorial), hide the function and promote use of the symbol, go for simplicity, hide underlying transformations whenever possible (ie. + instead of Add). Only expose functions that don't have an underlying symbolic representation.

• Ensure complex interactions can occur with the minimum amount of input. Each input event (ie. click, double-click, drag, right click) has a cost (each of these takes a physical toll and is time consuming - takes at least five seconds to zoom in and select a particular point, ten seconds for click/drag manoeuvres).

• Consider creating an outline browser If multiple levels of menus end up having to be created, provide a way to interact with this browser via simple pointer movements (which naturally follow my son's eye motions and have much less cost than the events mentioned above). Dwelling on a control for longer than a minimum length of time would cause it to be selected.

Constraints :

• My son uses a Tobii I12+ eye tracking PC, with 1024x768 resolution, 9.5" wide and 7.5" tall, which requires 20% of the width of the screen for an eye tracking computer control (which appears as a single or double-column rectangle on the right or left of the screen).

• My son uses a zoom or dwell paradigm to select a palette cell (for dwell, targets must be no smaller than a 1" (105 pixel) diameter circle, but for zoom the targets can be as small as a 1/5" (21 pixel) diameter circle with the 5x zoom this feature offers, since the computer zooms into a particular part of the screen until a more exact point can be picked). Dwell is faster (only around 1/2 second to click) but also greatly limits the number of targets possible on the screen. Zoom takes around 5 seconds to zoom in and click, but offers greater accuracy.

• As part of his eye tracking controls, my son can bring up an eye tracking keyboard, this will consume the bottom half of the screen, overlaying the palette, which must also take up the bottom half of the screen and 80% of its width.

-The notebook would consume the top half of the screen.

• The Tobii I12+ requires a nominal distance of around 24 inches from the eye, making a 125-150% magnification factor required in order for my son to be able to most easily perceive and access the notebook.

Required use cases :

• Typesetting (as opposed to free form drawing on a paint app)

• Manual step-by-step equations, introductory algebra, functions

• Inequalities and graphing

• Plotting / visualization

• Basic geometry and area/perimeter calculations.

Required functionality:

• Core

  • evaluate current cell

  • Evaluate current workbook

  • New cell (wolfram lanaguage)

  • Keyboard - like in Class Assistant - As an alternative to my son using his eye tracking keyboard as this allows zoom to click.

  • Typesetting (from Basic Math Assistant, Typesetting, superscript tab) - ?? some duplication from calculator - where would be the best place for these ???

    • fraction
    • square root
    • superscript (ie for powers)
    • matching parenthesis
    • matching square brackets
    • matching braces
    • matching single vertical lines (for absolute value)
    • Add < and > symbols to allow inequalities to be typed.

  • Color controls

  • Navigation

    • Add tab and shift/tab for traversing function templates, these are missing in current template

  • Calculator from basic math assistant

    • basic tab - mostly the same as the built-in palette,

      • Create Input Cell (create Wolfram input cell)

      • remove "input from above and output from above"

      • add Previous output (%), two outputs ago (%%), %%%, etc. to allow relative access and progressively solve an equation step-by-step manually. Relative reference are important to ensure the workbook doesn't get messed up if "evaluate workbook" is selected .

      • Single equals (assignment)

      • double equals (to express equations) Equals[]

      • remove "Create text cell" and "make template".

      • add semicolon so multiple lines can be provided in a single input without expecting any output.

    • advanced tab

      • add create free-form input cell or alpha input cell (double-check work by use of Alpha step-by-step feature).

      • remove all integration, derivative, definite product and matrix typesetting, keep definite sum and definite product

  • Basic commands

    • math constants and functions tab

      • mathematical constants : remove "more"

      • numeric functions : remove all the "Wave" functions

      • trig : only sin/cos/tan

    • algebra commands / expressions and equations

• Basic equation handling

  • Add/subtract/multiply/divide both sides of an equation (AddSides, SubtractSides, MultiplySides, DivideSides)
  • simplify / expand / factor / reduce

• Visualization

  Plot, Plot3d, Graphics

• Geometry

  Point, Line, Circle, Rectangle, Triangle, Polygon, Parallelogram, Cube, Sphere, Cylinder, Cone

I think there are two big misunderstandings. I'm misunderstanding how the eye tracking software is used and you are expecting something from the buttons that I'm not. Can you send a video of the "zoom and click" process, either all example from YouTube or one you create. Is it true that no matter what when your son like at a specific spot on the screen it will zoom 5x and click wherever he ends up looking after the zoom? So every button should be the smallest possible size, which is limited to be larger than 25pt font. Is that correct? These constraints are what allow a specific sketch to be made. Also please feel free to sketch ideas, it's often much more effective than words with me.

For the second misunderstanding about the workflow of a mathematica notebook we should be explicit about whether the standard workflow is what you expect. Normally one edits a piece of code by applying multiple templates in succession to create a large expression. Then one executed that expression to Connor the result. It does not behave like a calculator where you got one button at a time and each button performs an operation. If you want the calculator workflow then there are other simpler options. The editing process of a standard notebook is going to be cumbersome, but will allow your son to create more complex things from scratch. If you want the calculator workflow then you might want to explore a scientific calculator that saves a history of calculations and executes the calculation at every button press. Those exist and making a custom one here might end up worse.

Also, I've clearly done a bad job explaining my drawings, sorry I don't know how to be clearer.

Hi, Ahmed. Thanks for the reply. My son wouldn't be able to use a standard keyboard effectively, I am looking for a simplified graphical user interface to be a bridge between him and Mathematica. I understand Mathematica has a UIkit, so I'm wondering whether anybody has used this to develop a simplified interface to Mathematica online (access via web browser). Currently, my son uses a 5x4 grid (in which the screen is divided into 20 regions, with each region having a specific function that he accesses via fixing his gaze on the region and holding his gaze there).

The typical way he controls an external application (such as a web browser) is to shrink his grid to occupy only a part of the screen and allow his eyes to "become the mouse" and move/click onto the web page. Keyboard access is more tricky, which is why having a completely mouse-driven input model would be a much better idea.

Some links that describe current standards for online accessibility : https://www.w3.org/TR/WCAG21/

Mathematica has a Command Line Interface as well. I am using a Macintosh. There is a terminal that can be used the way MS/PC-DOS is. Would that be a workable interface, Windows Command Prompt? In graphical mode, palettes can be used to enter data, symbols, equations and operators more readily than solely by keyboard.

You've told us your son is a quadriplegic. Can a head pointer be used? Is there enough head motion control? I suppose eye control needs a certain amount of stability to make a selection, and have the machine accept the input. I know of people with spinal cord injury, but this sounds more like motor control issues and possible spasticity. How accomodations can be made is interesting. I have followed what can be done for those with ALS, amyotrophic lateral sclerosis. Technology has aided fighter pilots with on screen displays, to what the general public has available in automobiles as a better human machine interface.

To Myles Dear,

Mathematica Online might be what you are after. A web browser based way to do work in Mathematica. I've perused https://www.tobiidynavox.com/products/software/ . I saw this note: Tobii Dynavox has announced the End of Sale of the I-Series+ (I-12+ and I-15+) As of December 13th, 2019, these devices will no longer be available for sale. Mathematica Online might be the solution as it is web based. If the Dynavox can reach the Internet then it should work. Whether it is Windows based or iOS, it should work. Alternatively the Mathematica Front End might work on the Dynavox. The Front End would communicate with a kernel. The kernel could be on your home server. Or outside the house. Again Mathematica Online is probably the way to go. Are you aware of Mathematica running on Raspberry Pi? The Dynavox could be linked to a Raspberry Pi. Mathematica runs on Raspberry Pi on a free license. I have not set up a Virtual Machine to run a Raspberry Pi. The hardware is under $100, but it has so many variations, options, versions. Windows can run on it too. Let this Forum know what ideas you need to help your son reach his computing goal.

Send me a PM and we can have a zoom chat. I teach assistive tech at MIT and am an avid Wolfram user.

It is good to see that Kyle Keane has reached out to you. I am also curious if I can help a bit. Both the Basic Math Assistant and Classroom Assistant palettes were created with the idea that a keyboard was not available and they could be made large by changing the magnification level. However, as you have noticed, they are packed with functionality that is not useful to your son is this year. How would you suggest modifying the palette(s) so they could better serve your son? You can email me directly: ericschulz at mac.com.

This is good info Myles.

What is the size (in pixels) of a single on-screen keyboard button? If we stick a tiny template indicator inside of a button of the same size, will your son be able to see it? Currently the keyboard buttons have just a single easy to read letter in each button, will a template with thin lines inside of their work? Can you verify the smallest template that will be comfortable for him to read?

If we follow the keyboard button sizes, then we can fit a list of topics at the top, subtopics below that, a row of templates from the subtopic, and finally a row of commonly used buttons below that.

If the screen is 1024x768 and the tobii controls on the left are 15% of the width and we need the top half of the screen to display the math he is inputting, then we have about 870x380 pixels for the control area. I guess that means the buttons are around 65 pixels wide with 10 pixel padding on each side?

Attached is a notebook that you can modify to test various size buttons and layouts that will generate an image with controllable pixel size so you can open it on his computer to see if it look as you want and your son can read it.

Attachments:

test image layout.nb

Ah I think you want him to write things that look like typeset math not code but here is the code-ier version which we could use for some of the subtopics even if some subtopics are using templates. Having simple code commands might be very useful for things like Simplify[] and Solve[]

I have downloaded a desktop copy of Mathematica onto my home server, this is a big software package, I'm not completely sure it would run on my son's small resource-constrained Tobii I12+ eye tracking computer. I'm quite interested in looking into palettes - I agree, the basic palettes that ship with Mathematica are much too small (typically eye tracking targets can be no smaller than the size of a nickel, and I haven't yet experimented with zoom-dwell-to-click control models with my son that might allow him access to smaller targets). Currently, he is using dwell-to-click against fairly large targets.

Thanks for the replies, everybody. I really appreciate your kind interest in my son's development.

1. Could you please give me some pointers on how I can run Mathematica on my resource-constrained PC (using the palettes feature), while running the kernel on my more capable home-based server, with both located on my home network? Would doing this require more than one license?

2. I had a meeting with Andy Giddens (Sales) and Andy Dorsett (Sales Engineer) and they explained I could set magnification settings on a palette to ensure the targets were bigger for my son. They also recommended use of the CreatePalette command. This looks like a fairly complex command, does anybody have any examples on its use (other than the very few examples in the product documentation)? I am a software developer and am a quick study.

3. I looked at the built-in Classroom Assistant palette and although it had some good things my son would benefit from (basic algebraic entry, expressions and equations, navigation, evaluation to name a few), only around 40% of it would be immediately useful. Rather than starting from scratch, I'd love to be able to take a copy of the code that generated this template and cut it down to include only the basics for an early learner. Any idea how to best approach this ?

4. I'm attaching a sample 2d exercise I created to demonstrate to my son's tutor that Mathematica could be used to help my son with his daily work. Any idea why the Graphics[] function did not accept a Rectangle[] object as input ?

Attachments:

Kyram2D_Exercise.nb

Regarding item #1, I believe I've found a link that may help, I'll investigate further : https://reference.wolfram.com/language/howto/ConnectToARemoteKernel.html

Hi, Kyle. For your first point, please look at the enclosed video that will hopefully explain exactly what a Grid 3 zoom mouse control looks like (which I call zoom-to-select in my earlier comment). First, the function (left click, right click, double click, drag) is selected, and then the user moves their eyes and fixes their attention on a point. The zoom proceeds, with the user making course corrections during the progress of the zoom until the exact point is identified. The zoom ends, the original screen is restored, and the mouse action then occurs.

If my son deselects the zoom function he would then be free to read the contents of the screen without "interference" from the zoom process, and my request is that all text in the non-zoomed screen be 25 point or larger. He would not be reading anything while the zoom mouse is active. I understand that this means he might have to traverse several screens in order to see a whole sentence of help text.

For your second point, I am not looking for just a simple calculator workflow, I want to open up the full power of the tool to my son, whether it be typesetting, entering functions, or even writing programs. I understand that in order to get a cell ready for evaluation that it may take multiple palette functions to enter the data appropriately before evaluating the entire cell's contents, and I see now why the play button is necessary in order to trigger that evaluation. Sorry for my misunderstanding.

Attachments:

2021-02-27 15.52...mp4

Thanks, Kyle, I'll take a look at this second draft skeleton. As noted below in my Dec 30 post, any buttons cannot be smaller than 21 pixels by 21 pixels (which will appear five times larger when completely zoomed into with his zoom mouse).

A sad piece of news : my wonderful son passed away in my arms on July 31st of this year.

I would be honored if you could join us in the celebration of his life by subscribing and viewing the five videos we have produced : https://forever.link/arbormemorial-kelly-orleans/kyram-dear

This topic remains incredibly pertinent, however, as the lack of an onramp for those with physical disabilities to the Wolfram ecosystem remains a barrier that must still be surmounted.

I strongly believe, more so as I continue to listen to Stephen Wolfram's videos, that having this kind of onramp for those with physical disabilities who rely on assistive technology to perceive the world and interact with it would expose many fine new minds to the wonderful ecosystem Stephen has created. I truly believe we need EVERY voice to move forward to develop this New Kind of Science.

Who is with me?