Hi, Ahmed. Thanks for the reply. My son wouldn't be able to use a standard keyboard effectively, I am looking for a simplified graphical user interface to be a bridge between him and Mathematica. I understand Mathematica has a UIkit, so I'm wondering whether anybody has used this to develop a simplified interface to Mathematica online (access via web browser). Currently, my son uses a 5x4 grid (in which the screen is divided into 20 regions, with each region having a specific function that he accesses via fixing his gaze on the region and holding his gaze there).

The typical way he controls an external application (such as a web browser) is to shrink his grid to occupy only a part of the screen and allow his eyes to "become the mouse" and move/click onto the web page. Keyboard access is more tricky, which is why having a completely mouse-driven input model would be a much better idea.

Some links that describe current standards for online accessibility : https://www.w3.org/TR/WCAG21/

My son is nonverbal, so would rely on eye based access to interface with his computer. He is doing math at a Grade 8 level and we are bridging him with various online tools manually. However, he will reach a point very soon in which doing things in his head becomes next to impossible and he requires some kind of tool to allow him to write things such as algebraic equations so he can reduce/solve them. Rather than just having him use a "dumb" sketchpad app, I am looking for tools that could provide a higher level of value and help him engage with his math more independently.

Mathematica has a Command Line Interface as well. I am using a Macintosh. There is a terminal that can be used the way MS/PC-DOS is. Would that be a workable interface, Windows Command Prompt? In graphical mode, palettes can be used to enter data, symbols, equations and operators more readily than solely by keyboard.

You've told us your son is a quadriplegic. Can a head pointer be used? Is there enough head motion control? I suppose eye control needs a certain amount of stability to make a selection, and have the machine accept the input. I know of people with spinal cord injury, but this sounds more like motor control issues and possible spasticity. How accomodations can be made is interesting. I have followed what can be done for those with ALS, amyotrophic lateral sclerosis. Technology has aided fighter pilots with on screen displays, to what the general public has available in automobiles as a better human machine interface.

I have downloaded a desktop copy of Mathematica onto my home server, this is a big software package, I'm not completely sure it would run on my son's small resource-constrained Tobii I12+ eye tracking computer. I'm quite interested in looking into palettes - I agree, the basic palettes that ship with Mathematica are much too small (typically eye tracking targets can be no smaller than the size of a nickel, and I haven't yet experimented with zoom-dwell-to-click control models with my son that might allow him access to smaller targets). Currently, he is using dwell-to-click against fairly large targets.

To Myles Dear,

By the way, have you been here: https://www.wolfram.com/legal/accessibility/wolfram-mathematica/ ?

In listening to National Public Radio, I learned much from Edward Reynolds Price (February 1, 1933 – January 20, 2011). His cancer treatment with surgery and radiation led to paralysis below the waist. Chuck Close,[Born: July 5, 1940 (age 80 years)], the artist came up with this phrase: ‘temporarily abled.’ Mr. Price embraced it and ran with it[so to speak]. https://adogcalledpain.wordpress.com/2016/01/04/reynolds-price-a-whole-new-life-an-illness-and-a-healing/ https://www.washingtonpost.com/archive/entertainment/books/1994/06/12/though-much-is-taken-much-abides/e06d08f7-e315-42d2-abd7-4b9667ec5daa/ http://chuckclose.com

Regarding item 4, Andy D responded saying he didn't know why Graphics[] did not accept Rectangle[] but said that the developers must have had their reasons for doing it that way. Sometimes trial and error is needed to make things work !

Hello Myles Dear,

For item #3, if you only want to remove some things from the Classroom Assistant Palette for your son's use, here is something you might try: 1. Open the Classroom Assistant Palette. 2. Right-click somewhere inside. 3. Click "Generate Notebook." 4. Manually select and delete parts of the palette in the new notebook graphically. 4. Select the cell in the notebook. 5. In the Palettes menu, click "Generate Palette from Selection."

This process may be tedious. I hope it helps though. Maybe someone has a less tedious solution.

Send me a PM and we can have a zoom chat. I teach assistive tech at MIT and am an avid Wolfram user.

It is good to see that Kyle Keane has reached out to you. I am also curious if I can help a bit. Both the Basic Math Assistant and Classroom Assistant palettes were created with the idea that a keyboard was not available and they could be made large by changing the magnification level. However, as you have noticed, they are packed with functionality that is not useful to your son is this year. How would you suggest modifying the palette(s) so they could better serve your son? You can email me directly: ericschulz at mac.com.

Thanks for the reply, Jiabiana. I was able to get my workbook evaluating correctly, after reading the documentation fine print, I see Rectangle accepts two points, while Triangle accepts a list of points. Both are working as inputs to Graphics[] now.

Thanks very much, Kyle, for the sample draft palette framework. Thanks, Eric, for the reply also.

I've done a little more research and here's what I came up with, more details about what I'm looking for:

Questions : How can I make a workbook automatically execute a series of commands when it first loads (ie. set magnification to 150%, and re-evaluate the workbook) ?

Goals:

• Define a core Palette GUI framework that can be expanded on as my son moves from grade 8 math to more advanced math.

• If a symbol (such as !) is available instead of an underlying function (Factorial), hide the function and promote use of the symbol, go for simplicity, hide underlying transformations whenever possible (ie. + instead of Add). Only expose functions that don't have an underlying symbolic representation.

• Ensure complex interactions can occur with the minimum amount of input. Each input event (ie. click, double-click, drag, right click) has a cost (each of these takes a physical toll and is time consuming - takes at least five seconds to zoom in and select a particular point, ten seconds for click/drag manoeuvres).

• Consider creating an outline browser If multiple levels of menus end up having to be created, provide a way to interact with this browser via simple pointer movements (which naturally follow my son's eye motions and have much less cost than the events mentioned above). Dwelling on a control for longer than a minimum length of time would cause it to be selected.

Constraints :

• My son uses a Tobii I12+ eye tracking PC, with 1024x768 resolution, 9.5" wide and 7.5" tall, which requires 20% of the width of the screen for an eye tracking computer control (which appears as a single or double-column rectangle on the right or left of the screen).

• My son uses a zoom or dwell paradigm to select a palette cell (for dwell, targets must be no smaller than a 1" (105 pixel) diameter circle, but for zoom the targets can be as small as a 1/5" (21 pixel) diameter circle with the 5x zoom this feature offers, since the computer zooms into a particular part of the screen until a more exact point can be picked). Dwell is faster (only around 1/2 second to click) but also greatly limits the number of targets possible on the screen. Zoom takes around 5 seconds to zoom in and click, but offers greater accuracy.

• As part of his eye tracking controls, my son can bring up an eye tracking keyboard, this will consume the bottom half of the screen, overlaying the palette, which must also take up the bottom half of the screen and 80% of its width.

-The notebook would consume the top half of the screen.

• The Tobii I12+ requires a nominal distance of around 24 inches from the eye, making a 125-150% magnification factor required in order for my son to be able to most easily perceive and access the notebook.

Required use cases :

• Typesetting (as opposed to free form drawing on a paint app)

• Manual step-by-step equations, introductory algebra, functions

• Inequalities and graphing

• Plotting / visualization

• Basic geometry and area/perimeter calculations.

Required functionality:

• Core

  • evaluate current cell

  • Evaluate current workbook

  • New cell (wolfram lanaguage)

  • Keyboard - like in Class Assistant - As an alternative to my son using his eye tracking keyboard as this allows zoom to click.

  • Typesetting (from Basic Math Assistant, Typesetting, superscript tab) - ?? some duplication from calculator - where would be the best place for these ???

    • fraction
    • square root
    • superscript (ie for powers)
    • matching parenthesis
    • matching square brackets
    • matching braces
    • matching single vertical lines (for absolute value)
    • Add < and > symbols to allow inequalities to be typed.

  • Color controls

  • Navigation

    • Add tab and shift/tab for traversing function templates, these are missing in current template

  • Calculator from basic math assistant

    • basic tab - mostly the same as the built-in palette,

      • Create Input Cell (create Wolfram input cell)

      • remove "input from above and output from above"

      • add Previous output (%), two outputs ago (%%), %%%, etc. to allow relative access and progressively solve an equation step-by-step manually. Relative reference are important to ensure the workbook doesn't get messed up if "evaluate workbook" is selected .

      • Single equals (assignment)

      • double equals (to express equations) Equals[]

      • remove "Create text cell" and "make template".

      • add semicolon so multiple lines can be provided in a single input without expecting any output.

    • advanced tab

      • add create free-form input cell or alpha input cell (double-check work by use of Alpha step-by-step feature).

      • remove all integration, derivative, definite product and matrix typesetting, keep definite sum and definite product

  • Basic commands

    • math constants and functions tab

      • mathematical constants : remove "more"

      • numeric functions : remove all the "Wave" functions

      • trig : only sin/cos/tan

    • algebra commands / expressions and equations

• Basic equation handling

  • Add/subtract/multiply/divide both sides of an equation (AddSides, SubtractSides, MultiplySides, DivideSides)
  • simplify / expand / factor / reduce

• Visualization

  Plot, Plot3d, Graphics

• Geometry

  Point, Line, Circle, Rectangle, Triangle, Polygon, Parallelogram, Cube, Sphere, Cylinder, Cone

This is good info Myles.

What is the size (in pixels) of a single on-screen keyboard button? If we stick a tiny template indicator inside of a button of the same size, will your son be able to see it? Currently the keyboard buttons have just a single easy to read letter in each button, will a template with thin lines inside of their work? Can you verify the smallest template that will be comfortable for him to read?

If we follow the keyboard button sizes, then we can fit a list of topics at the top, subtopics below that, a row of templates from the subtopic, and finally a row of commonly used buttons below that.

If the screen is 1024x768 and the tobii controls on the left are 15% of the width and we need the top half of the screen to display the math he is inputting, then we have about 870x380 pixels for the control area. I guess that means the buttons are around 65 pixels wide with 10 pixel padding on each side?

Attached is a notebook that you can modify to test various size buttons and layouts that will generate an image with controllable pixel size so you can open it on his computer to see if it look as you want and your son can read it.

Attachments:

test image layout.nb

Hi Myles, sorry i got sick after the holidays and missed your message. Here is my proposed interface that I think accounts for all constraints in a fairly usable way. Everything is drawn to scale for a total control area of 384x820 pixels. The large squares are 105px wide so they can be clicked directly without zoom, only the most essential and frequent operations get that size. Everything else will require some amount of zoom and click (the text box buttons and template buttons).

The top view will apply the template directly in the notebook to the current selection when a template button is activated and the bottom view will display the extra info when a template button is activated. In the bottom view you can apply the activated template button by clicking the check mark.

The operations in the smaller menu illustrated by the 'Cell Operations' will not get any extra info and will be limited to the common interface operations that are typically in the standard menu bar or are usually performed with a mouse and keyboard like clicking a cell bracket and hitting some key combo.

The play button will run the code in the current cell, all other notebook operations will be less common and should go into the smaller menu on the right, such as "run all cells in notebook from top".

Let me know what you think... ideally you will print the image to the correct scale so you can hold it in front of your kid to ensure they can read all text and recognize the tiny template buttons. I will also email you the powerpoint that I used to make the mockup so you can change things to test other symbols.

Instead of tiny template buttons we can alternatively use function templates, I will tweak the powerpoint and send another image in a minute.

Thanks for the input, Kyle. I showed this to my son today and I have a few early reactions :

1) Any font smaller than around 25 point will be too difficult for my son to see easily, keep in mind the eye tracking computer must be around 19 inches from his face.

2) I think a better approach rather than integrated functionality and help text would be to have an "action" mode and a "help" mode where the real estate is used differently. The action mode would execute a function when a symbol is picked, while the help mode would display a help screen when the same symbol is picked. So we would have two identical inputs leading to two different outputs depending on the mode in which he was in. He should be able to transition between modes easily.

3) I don't like the idea of a "two hit" approach, as each click has a cost. I'd rather have a WSIWYG approach where when he selects a function it either does that function or displays help about that function. I don't see that the "Play" button would be very useful to him.

4) I don't quite understand the different between the play button and the check mark in your template.

5) I don't understand what the "Eye" icon is supposed to do.

6) Please be advised that my son's Grid 3 control is either programmed to operate in dwell-to-select or zoom-to-select mode, it is not possible to have him change between these two. Thus, he would use a zoom mouse to select all content. It may be excessive to consume real estate for big buttons because they themselves would be magnified 5x when my son zooms in.

Hi, Kyle. For your first point, please look at the enclosed video that will hopefully explain exactly what a Grid 3 zoom mouse control looks like (which I call zoom-to-select in my earlier comment). First, the function (left click, right click, double click, drag) is selected, and then the user moves their eyes and fixes their attention on a point. The zoom proceeds, with the user making course corrections during the progress of the zoom until the exact point is identified. The zoom ends, the original screen is restored, and the mouse action then occurs.

If my son deselects the zoom function he would then be free to read the contents of the screen without "interference" from the zoom process, and my request is that all text in the non-zoomed screen be 25 point or larger. He would not be reading anything while the zoom mouse is active. I understand that this means he might have to traverse several screens in order to see a whole sentence of help text.

For your second point, I am not looking for just a simple calculator workflow, I want to open up the full power of the tool to my son, whether it be typesetting, entering functions, or even writing programs. I understand that in order to get a cell ready for evaluation that it may take multiple palette functions to enter the data appropriately before evaluating the entire cell's contents, and I see now why the play button is necessary in order to trigger that evaluation. Sorry for my misunderstanding.

Attachments:

2021-02-27 15.52...mp4

A sad piece of news : my wonderful son passed away in my arms on July 31st of this year.

I would be honored if you could join us in the celebration of his life by subscribing and viewing the five videos we have produced : https://forever.link/arbormemorial-kelly-orleans/kyram-dear

This topic remains incredibly pertinent, however, as the lack of an onramp for those with physical disabilities to the Wolfram ecosystem remains a barrier that must still be surmounted.

I strongly believe, more so as I continue to listen to Stephen Wolfram's videos, that having this kind of onramp for those with physical disabilities who rely on assistive technology to perceive the world and interact with it would expose many fine new minds to the wonderful ecosystem Stephen has created. I truly believe we need EVERY voice to move forward to develop this New Kind of Science.

Who is with me?