Thanks very much to Eric Schulz for posting this sample / possible implementation.

Attachments:

Dear_Assistant_P...nb

I think there are two big misunderstandings. I'm misunderstanding how the eye tracking software is used and you are expecting something from the buttons that I'm not. Can you send a video of the "zoom and click" process, either all example from YouTube or one you create. Is it true that no matter what when your son like at a specific spot on the screen it will zoom 5x and click wherever he ends up looking after the zoom? So every button should be the smallest possible size, which is limited to be larger than 25pt font. Is that correct? These constraints are what allow a specific sketch to be made. Also please feel free to sketch ideas, it's often much more effective than words with me.

For the second misunderstanding about the workflow of a mathematica notebook we should be explicit about whether the standard workflow is what you expect. Normally one edits a piece of code by applying multiple templates in succession to create a large expression. Then one executed that expression to Connor the result. It does not behave like a calculator where you got one button at a time and each button performs an operation. If you want the calculator workflow then there are other simpler options. The editing process of a standard notebook is going to be cumbersome, but will allow your son to create more complex things from scratch. If you want the calculator workflow then you might want to explore a scientific calculator that saves a history of calculations and executes the calculation at every button press. Those exist and making a custom one here might end up worse.

Also, I've clearly done a bad job explaining my drawings, sorry I don't know how to be clearer.

Thanks very much, Kyle, for the sample draft palette framework. Thanks, Eric, for the reply also.

I've done a little more research and here's what I came up with, more details about what I'm looking for:

Questions : How can I make a workbook automatically execute a series of commands when it first loads (ie. set magnification to 150%, and re-evaluate the workbook) ?

Goals:

• Define a core Palette GUI framework that can be expanded on as my son moves from grade 8 math to more advanced math.

• If a symbol (such as !) is available instead of an underlying function (Factorial), hide the function and promote use of the symbol, go for simplicity, hide underlying transformations whenever possible (ie. + instead of Add). Only expose functions that don't have an underlying symbolic representation.

• Ensure complex interactions can occur with the minimum amount of input. Each input event (ie. click, double-click, drag, right click) has a cost (each of these takes a physical toll and is time consuming - takes at least five seconds to zoom in and select a particular point, ten seconds for click/drag manoeuvres).

• Consider creating an outline browser If multiple levels of menus end up having to be created, provide a way to interact with this browser via simple pointer movements (which naturally follow my son's eye motions and have much less cost than the events mentioned above). Dwelling on a control for longer than a minimum length of time would cause it to be selected.

Constraints :

• My son uses a Tobii I12+ eye tracking PC, with 1024x768 resolution, 9.5" wide and 7.5" tall, which requires 20% of the width of the screen for an eye tracking computer control (which appears as a single or double-column rectangle on the right or left of the screen).

• My son uses a zoom or dwell paradigm to select a palette cell (for dwell, targets must be no smaller than a 1" (105 pixel) diameter circle, but for zoom the targets can be as small as a 1/5" (21 pixel) diameter circle with the 5x zoom this feature offers, since the computer zooms into a particular part of the screen until a more exact point can be picked). Dwell is faster (only around 1/2 second to click) but also greatly limits the number of targets possible on the screen. Zoom takes around 5 seconds to zoom in and click, but offers greater accuracy.

• As part of his eye tracking controls, my son can bring up an eye tracking keyboard, this will consume the bottom half of the screen, overlaying the palette, which must also take up the bottom half of the screen and 80% of its width.

-The notebook would consume the top half of the screen.

• The Tobii I12+ requires a nominal distance of around 24 inches from the eye, making a 125-150% magnification factor required in order for my son to be able to most easily perceive and access the notebook.

Required use cases :

• Typesetting (as opposed to free form drawing on a paint app)

• Manual step-by-step equations, introductory algebra, functions

• Inequalities and graphing

• Plotting / visualization

• Basic geometry and area/perimeter calculations.

Required functionality:

• Core

  • evaluate current cell

  • Evaluate current workbook

  • New cell (wolfram lanaguage)

  • Keyboard - like in Class Assistant - As an alternative to my son using his eye tracking keyboard as this allows zoom to click.

  • Typesetting (from Basic Math Assistant, Typesetting, superscript tab) - ?? some duplication from calculator - where would be the best place for these ???

    • fraction
    • square root
    • superscript (ie for powers)
    • matching parenthesis
    • matching square brackets
    • matching braces
    • matching single vertical lines (for absolute value)
    • Add < and > symbols to allow inequalities to be typed.

  • Color controls

  • Navigation

    • Add tab and shift/tab for traversing function templates, these are missing in current template

  • Calculator from basic math assistant

    • basic tab - mostly the same as the built-in palette,

      • Create Input Cell (create Wolfram input cell)

      • remove "input from above and output from above"

      • add Previous output (%), two outputs ago (%%), %%%, etc. to allow relative access and progressively solve an equation step-by-step manually. Relative reference are important to ensure the workbook doesn't get messed up if "evaluate workbook" is selected .

      • Single equals (assignment)

      • double equals (to express equations) Equals[]

      • remove "Create text cell" and "make template".

      • add semicolon so multiple lines can be provided in a single input without expecting any output.

    • advanced tab

      • add create free-form input cell or alpha input cell (double-check work by use of Alpha step-by-step feature).

      • remove all integration, derivative, definite product and matrix typesetting, keep definite sum and definite product

  • Basic commands

    • math constants and functions tab

      • mathematical constants : remove "more"

      • numeric functions : remove all the "Wave" functions

      • trig : only sin/cos/tan

    • algebra commands / expressions and equations

• Basic equation handling

  • Add/subtract/multiply/divide both sides of an equation (AddSides, SubtractSides, MultiplySides, DivideSides)
  • simplify / expand / factor / reduce

• Visualization

  Plot, Plot3d, Graphics

• Geometry

  Point, Line, Circle, Rectangle, Triangle, Polygon, Parallelogram, Cube, Sphere, Cylinder, Cone

Ah I think you want him to write things that look like typeset math not code but here is the code-ier version which we could use for some of the subtopics even if some subtopics are using templates. Having simple code commands might be very useful for things like Simplify[] and Solve[]

This is good info Myles.

What is the size (in pixels) of a single on-screen keyboard button? If we stick a tiny template indicator inside of a button of the same size, will your son be able to see it? Currently the keyboard buttons have just a single easy to read letter in each button, will a template with thin lines inside of their work? Can you verify the smallest template that will be comfortable for him to read?

If we follow the keyboard button sizes, then we can fit a list of topics at the top, subtopics below that, a row of templates from the subtopic, and finally a row of commonly used buttons below that.

If the screen is 1024x768 and the tobii controls on the left are 15% of the width and we need the top half of the screen to display the math he is inputting, then we have about 870x380 pixels for the control area. I guess that means the buttons are around 65 pixels wide with 10 pixel padding on each side?

Attached is a notebook that you can modify to test various size buttons and layouts that will generate an image with controllable pixel size so you can open it on his computer to see if it look as you want and your son can read it.

Attachments:

test image layout.nb

Attached is a notebook that creates a skeleton interface that we can use to further the discussion. Myles can you post your screenshot of your son's computer so people can see the dimensional constraints (i.e. how many pixels by how many pixels do we have available)? It would be good if you could indicate an ideal button size in pixels as well, that will help size the buttons.

Let me know what you think of the tab+page view generated in the notebook, screenshot attached. We can of course change the sizes and padding around the buttons, but I just wanted to make sure it is generally in the right direction.

Attachments:

tobii.nb

Awesome, thanks Eric. I am thinking I will work to refine the design and code up a prototype and maybe you could take it from there to mature my implementation?

I just realized there is no private messaging on the community. you can email me at my mit.edu email kkeane

Thanks for the reply, Stephen. My son does not have the head motor control to consistently operate head-based switches or to use a head mouse, but he does not have spasticity (in fact, his muscles are limp like wet noodles except for a very little amount of slow left-to-right neck movement). In contrast, his eyes are very fast and he has trained on how to use his gaze-based eye tracking system for several years and has become quite good at it. I think a raw command-line interface would not be feasible for him, but rather a series of graphical elements that allow for higher-level control.

Hi, Ahmed. Thanks for the reply. My son wouldn't be able to use a standard keyboard effectively, I am looking for a simplified graphical user interface to be a bridge between him and Mathematica. I understand Mathematica has a UIkit, so I'm wondering whether anybody has used this to develop a simplified interface to Mathematica online (access via web browser). Currently, my son uses a 5x4 grid (in which the screen is divided into 20 regions, with each region having a specific function that he accesses via fixing his gaze on the region and holding his gaze there).

The typical way he controls an external application (such as a web browser) is to shrink his grid to occupy only a part of the screen and allow his eyes to "become the mouse" and move/click onto the web page. Keyboard access is more tricky, which is why having a completely mouse-driven input model would be a much better idea.

Some links that describe current standards for online accessibility : https://www.w3.org/TR/WCAG21/