Multi-modal STEM Accessibility Based on Rich Semantics

---

Workshop Laboratorio S. Polin

Motivation

• Not needed ...
• See previous two talks
• I will be talking about Math, Statistics, Chemistry, accessibility

Aims

• Making STEM content accessible on the web
• The technologies and components are already there
• The web is syntax, the semantic has to come from elsewhere
  • Note: LaTeX, MathML, CML, MOL, etc. are all only syntax!
• Web technology can embed and hide the semantic
• Suitable semantically rich structure allows for
  • Accessibility
  • Support of users with diverse needs
  • Generation of alternative formats: tactile, audio-tactile, sub-titles etc.

Aims (2)

• Source independence
  • Use the sources authors produce
• Tool independence
  • Do not require a particular system to author or read in
• Platform independence
  • Work with all operating systems, browsers, AT
• Note the sign of the times
  • Covid-19: Use the current opportunity, avoid its threats

Sourcing Semantics

• Authors?
  • Never rely on authors doing their part
  • Even if they are willing they often get it wrong
• Remedial Officers?
  • Often lack the expert knowledge
  • ALT text is usually not enough!
  • Too muchcontent to catch up
• Automatic
  • Exploit available sources
  • Use intelligent methods to automatically interpret syntax
  • Avoid manual intervention as much as possible

Maths Formulas

• How to generate accessible Maths?
• Be careful when you convert your sources

Convert from Sources

Open source tools for converting maths documents from diverse sources

• Translate a LaTeX or Markdown document with pandoc

pandoc --standalone -t html -o example.html example.md

• Or with LaTeXML

latexml example.tex | latexmlpost --dest=example.html

• Or from Word sources

pandoc --self-contained -s example.docx -t html -o example.html

Embed Semantics

• Enrich Math using Speech Rule Engine
• Javascript library for translating XML into Speech (markup)
• Open source library available at github
• Only Math speech solution in JavaScript
• Runs in browser, command line, as node module.
• Initially implemented in the context of ChromeVox
• Speech solution for: ChromeVox, MathJax, EquatIO, MathLive, ...

Generate Semantics

• Math markup (e.g., LaTeX, MathML) are not very expressive
• SRE rewrites internally into semantically improved term structure

    ax^2+bx+c=0

<math>
  <mi>a</mi>
  <msup>
    <mi>x</mi>
    <mn>2</mn>
  </msup>
  <mo>+</mo>
  <mi>b</mi>
  <mi>x</mi>
  <mo>+</mo>
  <mi>c</mi>
  <mo>=</mo>
  <mn>0</mn>
</math>

Render Accessibly

• For example use MathJax
• A JavaScript library for rendering Mathematics in all browsers
• Input: LaTeX, AsciiMath, and MathML
• Output: HTML/CSS, SVG
• De facto rendering solution of (nearly) all Mathematics on the web
• Uses SRE for accessibility

Example

$$ x = \frac{-b \pm \sqrt{b^2-4ac}}{2a} $$

• Source in LaTeX
• Rendered with MathJax
• Embedded semantics generated with SRE
• Accessibility "built-in"

Multi-modality

• Speech or Braille output (e.g., Nemeth)
• Linear nemeth output $$ \vec{\nabla} \times \vec{F} = \left( \frac{\partial F_z}{\partial y} - \frac{\partial F_y}{\partial z} \right) \mathbf{i} + \left( \frac{\partial F_x}{\partial z} - \frac{\partial F_z}{\partial x} \right) \mathbf{j} + \left( \frac{\partial F_y}{\partial x} - \frac{\partial F_x}{\partial y} \right) \mathbf{k} $$

LaTeX to Nemeth

• Generating tactile Maths in Braille from PreTeXt/LaTeX Input.
• Ongoing project with NFB, PreTeXt and AIM.
• PreTeXt books are translated automatically into Braille with
  • Liblouis for the text
  • MathJax for formulas using SRE's Nemeth translation
• Multistep translation pipeline server-side using MathJax3
• Currently working on 2D layout structures and diagrams

Do NOT Rely on Single Standards

• Try to express this in MathML: $$ \begin{prooftree} \AxiomC{} \RightLabel{$Hyp^{1}$} \UnaryInfC{$P$} \AXC{$P\to Q$} \RL{$\to_E$} \BIC{$Q^2$} \AXC{$Q\to R$} \RL{$\to_E$} \BIC{$R$} \AXC{$Q$} \RL{Rit$^2$} \UIC{$Q$} \RL{$\wedge_I$} \BIC{$Q\wedge R$} \RL{$\to_I^1$} \UIC{$P\to Q\wedge R$} \end{prooftree} $$

... or Single Sources

• Use PDF as a source
• Ongoing project with the AssisTech Group IIT Delhi
• Resurrects an old project of mine: MaxTract (the good parts)
• Translate inaccessible PDF into accessible ePub
• We use pdf2htmlEX
• Builds a client side accessible rendering for PDF documents
• Runs server-side with puppeteer
• Can handle Maths, Tables, some diagrams

What about Diagrams?

• Semantic representations?
• SVG is an ideal container format:
  • Highly structured
  • Embedding of hierarchical content
  • Foreign objects embedded via namespaces
  • Muliple views on content
• Support of diverse audiences
• Multi-modal output
• Sources can be images, unique identifiers, models, etc.

Generate with Knowledge

• Easiest way to embed knowledge is, if I know what I am talking about
• Data visualisations directly generated from statistical models (e.g., in R)
• Authors do not even realise that it is accessible out of the box

Get Semantics from Scratch

• Image analysis and recognition
• Alternatively: Unique Identifiers
• Generation of annotated SVG
• Semantic enrichment
  • Recursively identify major components
  • Use web services for naming
• Generate hierarchical navigation structure

Chemistry Example

• This also works as a tactile or audio-tactile graphic
• More examples here
• Also for Physics

Test Live

Live web service

Combined in STEM Documents

• All this technology plays well together in single web documents
• Example: Take these slides
• Or my Report for the DIAGRAM Center.

Conclusions

• Basic mechanisms for accessible STEM on the web are available
• We just have to use them right
• Let's not be too hung-up on standards
• Always avoid limiting the creativity of authors
• Covid-19:
  • Organisations are forced to move content online
  • If done badly, it will be a lost opportunity!
  • E.g., Training sessions with the AMS