Using Integrative Sensorimotor Capabilities

• Observations on Sensorimotor Integration of Eye-Head-Hand Coordination:
  • Developmental stages:
    • coordinated eye-head before hand-arm movements
      • VOR (vestibulo-ocular reflex, 3-neuron arc) is important
      • sampling environment, sensory integration initially more important?
    • early vision-guided hand-arm
      • mostly smooth pursuit eye movements help guide hand
      • requires attentional resources
    • arm reaching and hand grasping gradually integrated into smooth movement
      • hand: more sensorimotor cortex and direct pyramidal tract
      • arm: more coordination of posture and movement, gravity and intertial dynamics
    • general: gradual increase in use of cerebellum, basal ganglia, and extrapyramidal track
  • Brain structural integration
    • sensorimotor cortex: integrated, maps
    • cerebellum:
      • receives both "higher" and wealth of sensory info
      • systematic internal structure and computational cells
        • sensory #1: conventional (mossy): receives converging sensory info from everywhere, timing on parallel fibers
        • sensory #2: "reinforcement" learning signals via inferior olive
        • output: inhibition via pyramidal cells (sculpting of neuromotor signals)
      • with dysfunction: poor timing, less fluid coordination
  • Dynamic movement times, fastest-to-slowest: eye, hand, head, arm
  • Predictive capability: hand better than head, head better than eye
  • "Virtual" spatial mapping capability: conceptual eye-hand coordination
    • ex: hand pointer (e.g., mouse) to image (e.g., monitor screen)
  • Integration in Goal-Directed Performance, and Skill Development
    • "real-time" and "off-line" (learning) use of sensing
    • vision often seems to provide key part of reinforcement signal (e.g., to cerebellum)
      • What if the person has no vision?
    • simple movements: improvement mostly in first 4-8 trials
      • tuning neurocontrol signal (not really adaptive learning)
    • complex movements: skills may take years of practice
• "Extended Physiological Proprioception" (EPP):
  • capacity to encompass a technology as if it were an extension of the body
  • key requirement: bi-causal power transfer (force & velocity/position across interface)
  • examples:
    • body-powered upper extremity prosthesis
    • tennis racquet, golf club, baseball bat, pencil, brush
  • what if no hand?

Example: Efforts at Multimodal Accessibility Standards

• Definition of multimodal sensory interfaces
• Relation to accessibility and reduction of social exclusion
• W3C's Multimodal Interaction Activity (Standards/Guidelines) for Web
  • HTML/XML Web Accessibility Guidelines; Cascading Style Sheets (CSS); Xforms
  • Synchronous Multimedia Interaction Language (SMIL); Scalable Vector Graphics; VoiceXML
  • User Agent Accessibility Guidelines Working Group (usability of client browsers)
• Europe's ETSI EG 202 191: Human Factors: Multimodal Interaction, Communication and Navigation Guidelines
  • addresses multimodality to reduct social exclusion, improve accessibility, and human factors recommendations for multimodal interfaces
• V2 Standard for Universal Remote Consoles for Target Devices and Services
  • With possible implementation with UPnP (Universal Plug and Play)
  • Example: V2 simulation environment from Trace Center

Example: Goal-Directed Movements in Vitual Environments as Therapy

• Conceptual Foundations and Design of Virtual Environments
  • Components of a Virtual Reality System
  • Concept of Immersion
  • Concept of Interactivity
• Examples of Virtual Environments in Rehabilitation
  • Aims (and possible advantages):
    • centered on abilities
    • desire of overcoming task/subtask hurdles
    • can perform within "safe" environment
    • motivation may be higher ("fun" therapy)
    • integrated assessment, possibly more targeted
  • Concerns/disadvantages:
    • rehab ultimately directed to function in the real world
    • sensory conflict (e.g., VOR & smooth pursuit, eye & hand)
    • motor tasks designed around interface (e.g., hand)
    • limited access to technologies
  • Examples

| overview | multimodal | virtual reality/terms | virtual environments/rehab