Research
The four themes below overlap a lot. Robotic arms and hands have been part of nearly all of Joe's experimental work, and in most of our experiments one of them is the device the interface controls.
Somatosensory neuroprosthetics via microstimulation
The goal is a prosthetic limb its user can feel. To get there we are building somatosensory neuroprostheses meant to deliver touch and proprioception (the sense of where your limb is) through microstimulation of peripheral nerves, the ventral posterolateral (VPL) nucleus of the thalamus and primary somatosensory cortex (S1). We treat the stimulation as a control problem. We stimulate upstream, in VPL for example, and record what happens downstream in S1, then keep adjusting the stimulation until the cortical response matches what a real touch produces.
Earlier we mapped proprioceptive and cutaneous representations in VPL and modeled the cortical field potentials that thalamic stimulation produces. We showed that optimized microstimulation can elicit naturalistic cortical responses. In macaques, stimulating VPL gives S1 responses similar to those from natural touch. Now we're pairing a real-time electrophysiology system with deep networks so stimulation parameters can be updated online. Our current NIH R01 is developing a regenerative micro-electrode peripheral nerve interface aimed specifically at proprioceptive and cutaneous fibers.
Reward, motivation and autonomous brain-machine interfaces
We found that reward expectation modulates single units, local field potentials and spike-field coherence in primary motor cortex (M1). Sensorimotor cortex, in other words, does more than encode movement. Reward value can also be decoded from S1, and mirror neurons in the sensorimotor cortices respond to grip force and to reward expectation too. Activity in dorsal premotor cortex (PMd), M1 and S1 is normalized by valence (whether the outcome will be good or bad) and by motivational intensity, and M1 activity shows hallmarks of a temporal difference reinforcement learning process.
Decoders that take affect, meaning reward and motivation, into account perform better, which is useful by itself. The bigger idea for us is that the reward signal can act as a critic, the part of a reinforcement learning system that judges how good each action was, so a brain-machine interface could learn what its user wants and update itself without an external teacher. We have built reinforcement learning interfaces on that idea. Right now we are looking at how reward and effort shape phase-amplitude coupling and Granger causality within and between sensorimotor areas before movement.
Machine learning for neural decoding and control
We develop the computational tools that closed-loop interfaces need. Lately that means deep learning. Convolutional networks classify touch location from S1 local field potentials, and grip force prediction got better when we trained with a loss function that penalizes reward-related neural information. We also use CNN-RNN models to drive a somatosensory neuroprosthesis.
Before that, working with Jose Principe, William Lytton and other collaborators, we helped develop kernel-based methods for neural decoding and for spike train metrics. Kernel temporal difference reinforcement learning came out of that work too, as did adaptive inverse controllers that pick microstimulation patterns online. We also worked on large-scale spiking models of sensorimotor cortex that learn to reach by reinforcement and drive virtual and robotic arms. From 2010 to 2015 Joe was a co-PI of the DARPA REPAIR program and PI of its SUNY Downstate team. The program combined computational and biological tools to repair and replace networks of neurons in the brain.
Noninvasive brain-computer interfaces and human-robot interaction
From 2016 to 2021 we ran an NSF National Robotics Initiative project with Columbia University that combined EEG, functional near-infrared spectroscopy (fNIRS), eye tracking and pupil size to pick up reward and expectation signals a robot could learn from. The questions we ask with implanted electrodes, reward in particular, carry over to recordings made from outside the head, and those don't need surgery.
Before the NSF project we showed that frontal lobe hemodynamic signals can serve as reinforcement for an adaptive controller. With collaborators we've also classified Alzheimer's disease from combined EEG and fNIRS, and used connectivity analysis to improve an auditory brain-computer interface. Other projects tracked seizure state in real time from multichannel EEG, and emotional valence from facial electromyography.
Equipment
Most experiments in the lab run on the equipment below. The recording and stimulation systems and the Kinarm robots are used in our nonhuman primate studies. The robotic arms are what our brain-machine interfaces control, and the fNIRS and EEG systems are used together for noninvasive work with human participants.
Product photos are from the manufacturers. Sawyer photo by Jeff Green/Rethink Robotics, CC BY 4.0.
Blackrock Neurotech Cerebus
Four systems, 512 channels in all
Plexon MAP
Three systems
Intan RHS
128 channels, recording and stimulation
NIRx fNIRS
Run together with EEG
Brain Products EEG
Run together with fNIRS
Kinarm Exoskeleton
Two robots, for NHP studies
Kinarm End-Point
One robot, for NHP studies
Barrett WAM
Arm with wrist and BarrettHand
Universal Robots UR3
Robotic arm
Rethink Robotics Sawyer
Robotic arm
Funding
This work has been funded by the National Institutes of Health (NINDS), the National Science Foundation, DARPA, the New York State Spinal Cord Injury Research Board, the Christopher and Dana Reeve Foundation and the National Academies Keck Futures Initiative.
Francis Lab
Department of Biomedical Engineering
Cullen College of Engineering
University of Houston, Houston, Texas
Contact
jtfrancis@uh.edu
713-743-0985