Joel Shor

I'm a Google / Verily AI researcher working in AI for Social Good. I'm focused on healthcare, accessibility, and speech.

Email / Google Scholar / Patents

Low-tech intervention to reduce anxiety for patients diagnosed with chronic disease, published in JMIR
AI models for gastroenterology: real-time polyp detectors in colonoscopies, medically-aware video compression [1, 2], improved transcription tools for physicians [1, 2]. Publications in United European Gastroenterology Week, Digestive Disease Week, Gastrointestinal Endoscopy, NeurIPS, ACL Anthology
Improved long-read DNA sequencing with transformers, NeurIPS paper and used by PacBio
Started project Euphonia, which developed personalized speech recognition and voice imitation text-to-speech to let people with medically-impacted speech communicate with their original voice [paper at Interspeech 2019, Google AI blog post, other publications].
Started COVID-19 Forecasting in Japan project, which was seen by millions of people and by the national government to shape their COVID-19 policy [blog, npj Digital Medicine paper. Released one of Big Tech's first AI Fairness reports in Japan for our Covid forecasting model.
Creator of TensorFlow GAN, the Google standard for training and evaluating Generative Adversarial Networks. Created a Google Machine learning crash course on GANs, and an ML tech talk.
Created a series of state-of-the-art speech embeddings for understanding emotion and medical conditions. Used by Google and others for sleep monitoring and cough detection. Publications in Interspeech and ICASSP [papers 1 2 3 4, blogs 1 2]
On the team that created the first indistinguishable-from-human text-to-speech system, Tacotron. Published in ICML
Image and video compression using neural networks
Joined Google through acquisition of targeted video search startup

Some projects:

Research on Inflammatory Bowel Disease

Developing product to detect and classify polyps in realtime
Custom video compression for preserving medically relevant video content (NeurIPS 2022 workshop paper: Medical imaging meets NeurIPS)
Superresolution and video compression patents
Exploring metabolomic data as a disease predictor

DeepConsensus sequence transformers for PacBio Circular Consensus Sequencing data

Co-first author on NeurIPS workshop paper describing an on-device, error-correcting, long-read gene sequence called DeepConsensus
Pacific Biosciences (PacBio) announced a new instrument called Revio which DeepConsensus onto the instrument. The Revio machine can sequence 1,300 genomes a year (a 15x increase) for $1,000 per genome (4x cheaper) at the highest quality possible for genome analysis [keyword blog, video describing the instrument, marketing material]
Community reception is very positive: “DeepConsensus will run natively on the Revio instrument and will polish every read that comes off it. If you want a peek of the future, it's right here.” An industry expert highlighted DeepConsensus contributions, including enabling machine runtime to speed up from 30h to 24h.

Covid-19 Forecasting in Japan and the US

Launched a 28-day Covid19 public forecast for Japan. Used by top government policymakers to shape Japan’s Covid response. Seen by millions of people worldwide [paper, blog post]
Work won VLED Award for enhancing public good with Big Data, received “Feats of Engineering” award from Google Cloud, and I was recognized with a “Citizen Awards: Collaboration Category”
“Visiting Researcher” at Japan’s top-ranked research university, Keio University, in the school of Medicine, to investigate the reliability of large-scale surveys in helping model the spread of COVID-19 [announcement]

Non-text aspects of speech

Created many novel representations for paralinguistic speech [blog post, Interspeech papers 1, 2, code, models]
Uses include fake audio detection (patent), cough detection (patent), and disordered speech detection [paper], and tracking mental wellbeing (patents 2x). Many external academic and start-up collaborators
State-of-the-art paralinguistic representations, applied to better understanding emotion, user intent, and medically-impacted speakers [papers ex. 1, 2, 3, 4, 5], in ex. ICASSP, Interspeech, and IEEE Signal Processing [patents ex 1, 2, 3]
Work successfully ported to the open source Hugging Face (github)

Accessibility for speech

Co-creator and former tech lead of project Euphonia, which gives patients who lost their voice back the ability to speak with their original voice
Announced at Google I/O 2018 [blog post, paper, Google I/O video, documentary video] and mentioned in Google’s 2018 “Letter to the Founders”

Co-create of TensorFlow GAN, an open-source library for easily training and evaluating Generative Adversarial Networks

[github, blog posts 1, 2, patent, public lecture]
4M downloads; papers using TF-GAN have thousands of citations; 170K students took TF-GAN course in 2020

Invited talks

(2022-10) Google research conference
(2022-04) Tech talk at Boston University’s Math & Statistics department
(2022-04) Tech talk at Boston University’s Mathematical Association of America
(2022-04) Tech talk at MIT’s Department of Electrical Engineering & Computer Science course 6.345/HST.728
(2022-03) Tech Talk at Boston University (link)
(2022-02) Co-presenter at the 4th Conference of the Japan Society of Colon Examination in Hokkaido [poster]
(2022-02) National Institute of Informations (NII), Japan
(2021-12) MIT CSAIL’s Spoken Language Systems Group

AI Blog posts

Public courses

Welcome to Introduction to GANs [full course, lecture]

Education

Princeton BA in Mathematics
Manfred Memorial Manfred Pyka Memorial Physics Prize