A Unified, Scalable Framework for Neural Population Decoding
In preparation
Half-Hop: A graph upsampling approach for slowing down message passing
Accepted at ICML 2023
Abstract: Message passing neural networks have shown a lot of success on graph-structured data.
However, there are many instances where message passing can lead to over-smoothing or fail when neighboring nodes belong to different classes.
In this work, we introduce a simple yet general framework for improving learning in message passing neural networks.
Our approach essentially upsamples edges in the original graph by adding ``slow nodes'' at each
edge that can mediate communication between a source and a target node. Our method only modifies the input graph,
making it plug-and-play and easy to use with existing models. To understand the benefits of slowing down message passing,
we provide theoretical and empirical analyses. We report results on several supervised and self-supervised benchmarks,
and show improvements across the board, notably in heterophilic conditions where adjacent nodes may have different
labels. We also show how our method can be used to generate multi-scale views for graph self-supervised learning.
Transcriptomic cell type structures in vivo neuronal activity across multiple time scales
+Contributed equally as co-first authors
Cell Reports, Volume 42, Issue 4, April 2023.
Abstract: Cell type is hypothesized to be a key determinant of the role of a neuron within a
circuit. However, it is unknown whether a neuron’s transcriptomic type influences the timing of its
activity in the intact brain. In other words, can transcriptomic cell type be extracted from the
time series of a neuron’s activity? To address this question, we developed a new deep learning
architecture that learns features of interevent intervals across multiple timescales (milliseconds
to >30 min). We show that transcriptomic cell class information is robustly embedded in the timing
of single neuron activity recorded in the intact brain of behaving animals (calcium imaging and
extracellular electrophysiology), as well as in a bio-realistic model of visual cortex. In contrast,
we were unable to reliably extract cell identity from summary measures of rate, variance, and
interevent interval statistics. We applied our analyses to the question of whether transcriptomic
subtypes of excitatory neurons represent functionally distinct classes. In the calcium imaging
dataset, which contains a diverse set of excitatory Cre lines, we found that a subset of excitatory
cell types are computationally distinguishable based upon their Cre lines, and that excitatory types
can be classified with higher accuracy when considering their cortical layer and projection class.
Here we address the fundamental question of whether a neuron, within a complex cortical network,
embeds a fingerprint of its transcriptomic identity into its activity. Our results reveal robust
computational fingerprints for transcriptomic types and classes across diverse contexts, defined
over multiple timescales.
Cite this paper
@article{SCHNEIDER2023112318,
title = {Transcriptomic cell type structures in vivo neuronal activity across multiple timescales},
journal = {Cell Reports},
volume = {42},
number = {4},
pages = {112318},
year = {2023},
issn = {2211-1247},
doi = {https://doi.org/10.1016/j.celrep.2023.112318},
url = {https://www.sciencedirect.com/science/article/pii/S2211124723003297},
author = {Aidan Schneider and Mehdi Azabou and Louis McDougall-Vigier and David F. Parks and Sahara Ensley and Kiran Bhaskaran-Nair and Tomasz Nowakowski and Eva L. Dyer and Keith B. Hengen},
}
Schneider, A., Azabou, M., McDougall-Vigier, L., Parks, D. B., Ensley, S., Bhaskaran-Nair, K., Nowakowski, T., Dyer, E. L. & Hengen, K. B. (2023). Transcriptomic cell type structures in vivo neuronal activity across multiple time scales. Cell Reports, Volume 42, Issue 4, 2023
Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis
Preprint, Mar 2023
Abstract: Natural behavior consists of dynamics that are complex
and unpredictable, especially when trying to predict many
steps into the future. While some success has been found
in building representations of behavior under constrained
or simplified task-based conditions, many of these models
cannot be applied to free and naturalistic settings where
behavior becomes increasingly hard to model. In this work,
we develop a multi-task representation learning model for
behavior that combines two novel components: (i) An action
prediction objectiåve that aims to predict the distribution of
actions over future timesteps, and (ii) A multi-scale architecture that builds separate latent spaces to accommodate
short- and long-term dynamics. After demonstrating the ability of the method to build representations of both local and
global dynamics in realistic robots in varying environments
and terrains, we apply our method to the MABe 2022 Multi-agent behavior challenge, where our model ranks 1st overall
and on all global tasks, and 1st on 4 out of 9 frame-level
subtasks. In all of these cases, we show that our model can
build representations that capture the many different factors
that drive behavior and solve a wide range of downstream
tasks.
Cite this paper
@misc{azabou2023relax,
doi = {10.48550/ARXIV.2303.08811},
url = {https://arxiv.org/abs/2303.08811},
author = {Azabou, Mehdi and Mendelson, Michael and Ahad, Nauman and Sorokin, Maks and Thakoor, Shantanu and Urzay, Carolina and Dyer, Eva L.},
title = {Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis},
publisher = {arXiv},
year = {2023}
}Mendelson, M., Azabou, M., Jacob, S., Grissom, N., Darrow, D., Ebitz, B., Herman, A. & Dyer, E. L. (2023). Learning signatures of decision making from many individuals playing the same game. arXiv.
Learning signatures of decision making from many individuals playing the same game
+Contributed equally as co-first authors
2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), Baltimore, Maryland, April 2023
Abstract: Human behavior is incredibly complex and the factors that drive decision making—from
instinct, to strategy, to biases between individuals—often vary over multiple timescales.
In this paper, we design a predictive framework that learns representations to encode an individual's
`behavioral style', i.e. long-term behavioral trends, while simultaneously predicting future actions
and choices. The model explicitly separates representations into three latent spaces: the recent
past space, the short-term space, and the long-term space where we hope to capture individual
differences. To simultaneously extract both global and local variables from complex human behavior,
our method combines a multi-scale temporal convolutional network with latent prediction tasks,
where we encourage embeddings across the entire sequence, as well as subsets of the sequence,
to be mapped to similar points in the latent space.
We develop and apply our method to a large-scale behavioral dataset from 1,000 humans playing a
3-armed bandit task, and analyze what our model's resulting embeddings reveal about the human
decision making process. In addition to predicting future choices, we show that our model can learn
rich representations of human behavior over multiple timescales and provide signatures of
differences in individuals.
Cite this paper
@INPROCEEDINGS{10123846,
author={Mendelson, Michael J. and Azabou, Mehdi and Jacob, Suma and Grissom, Nicola and Darrow, David and Ebitz, Becket and Herman, Alexander and Dyer, Eva L.},
booktitle={2023 11th International IEEE/EMBS Conference on Neural Engineering (NER)},
title={Learning signatures of decision making from many individuals playing the same game},
year={2023},
volume={},
number={},
pages={1-5},
doi={10.1109/NER52421.2023.10123846}
}Mendelson, M., Azabou, M., Jacob, S., Grissom, N., Darrow, D., Ebitz, B., Herman, A. & Dyer, E. L. (2023). Learning signatures of decision making from many individuals playing the same game. 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), Baltimore, MD, USA, 2023, pp. 1-5, doi: 10.1109/NER52421.2023.10123846.
Detecting change points in neural population activity with contrastive metric learning
2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), Baltimore, Maryland, April 2023
Cite this paper
@INPROCEEDINGS{10123821,
author={Urzay, Carolina and Ahad, Nauman and Azabou, Mehdi and Schneider, Aidan and Atamkuri, Geethika and Hengen, Keith B. and Dyer, Eva L.},
booktitle={2023 11th International IEEE/EMBS Conference on Neural Engineering (NER)},
title={Detecting change points in neural population activity with contrastive metric learning},
year={2023},
volume={},
number={},
pages={1-4},
doi={10.1109/NER52421.2023.10123821}
}Urzay, C., Ahad, N., Azabou, M., Schneider, A., Atmakuri, G., Hengen, K. B., & Dyer, E. L., Detecting change points in neural population activity with contrastive metric learning, 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), Baltimore, MD, USA, 2023, pp. 1-4, doi: 10.1109/NER52421.2023.10123821.2022
Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers
Neural Information Processing Systems (NeurIPS), 2022
Abstract: Complex time-varying systems are often studied by abstracting away from the
dynamics of individual components to build a model of the population-level dynamics from the start.
However, when building a population-level description, it can be easy to lose sight of each
individual and how each contributes to the larger picture. In this paper, we present a novel
transformer architecture for learning from time-varying data that builds descriptions of both the
individual as well as the collective population dynamics. Rather than combining all of our data into
our model at the onset, we develop a separable architecture that operates on individual time-series
first before passing them forward; this induces a permutation-invariance property and can be used to
transfer across systems of different size and order. After demonstrating that our model can be
applied to successfully recover complex interactions and dynamics in many-body systems, we apply our
approach to populations of neurons in the nervous system. On neural activity datasets, we show that
our multi-scale transformer not only yields robust decoding performance, but also provides
impressive performance in transfer. Our results show that it is possible to learn from neurons in
one animal's brain and transfer the model on neurons in a different animal's brain, with
interpretable neuron correspondence across sets and animals. This finding opens up a new path to
decode from and represent large collections of neurons.
Cite this paper
@misc{https://doi.org/10.48550/arxiv.2206.06131,
doi = {10.48550/ARXIV.2206.06131},
url = {https://arxiv.org/abs/2206.06131},
author = {Liu, Ran and Azabou, Mehdi and Dabagia, Max and Xiao, Jingyun and Dyer, Eva L.},
keywords = {Neurons and Cognition (q-bio.NC), Machine Learning (cs.LG), FOS: Biological sciences, FOS: Biological sciences, FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}Liu, R., Azabou, M., Dabagia, M., Xiao, J., & Dyer, E. L. (2022). Seeing the forest and the tree: Building representations of both individual and collective dynamics with transformers. arXiv preprint arXiv:2206.06131.
Abstract: Self-supervised learning provides a promising path towards eliminating the need for
costly label information in representation learning on graphs. However, to achieve state-of-the-art
performance, methods often need large numbers of negative examples and rely on complex
augmentations. This can be prohibitively expensive, especially for large graphs. To address these
challenges, we introduce Bootstrapped Graph Latents (BGRL) - a graph representation learning method
that learns by predicting alternative augmentations of the input. BGRL uses only simple
augmentations and alleviates the need for contrasting with negative examples, and is thus scalable
by design. BGRL outperforms or matches prior methods on several established benchmarks, while
achieving a 2-10x reduction in memory costs. Furthermore, we show that BGRL can be scaled up to
extremely large graphs with hundreds of millions of nodes in the semi-supervised regime - achieving
state-of-the-art performance and improving over supervised baselines where representations are
shaped only through label information. In particular, our solution centered on BGRL constituted one
of the winning entries to the Open Graph Benchmark - Large Scale Challenge at KDD Cup 2021, on a
graph orders of magnitudes larger than all previously available benchmarks, thus demonstrating the
scalability and effectiveness of our approach.
Cite this paper
@inproceedings{
thakoor2022largescale,
title={Large-Scale Representation Learning on Graphs via Bootstrapping},
author={Shantanu Thakoor and Corentin Tallec and Mohammad Gheshlaghi Azar and Mehdi Azabou and Eva L Dyer and Remi Munos and Petar Veli{\v{c}}kovi{\'c} and Michal Valko},
booktitle={International Conference on Learning Representations},
year={2022},
url={https://openreview.net/forum?id=0UXT6PpRpW}
}Thakoor, S., Tallec, C., Azar, M. G., Azabou, M., Dyer, E. L., Munos, R., Veli{\v{c}}kovi{\'c}, P., & Valko, M. (2021). Large-scale representation learning on graphs via bootstrapping. International Conference on Learning Representations.
MTNeuro: A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstraction
Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2022
Abstract: There are multiple scales of abstraction from which we can describe the same image,
depending on whether we are focusing on fine-grained details or a more global attribute of the
image. In brain mapping, learning to automatically parse images to build representations of both
small-scale features (e.g., the presence of cells or blood vessels) and global properties of an
image (e.g., source brain region) is a crucial and open challenge. However, most existing datasets
and benchmarks for neuroanatomy consider only a single downstream task at a time. We introduce a new
dataset, annotations, and multiple downstream tasks that provide diverse ways to readout information
about brain structure and architecture from the same image. Our multi-task neuroimaging benchmark
(MTNeuro) is built on volumetric, micrometer-resolution X-ray microtomography imaging of a large
thalamocortical section of mouse brain, encompassing multiple cortical and subcortical regions, that
reveals dense reconstructions of the underlying microstructure (i.e., cell bodies, vasculature, and
axons). We generated a number of different prediction challenges and evaluated several supervised
and self-supervised models for brain-region prediction and pixel-level semantic segmentation of
microstructures. Our experiments not only highlight the rich heterogeneity of this dataset, but also
provide insights into how self-supervised approaches can be used to learn representations that
capture multiple attributes of a single image and perform well on a variety of downstream tasks.
Datasets, code, and pre-trained baseline models are provided at: https://mtneuro.github.io/.
Cite this paper
@article{quesadamtneuro,
title={MTNeuro: A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstraction},
author={Quesada, Jorge and Sathidevi, Lakshmi and Liu, Ran and Ahad, Nauman and Jackson, Joy M and Azabou, Mehdi and Xiao, Jingyun and Liding, Chris and Urzay, Carolina and Gray-Roncal, William and Johnson, Erik Christopher, Dyer, Eva L.}
}Quesada, J., Sathidevi, L., Liu, R., Ahad, N., Jackson, J. M., Azabou, M., Xiao, J. and Liding, C. and Urzay, C. and Gray-Roncal, W. and Johnson, E. C. & Dyer, E. L. MTNeuro: A Benchmark for Evaluating Representations of Brain Structure Across Multiple Levels of Abstraction.
Cite this paper
@article{azabou2022learning,
title={Learning Behavior Representations Through Multi-Timescale Bootstrapping},
author={Azabou, Mehdi and Mendelson, Michael and Sorokin, Maks and Thakoor, Shantanu and Ahad, Nauman and Urzay, Carolina and Dyer, Eva L},
journal={arXiv preprint arXiv:2206.07041},
year={2022}
}Azabou, M., Mendelson, M., Sorokin, M., Thakoor, S., Ahad, N., Urzay, C., & Dyer, E. L. (2022). Learning Behavior Representations Through Multi-Timescale Bootstrapping. arXiv preprint arXiv:2206.07041.2021
Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity
Neural Information Processing Systems (NeurIPS), accepted for Oral (1% submissions), 2021
Abstract: Meaningful and simplified representations of neural activity can yield insights
into how and what information is being processed within a neural circuit. However, without labels,
finding representations that reveal the link between the brain and behavior can be challenging.
Here, we introduce a novel unsupervised approach for learning disentangled representations of neural
activity called SwapVAE. Our approach combines a generative modeling framework with an
instance-specific alignment loss that tries to maximize the representational similarity between
transformed views of the input (brain state). These transformed (or augmented) views are created by
dropping out neurons and jittering samples in time, which intuitively should lead the network to a
representation that maintains both temporal consistency and invariance to the specific neurons used
to represent the neural state. Through evaluations on both synthetic data and neural recordings from
hundreds of neurons in different primate brains, we show that it is possible to build
representations that disentangle neural datasets along relevant latent dimensions linked to
behavior.
Cite this paper
@inproceedings{NEURIPS2021_58182b82,
author = {Liu, Ran and Azabou, Mehdi and Dabagia, Max and Lin, Chi-Heng and Gheshlaghi Azar, Mohammad and Hengen, Keith and Valko, Michal and Dyer, Eva},
booktitle = {Advances in Neural Information Processing Systems},
editor = {M. Ranzato and A. Beygelzimer and Y. Dauphin and P.S. Liang and J. Wortman Vaughan},
pages = {10587--10599},
publisher = {Curran Associates, Inc.},
title = {Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity},
url = {https://proceedings.neurips.cc/paper/2021/file/58182b82110146887c02dbd78719e3d5-Paper.pdf},
volume = {34},
year = {2021}
}Liu, R., Azabou, M., Dabagia, M., Lin, C. H., Gheshlaghi Azar, M., Hengen, K., Valko, M. & Dyer, E. (2021). Drop, swap, and generate: A self-supervised approach for generating neural activity. Advances in Neural Information Processing Systems, 34, 10587-10599.
Abstract: State-of-the-art methods for self-supervised learning (SSL) build representations
by maximizing the similarity between different transformed "views" of a sample. Without sufficient
diversity in the transformations used to create views, however, it can be difficult to overcome
nuisance variables in the data and build rich representations. This motivates the use of the dataset
itself to find similar, yet distinct, samples to serve as views for one another. In this paper, we
introduce Mine Your Own vieW (MYOW), a new approach for self-supervised learning that looks within
the dataset to define diverse targets for prediction. The idea behind our approach is to actively
mine views, finding samples that are neighbors in the representation space of the network, and then
predict, from one sample's latent representation, the representation of a nearby sample. After
showing the promise of MYOW on benchmarks used in computer vision, we highlight the power of this
idea in a novel application in neuroscience where SSL has yet to be applied. When tested on
multi-unit neural recordings, we find that MYOW outperforms other self-supervised approaches in all
examples (in some cases by more than 10%), and often surpasses the supervised baseline. With MYOW,
we show that it is possible to harness the diversity of the data to build rich views and leverage
self-supervision in new domains where augmentations are limited or unknown.
Cite this paper
@article{azabou2021mine,
title={Mine your own view: Self-supervised learning through across-sample prediction},
author={Azabou, Mehdi and Azar, Mohammad Gheshlaghi and Liu, Ran and Lin, Chi-Heng and Johnson, Erik C and Bhaskaran-Nair, Kiran and Dabagia, Max and Avila-Pires, Bernardo and Kitchell, Lindsey and Hengen, Keith B and Gray-Roncal, William and Valko, Michal and Dyer, Eva L},
journal={arXiv preprint arXiv:2102.10106},
year={2021}
}Azabou, M., Azar, M. G., Liu, R., Lin, C. H., Johnson, E. C., Bhaskaran-Nair, K., Dabagia, M., Avila-Pires, B., Kitchell, L., Hengen, K. B., Gray-Roncal, W., Valko, M. & Dyer, E. L. (2021). Mine your own view: Self-supervised learning through across-sample prediction. arXiv preprint arXiv:2102.10106.
Cite this paper
@article{azabouusing,
title={Using self-supervision and augmentations to build insights into neural coding},
author={Azabou, Mehdi and Dabagia, Max and Liu, Ran and Lin, Chi-Heng and Hengen, Keith B and Dyer, Eva L},
journal={NeurIPS 2021 Workshop on Self-supervised Learning: Theory and Practice},
year = {2021}
}Azabou, M., Dabagia, M., Liu, R., Lin, C. H., Hengen, K. B. & Dyer, E. L. Using self-supervision and augmentations to build insights into neural coding.
Making transport more robust and interpretable by moving data through a small number of anchor points
International Conference on Machine Learning (ICML), 2021
Abstract: Optimal transport (OT) is a widely used technique for distribution alignment, with
applications throughout the machine learning, graphics, and vision communities. Without any
additional structural assumptions on transport, however, OT can be fragile to outliers or noise,
especially in high dimensions. Here, we introduce Latent OptimalTransport (LOT), a new approach for
OT that simultaneously learns low-dimensional structure in data while leveraging this structure to
solve the alignment task. The idea behind our approach is to learn two sets of “anchors” that
constrain the flow of transport between a source and target distribution. In both theoretical and
empirical studies, we show thatLOTregularizes the rank of transport and makes it more robust to
outliers and the sampling density. We show that by allowing the source and target to have different
anchors, and using LOT to align the latent spaces between anchors, the resulting transport plan has
better structural interpretability and highlights connections between both the individual data
points and the local geometry of the datasets.
Cite this paper
@InProceedings{lin2021,
title = {Making transport more robust and interpretable by moving data through a small number of anchor points},
author = {Lin, Chi-Heng and Azabou, Mehdi and Dyer, Eva},
booktitle = {Proceedings of the 38th International Conference on Machine Learning},
pages = {6631--6641},
year = {2021},
editor = {Meila, Marina and Zhang, Tong},
volume = {139},
series = {Proceedings of Machine Learning Research},
month = {18--24 Jul},
publisher = {PMLR},
}Lin, C., Azabou, M. & Dyer, E.. (2021). Making transport more robust and interpretable by moving data through a small number of anchor points. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:6631-6641 Available from https://proceedings.mlr.press/v139/lin21a.html.Abstracts
Detecting change points in neural population activity with contrastive metric learning
Conference on Cognitive Computational Neuroscience (CCN), 2022
Cite this paper
@misc{Urzay_Ahad_Azabou_Schneider_Atmakuri_Hengen_Dyer_2022,
title={Detecting change points in neural population activity with contrastive metric learning},
url={http://dx.doi.org/10.32470/CCN.2022.1261-0},
DOI={10.32470/ccn.2022.1261-0},
journal={2022 Conference on Cognitive Computational Neuroscience},
publisher={Cognitive Computational Neuroscience},
author={Urzay, Carolina and Ahad, Nauman and Azabou, Mehdi and Schneider, Aidan and Atmakuri, Geethika and Hengen, Keith B. and Dyer, Eva L.},
year={2022}
}Urzay, C., Ahad, N., Azabou, M., Schneider, A., Atmakuri, G., Hengen, K. B., & Dyer, E. L. (2022). Detecting change points in neural population activity with contrastive metric learning. In 2022 Conference on Cognitive Computational Neuroscience. https://doi.org/10.32470/ccn.2022.1261-0