Filter:

Multi-session, multi-task neural decoding from distinct cell-types and brain regions

Neuroscience

Transformers

Mehdi Azabou⁺, Krystal Xuejing Pan⁺, Vinam Arora, Ian Knight, Eva L. Dyer^*, Blake Richards^*

⁺Contributed equally as co-first authors

International Conference on Learning Representations (ICLR), accepted as Spotlight, 2025

Abstract: Recent work has shown that scale is important for improved brain decoding, with more data leading to greater decoding accuracy. However, large-scale decoding across many different datasets is challenging because neural circuits are heterogeneous, each brain region contains a unique mix of cellular sub-types, and the responses to different stimuli are diverse across regions and sub-types. It is unknown whether it is possible to pre-train and transfer brain decoding models between distinct tasks, cellular sub-types, and brain regions. To address these questions, we developed a multi-task transformer architecture and trained it on the entirety of the Allen Institute's Brain Observatory dataset. This dataset contains responses from over 100,000 neurons in 6 areas of the brains of mice, observed with two-photon calcium imaging, recorded while the mice observed different types of visual stimuli. Our results demonstrate that transfer is indeed possible -combining data from different sources is beneficial for a number of downstream decoding tasks. As well, we can transfer the model between regions and sub-types, demonstrating that there is in fact common information in diverse circuits that can be extracted by an appropriately designed model. Interestingly, we found that the model's latent representations showed clear distinctions between different brain regions and cellular sub-types, even though it was never given any information about these distinctions. Altogether, our work demonstrates that training a large-scale neural decoding model on diverse data is possible, and this provides a means of studying the differences and similarities between heterogeneous neural circuits.

Project page Paper OpenReview Poster

Know Thyself by Knowing Others: Learning Neuron Identity from Population Context

Neuroscience

SSL

Vinam Arora, Divyansha Lachi, Ian Jarratt Knight, Mehdi Azabou, Blake Aaron Richards, Cole Lincoln Hurwitz, Josh Siegle, Eva L Dyer

Neural Information Processing Systems (NeurIPS), 2025

OpenReview

BibTex

@misc{ryoo2025generalizable,
    title={Generalizable, real-time neural decoding with hybrid state-space models}, 
    author={Avery Hee-Woon Ryoo and Nanda H. Krishna and Ximeng Mao and Mehdi Azabou and Eva L. Dyer and Matthew G. Perich and Guillaume Lajoie},
    year={2025},
    eprint={2506.05320},
    archivePrefix={arXiv},
    primaryClass={q-bio.NC},
    url={https://arxiv.org/abs/2506.05320}, 
}

APA

Ryoo, A. H.-W., Krishna, N. H., Mao, X., Azabou, M., Dyer, E. L., Perich, M. G., & Lajoie, G. (2025). Generalizable, real-time neural decoding with hybrid state-space models. arXiv 2506.05320

Generalizable, real-time neural decoding with hybrid state-space models

Neuroscience

Time Series

Avery Hee-Woon Ryoo, Nanda H. Krishna, Ximeng Mao, Mehdi Azabou, Eva L. Dyer, Matthew G. Perich, Guillaume Lajoie

Neural Information Processing Systems (NeurIPS), 2025

Abstract: Real-time decoding of neural activity is central to neuroscience and neurotechnology applications, from closed-loop experiments to brain-computer interfaces, where models are subject to strict latency constraints. Traditional methods, including simple recurrent neural networks, are fast and lightweight but often struggle to generalize to unseen data. In contrast, recent Transformer-based approaches leverage large-scale pretraining for strong generalization performance, but typically have much larger computational requirements and are not always suitable for low-resource or real-time settings. To address these shortcomings, we present POSSM, a novel hybrid architecture that combines individual spike tokenization via a cross-attention module with a recurrent state-space model (SSM) backbone to enable (1) fast and causal online prediction on neural activity and (2) efficient generalization to new sessions, individuals, and tasks through multi-dataset pretraining. We evaluate POSSM's decoding performance and inference speed on intracortical decoding of monkey motor tasks, and show that it extends to clinical applications, namely handwriting and speech decoding in human subjects. Notably, we demonstrate that pretraining on monkey motor-cortical recordings improves decoding performance on the human handwriting task, highlighting the exciting potential for cross-species transfer. In all of these tasks, we find that POSSM achieves decoding accuracy comparable to state-of-the-art Transformers, at a fraction of the inference cost (up to 9x faster on GPU). These results suggest that hybrid SSMs are a promising approach to bridging the gap between accuracy, inference speed, and generalization when training neural decoders for real-time, closed-loop applications.

Paper arXiv

BibTex

@misc{ryoo2025generalizable,
    title={Generalizable, real-time neural decoding with hybrid state-space models}, 
    author={Avery Hee-Woon Ryoo and Nanda H. Krishna and Ximeng Mao and Mehdi Azabou and Eva L. Dyer and Matthew G. Perich and Guillaume Lajoie},
    year={2025},
    eprint={2506.05320},
    archivePrefix={arXiv},
    primaryClass={q-bio.NC},
    url={https://arxiv.org/abs/2506.05320}, 
}

APA

Ryoo, A. H.-W., Krishna, N. H., Mao, X., Azabou, M., Dyer, E. L., Perich, M. G., & Lajoie, G. (2025). Generalizable, real-time neural decoding with hybrid state-space models. arXiv 2506.05320

Neural Encoding and Decoding at Scale

Neuroscience

Transformers

Yizi Zhang⁺, Yanchen Wang⁺, Mehdi Azabou, Alexandre Andre, Zixuan Wang, Hanrui Lyu, The International Brain Laboratory, Eva Dyer, Liam Paninski, Cole Hurwitz

International Conference on Machine Learning (ICML), accepted as Spotlight, 2025

Abstract: Recent work has demonstrated that large-scale, multi-animal models are powerful tools for characterizing the relationship between neural activity and behavior. Current large-scale approaches, however, focus exclusively on either predicting neural activity from behavior (encoding) or predicting behavior from neural activity (decoding), limiting their ability to capture the bidirectional relationship between neural activity and behavior. To bridge this gap, we introduce a multimodal, multi-task model that enables simultaneous Neural Encoding and Decoding at Scale (NEDS). Central to our approach is a novel multi-task-masking strategy, which alternates between neural, behavioral, within-modality, and cross-modality masking. We pretrain our method on the International Brain Laboratory (IBL) repeated site dataset, which includes recordings from 83 animals performing the same visual decision-making task. In comparison to other large-scale models, we demonstrate that NEDS achieves state-of-the-art performance for both encoding and decoding when pretrained on multi-animal data and then fine-tuned on new animals. Surprisingly, NEDS's learned embeddings exhibit emergent properties: even without explicit training, they are highly predictive of the brain regions in each recording. Altogether, our approach is a step towards a foundation model of the brain that enables seamless translation between neural activity and behavior

Paper arXiv

BibTex

@misc{zhang2025neuralencodingdecodingscale,
    title={Neural Encoding and Decoding at Scale}, 
    author={Yizi Zhang and Yanchen Wang and Mehdi Azabou and Alexandre Andre and Zixuan Wang and Hanrui Lyu and The International Brain Laboratory and Eva Dyer and Liam Paninski and Cole Hurwitz},
    year={2025},
    eprint={2504.08201},
    archivePrefix={arXiv},
    primaryClass={q-bio.NC},
    url={https://arxiv.org/abs/2504.08201}, 
}

APA

Zhang, Y., Wang, Y., Azabou, M., Andre, A., Wang, Z., Lyu, H., The International Brain Laboratory, Dyer, E., Paninski, L., & Hurwitz, C. (2025). Neural Encoding and Decoding at Scale. arXiv 2504.08201

Unified Pretraining on Mixed Optophysiology and Electrophysiology Data Across Brain Regions

Neuroscience

Transformers

Ian Jarratt Knight, Vinam Arora, Mehdi Azabou, Eva L Dyer

NeurIPS 2025 Foundation models for the brain and body workshop (Selected for an oral)

OpenReview

Exploiting All Laplacian Eigenvectors for Node Classification with Graph Transformers

Geometric Deep Learning

Vinam Arora, Divyansha Lachi, Shivashriganesh P. Mahato, Mehdi Azabou, Zihao Chen, Eva L. Dyer

NeurIPS 2025 New Perspectives in Graph Machine Learning workshop

OpenReview

2024

Building a Foundation Model for Neuroscience

Neuroscience

Self-Supervised Learning

Time Series

Mehdi Azabou

Ph.D. Dissertation, Georgia Institute of Technology, 2024

PDF Georgia Tech Library

BibTex

@thesis{azabou2024foundation,
    title={Building a Foundation Model for Neuroscience},
    author={Azabou, Mehdi},
    year={2024},
    school={Georgia Institute of Technology},
    url={https://hdl.handle.net/1853/76871}
}

APA

Azabou, M. (2024). Building a Foundation Model for Neuroscience [Doctoral dissertation, Georgia Institute of Technology]. https://hdl.handle.net/1853/76871

GraphFM: A Scalable Framework For Multi-Graph Pretraining

Geometric Deep Learning

Transformers

Divyansha Lachi, Mehdi Azabou, Vinam Arora, Eva L. Dyer

Preprint, 2024

Abstract: Graph neural networks are typically trained on individual datasets, often requiring highly specialized models and extensive hyperparameter tuning. This datasetspecific approach arises because each graph dataset often has unique node features and diverse connectivity structures, making it difficult to build a generalist model. To address these challenges, we introduce a scalable multi-graph multi-task pretraining approach specifically tailored for node classification tasks across diverse graph datasets from different domains. Our method, Graph Foundation Model (GraphFM), leverages a Perceiver-based encoder that employs learned latent tokens to compress domain-specific features into a common latent space. This approach enhances the model's ability to generalize across different graphs and allows for scaling across diverse data. We demonstrate the efficacy of our approach by training a model on 152 different graph datasets comprising over 7.4 million nodes and 189 million edges, establishing the first set of scaling laws for multi-graph pretraining on datasets spanning many domains (e.g., molecules, citation and product graphs). Our results show that pretraining on a diverse array of real and synthetic graphs improves the model's adaptability and stability, while performing competitively with state-of-the-art specialist models. This work illustrates that multi-graph pretraining can significantly reduce the burden imposed by the current graph training paradigm, unlocking new capabilities for the field of graph neural networks by creating a single generalist model that performs competitively across a wide range of datasets and tasks.

Paper arXiv

BibTex

@inproceedings{
    azabou2025multisession,
    title={Multi-session, multi-task neural decoding from distinct cell-types and brain regions},
    author={Mehdi Azabou and Krystal Xuejing Pan and Vinam Arora and Ian Jarratt Knight and Eva L Dyer and Blake Aaron Richards},
    booktitle={The Thirteenth International Conference on Learning Representations},
    year={2025},
    url={https://openreview.net/forum?id=IuU0wcO0mo}
}

APA

Azabou, M., Pan, K. X., Arora, V., Knight, I. J., Dyer, E. L. & Richards, B. A. (2025). Multi-session, multi-task neural decoding from distinct cell-types and brain regions. The Thirteenth International Conference on Learning Representations

BibTex

@Misc{lachi2024graphfm,
Title={GraphFM: A Scalable Framework For Multi-Graph Pretraining}, 
Author={Divyansha Lachi And Mehdi Azabou And Vinam Arora And Eva Dyer},
Year={2024},
Eprint={2407.11907},
ArchivePrefix={ArXiv},
PrimaryClass={Cs.LG},
Url={Https://Arxiv.Org/Abs/2407.11907}, 
}

APA

Lachi, D., Azabou, M., Arora, V. & Dyer, E. L. (2024). GraphFM: A Scalable Framework For Multi-Graph Pretraining.  ArXiv 2407.11907

Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution

Neuroscience

Transformers

Yizi Zhang, Yanchen Wang, Donato Jimenez-Beneto, Zixuan Wang, Mehdi Azabou, Blake Richards, Olivier Winter, The International Brain Laboratory, Eva Dyer, Liam Paninski, Cole Hurwitz

Neural Information Processing Systems (NeurIPS), 2024

Abstract: Neuroscience research has made immense progress over the last decade, but our understanding of the brain remains fragmented and piecemeal: the dream of probing an arbitrary brain region and automatically reading out the information encoded in its neural activity remains out of reach. In this work, we build towards a first foundation model for neural spiking data that can solve a diverse set of tasks across multiple brain areas. We introduce a novel self-supervised modeling approach for population activity in which the model alternates between masking out and reconstructing neural activity across different time steps, neurons, and brain regions. To evaluate our approach, we design unsupervised and supervised prediction tasks using the International Brain Laboratory repeated site dataset, which is comprised of Neuropixels recordings targeting the same brain locations across 48 animals and experimental sessions. The prediction tasks include single-neuron and region-level activity prediction, forward prediction, and behavior decoding. We demonstrate that our multi-task-masking (MtM) approach significantly improves the performance of current state-of-the-art population models and enables multitask learning. We also show that by training on multiple animals, we can improve the generalization ability of the model to unseen animals, paving the way for a foundation model of the brain at single-cell, single-spike resolution. Project page and code: https://ibl-mtm.github.io/

Project page Paper Code Model arXiv

BibTex

@InProceedings{Zhang_2024_arXiv,
author    = {Zhang, Yizi and Wang, Yanchen and Benetó, Donato Jiménez and Wang, Zixuan and Azabou, Mehdi and Richards, Blake and Winter, Olivier and The International Brain Laboratory and Dyer, Eva and Paninski, Liam and Hurwitz, Cole},
title     = {Towards a “universal translator” for neural dynamics at single-cell, single-spike resolution},
booktitle = {arXiv},
month     = {July},
year      = {2024},
url       = {http://arxiv.org/abs/2407.14668}
}

APA

Zhang, Y., Wang, Y., Jimenez-Beneto, D., Wang, Z., Azabou, M., Richards, B., Winter, O., The International Brain Laboratory, Dyer, E., Paninski, L. & Hurwitz, C. (2024). Towards a “universal translator” for neural dynamics at single-cell, single-spike resolution ArXiv 2407.14668

Large-scale pretraining on neural data allows for transfer across subjects, tasks and species

Abstract

Neuroscience

Transformers

Mehdi Azabou, Vinam Arora, Patrick Mineault, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, Eva L. Dyer

Computational and Systems Neuroscience (COSYNE), 2024

Abstract: Current population-level models of neural data are built to operate on a single dataset, condition, or task at a time. This both limits the amount of training data we can build models on, and also makes it difficult to build unified models where it may be possible to compare across individuals and over distinct tasks. To address this gap, we introduce a novel approach for large-scale training over many diverse datasets spanning many sessions and tasks. Much like words or groups of letters are passed into transformers in natural language processing, our approach centers around a novel way to breakdown neural population activity into parts or “tokens” that can be processed by a large transformer model. Rather than binning data, we represent each individual spike as a token and associate these spike events with a learned unit-level embedding which can capture information about the identity of a neural unit or channel (in the case of multiunits). We combine this input tokenization method with an encoder- decoder architecture that compresses the input spikes into a latent space and learns interactions across spikes in time and across neurons. To demonstrate the flexibility and scalability of our multi-session training framework, we apply it to a wide range of motor, visual, and decision making tasks from electrophysiology datasets collected in monkeys, humans, and mice. In particular, we train three large models, the first being from 7 nonhuman primates (27,373 units) engaging in four different motor tasks across three laboratories, the second combining human and monkey motor cortical datasets (9,789 monkey units + 1,920 human units) in diverse tasks, and the third from 58 mice (99,180 units) engaged in visual tasks during recordings to multiple regions in the cortex, deep, and midbrain structures. In these cases, we demonstrate that our pretrained models can be rapidly adapted to new, unseen sessions, enabling few-shot performance across animals and species with minimal labels. Our results highlight the power of scale and demonstrate how pretraining on large datasets can benefit neural data analysis.

Project page

2023

A Unified, Scalable Framework for Neural Population Decoding

Neuroscience

Transformers

We present a novel transformer architecture based the PerceiverIO and trained on a large-scale dataset of neural recordings from nonhuman primates.

Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, Eva L. Dyer

Neural Information Processing Systems (NeurIPS), 2023

Abstract: Our ability to use deep learning approaches to decipher neural activity would likely benefit from greater scale, in terms of both the model size and the datasets. However, the integration of many neural recordings into one unified model is challenging, as each recording contains the activity of different neurons from different individual animals. In this paper, we introduce a training framework and architecture designed to model the population dynamics of neural activity across diverse, large-scale neural recordings. Our method first tokenizes individual spikes within the dataset to build an efficient representation of neural events that captures the fine temporal structure of neural activity. We then employ cross-attention and a PerceiverIO backbone to further construct a latent tokenization of neural population activities. Utilizing this architecture and training framework, we construct a large- scale multi-session model trained on large datasets from seven nonhuman primates, spanning over 158 different sessions of recording from over 27,373 neural units and over 100 hours of recordings. In a number of different tasks, we demonstrate that our pretrained model can be rapidly adapted to new, unseen sessions with unspecified neuron correspondence, enabling few-shot performance with minimal labels. This work presents a powerful new approach for building deep learning tools to analyze neural data and stakes out a clear path to training at scale for neural decoding models.

Project page Paper Poster arXiv OpenReview

BibTex

@inproceedings{azabou2023a,
    title={A Unified, Scalable Framework for Neural Population Decoding},
    author={Mehdi Azabou and Vinam Arora and Venkataramana Ganesh and Ximeng Mao and Santosh B Nachimuthu and Michael Jacob Mendelson and Blake Aaron Richards and Matthew G Perich and Guillaume Lajoie and Eva L Dyer},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
    year={2023},
    url={https://openreview.net/forum?id=sw2Y0sirtM}
}

Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis

Behavior

SSL

Optimal Transport

Mehdi Azabou, Michael Mendelson, Nauman Ahad, Maks Sorokin, Shantanu Thakoor, Carolina Urzay, Eva L. Dyer

Neural Information Processing Systems (NeurIPS), accepted as Spotlight (3% submissions), 2023

Project page Paper Poster arXiv OpenReview

Half-Hop: A graph upsampling approach for slowing down message passing

Geometric Deep Learning

Half-Hop is a plug-and-play augmentation that works with any message passing neural network. We show that it leads to improved performance on various supervised and self-supervised benchmarks.

SSL

Self-Supervised Learning

Half-Hop is used as an augmentation to generate more diverse views for methods like BGRL and GRACE.

Mehdi Azabou, Venkataramana Ganesh, Shantanu Thakoor, Chi-Heng Lin, Lakshmi Sathidevi, Ran Liu, Michal Valko, Petar Veličković, Eva L. Dyer

International Conference on Machine Learning (ICML), 2023

Paper Code Article PyTorch Geometric Talk Poster OpenReview

BibTex

@InProceedings{pmlr-v202-azabou23a,
  title = 	 {Half-Hop: A graph upsampling approach for slowing down message passing},
  author =       {Azabou, Mehdi and Ganesh, Venkataramana and Thakoor, Shantanu and Lin, Chi-Heng and Sathidevi, Lakshmi and Liu, Ran and Valko, Michal and Veli\v{c}kovi\'{c}, Petar and Dyer, Eva L},
  booktitle = 	 {Proceedings of the 40th International Conference on Machine Learning},
  pages = 	 {1341--1360},
  year = 	 {2023},
  editor = 	 {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
  volume = 	 {202},
  series = 	 {Proceedings of Machine Learning Research},
  month = 	 {23--29 Jul},
  publisher =    {PMLR},
  pdf = 	 {https://proceedings.mlr.press/v202/azabou23a/azabou23a.pdf},
  url = 	 {https://proceedings.mlr.press/v202/azabou23a.html},
}

APA

Azabou, M., Ganesh, V., Thakoor, S., Lin, C. H., Sathidevi, L., Liu, R., Valko, M., Veličković, P. & Dyer, E. L. (2023). Half-Hop: A graph upsampling approach for slowing down message passing.  Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research

Transcriptomic cell type structures in vivo neuronal activity across multiple time scales

Neuroscience

We reveal the existence of characteristic computational fingerprints of transcriptomic cell types, during neuronal activity across diverse contexts: drifting gratings, naturalistic movies.

Attention

LOLCAT uses soft-attention, which enables the model to aggregate trial information across different scales.

Aidan Schneider⁺, Mehdi Azabou⁺, Louis McDougall-Vigier, David Parks, Sahara Ensley, Kiran Bhaskaran-Nair, Tom Nowakowski, Eva L. Dyer^*, Keith B. Hengen^*

⁺Contributed equally as co-first authors

Cell Reports, Volume 42, Issue 4, April 2023

Abstract: Cell type is hypothesized to be a key determinant of the role of a neuron within a circuit. However, it is unknown whether a neuron’s transcriptomic type influences the timing of its activity in the intact brain. In other words, can transcriptomic cell type be extracted from the time series of a neuron’s activity? To address this question, we developed a new deep learning architecture that learns features of interevent intervals across multiple timescales (milliseconds to >30 min). We show that transcriptomic cell class information is robustly embedded in the timing of single neuron activity recorded in the intact brain of behaving animals (calcium imaging and extracellular electrophysiology), as well as in a bio-realistic model of visual cortex. In contrast, we were unable to reliably extract cell identity from summary measures of rate, variance, and interevent interval statistics. We applied our analyses to the question of whether transcriptomic subtypes of excitatory neurons represent functionally distinct classes. In the calcium imaging dataset, which contains a diverse set of excitatory Cre lines, we found that a subset of excitatory cell types are computationally distinguishable based upon their Cre lines, and that excitatory types can be classified with higher accuracy when considering their cortical layer and projection class. Here we address the fundamental question of whether a neuron, within a complex cortical network, embeds a fingerprint of its transcriptomic identity into its activity. Our results reveal robust computational fingerprints for transcriptomic types and classes across diverse contexts, defined over multiple timescales.

Paper Code Article Cell Reports

BibTex

@article{SCHNEIDER2023112318,
title = {Transcriptomic cell type structures in vivo neuronal activity across multiple timescales},
journal = {Cell Reports},
volume = {42},
number = {4},
pages = {112318},
year = {2023},
issn = {2211-1247},
doi = {https://doi.org/10.1016/j.celrep.2023.112318},
url = {https://www.sciencedirect.com/science/article/pii/S2211124723003297},
author = {Aidan Schneider and Mehdi Azabou and Louis McDougall-Vigier and David F. Parks and Sahara Ensley and Kiran Bhaskaran-Nair and Tomasz Nowakowski and Eva L. Dyer and Keith B. Hengen},
}

APA

Schneider, A., Azabou, M., McDougall-Vigier, L., Parks, D. B., Ensley, S., Bhaskaran-Nair, K., Nowakowski, T., Dyer, E. L. & Hengen, K. B. (2023). Transcriptomic cell type structures in vivo neuronal activity across multiple time scales. Cell Reports, Volume 42, Issue 4, 2023

BibTex

@inproceedings{azabou2023relax,
    title={Relax, it doesn{\textquoteright}t matter how you get there: A new self-supervised approach for multi-timescale behavior analysis},
    author={Mehdi Azabou and Michael Jacob Mendelson and Nauman Ahad and Maks Sorokin and Shantanu Thakoor and Carolina Urzay and Eva L Dyer},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
    year={2023},
    url={https://openreview.net/forum?id=RInTOCEL3l}
}

Learning signatures of decision making from many individuals playing the same game

Behavior

We study a large-scale behavioral dataset from 1,000 humans playing a 3-armed bandit task. By learning to predict future choices, we identify characteristic playing strategies and signatures of differences across individuals.

SSL

Self-Supervised Learning

To simultaneously extract both global and local variables from complex human behavior, our method combines a multi-scale temporal convolutional network with latent prediction tasks, where we encourage embeddings across the entire sequence, as well as subsets of the sequence, to be mapped to similar points in the latent space.

Michael Mendelson⁺, Mehdi Azabou⁺, Suma Jacob, Nicola Grissom, David P. Barrow, Becket Ebitz, Alexander Herman, Eva L. Dyer

⁺Contributed equally as co-first authors

2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), Baltimore, Maryland, April 2023

Abstract: Human behavior is incredibly complex and the factors that drive decision making—from instinct, to strategy, to biases between individuals—often vary over multiple timescales. In this paper, we design a predictive framework that learns representations to encode an individual's `behavioral style', i.e. long-term behavioral trends, while simultaneously predicting future actions and choices. The model explicitly separates representations into three latent spaces: the recent past space, the short-term space, and the long-term space where we hope to capture individual differences. To simultaneously extract both global and local variables from complex human behavior, our method combines a multi-scale temporal convolutional network with latent prediction tasks, where we encourage embeddings across the entire sequence, as well as subsets of the sequence, to be mapped to similar points in the latent space. We develop and apply our method to a large-scale behavioral dataset from 1,000 humans playing a 3-armed bandit task, and analyze what our model's resulting embeddings reveal about the human decision making process. In addition to predicting future choices, we show that our model can learn rich representations of human behavior over multiple timescales and provide signatures of differences in individuals.

Paper IEEE Xplore arXiv

BibTex

@INPROCEEDINGS{10123846,
    author={Mendelson, Michael J. and Azabou, Mehdi and Jacob, Suma and Grissom, Nicola and Darrow, David and Ebitz, Becket and Herman, Alexander and Dyer, Eva L.},
    booktitle={2023 11th International IEEE/EMBS Conference on Neural Engineering (NER)}, 
    title={Learning signatures of decision making from many individuals playing the same game}, 
    year={2023},
    volume={},
    number={},
    pages={1-5},
    doi={10.1109/NER52421.2023.10123846}
}

APA

Mendelson, M., Azabou, M., Jacob, S., Grissom, N., Darrow, D., Ebitz, B., Herman, A. & Dyer, E. L. (2023). Learning signatures of decision making from many individuals playing the same game. 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), Baltimore, MD, USA, 2023, pp. 1-5, doi: 10.1109/NER52421.2023.10123846.

Detecting change points in neural population activity with contrastive metric learning

Neuroscience

We introduce an approach for identifying switches in neural population during free behavior, and find that different groups of neurons are useful for different contexts.

Carolina Urzay, Nauman Ahad, Mehdi Azabou, Aidan Schneider, Geethika Atamkuri, Keith B. Hengen, Eva L. Dyer

2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), Baltimore, Maryland, April 2023

Multi-session, multi-task neural decoding from distinct cell-types and brain regions Neuroscience Transformers

Know Thyself by Knowing Others: Learning Neuron Identity from Population Context Neuroscience SSL

Generalizable, real-time neural decoding with hybrid state-space models Neuroscience Time Series

Neural Encoding and Decoding at Scale Neuroscience Transformers

Unified Pretraining on Mixed Optophysiology and Electrophysiology Data Across Brain Regions Neuroscience Transformers

Exploiting All Laplacian Eigenvectors for Node Classification with Graph Transformers Geometric Deep Learning

2024

Building a Foundation Model for Neuroscience Neuroscience Self-Supervised Learning Time Series

GraphFM: A Scalable Framework For Multi-Graph Pretraining Geometric Deep Learning Transformers

Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution Neuroscience Transformers

Large-scale pretraining on neural data allows for transfer across subjects, tasks and species Abstract Neuroscience Transformers

2023

A Unified, Scalable Framework for Neural Population Decoding Neuroscience Transformers We present a novel transformer architecture based the PerceiverIO and trained on a large-scale dataset of neural recordings from nonhuman primates.

Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis Behavior SSL Optimal Transport

Detecting change points in neural population activity with contrastive metric learning Neuroscience We introduce an approach for identifying switches in neural population during free behavior, and find that different groups of neurons are useful for different contexts.

Multi-session, multi-task neural decoding from distinct cell-types and brain regions

Neuroscience

Transformers

Know Thyself by Knowing Others: Learning Neuron Identity from Population Context

Neuroscience

SSL

Generalizable, real-time neural decoding with hybrid state-space models

Neuroscience

Time Series

Neural Encoding and Decoding at Scale

Neuroscience

Transformers

Unified Pretraining on Mixed Optophysiology and Electrophysiology Data Across Brain Regions

Neuroscience

Transformers

Exploiting All Laplacian Eigenvectors for Node Classification with Graph Transformers

Geometric Deep Learning

Building a Foundation Model for Neuroscience

Neuroscience

Self-Supervised Learning

Time Series

GraphFM: A Scalable Framework For Multi-Graph Pretraining

Geometric Deep Learning

Transformers

Towards a "universal translator" for neural dynamics at single-cell, single-spike resolution

Neuroscience

Transformers

Large-scale pretraining on neural data allows for transfer across subjects, tasks and species

Abstract

Neuroscience

Transformers

A Unified, Scalable Framework for Neural Population Decoding

Neuroscience

Transformers

We present a novel transformer architecture based the PerceiverIO and trained on a large-scale dataset of neural recordings from nonhuman primates.

Relax, it doesn't matter how you get there: A new self-supervised approach for multi-timescale behavior analysis

Behavior

SSL

Optimal Transport

Detecting change points in neural population activity with contrastive metric learning

Neuroscience

We introduce an approach for identifying switches in neural population during free behavior, and find that different groups of neurons are useful for different contexts.