In this work, we developed an automated pipeline for generating audiovisual stimuli targeted at neuroimaging experiments with children. The pipeline consists of two main components: the first generates auditory stimuli from text, and the second creates video stimuli in which an animated character says the stimuli out loud. highlights the value of automated tools in developing large sets of standardized, engaging, and well-controlled stimuli--making condition-rich paradigms more feasible in developmental populations.

Our findings point to semantics (rather than form-level properties of language). We ruled out shallow form-level predictors (word length and frequency) and found no evidence that formal linguistic similarity (e.g., syntax or phonology) explained cross-linguistic generalization. These findings suggest that shared brain responses across languages are primarily driven by linguistic meaning. Altogether, our findings provide strong evidence for a shared, meaning-based component in how the human brain processes diverse languages.

Second, we evaluate the LLMs' ability to perform various linguistic tasks such as question-answering (GLUE), acceptability judgments (BLiMP), and judging syntax (SyntaxGym). We show that ablating "language network" units leads to large performance drops (up to 40%), while lesioning random units has ~no impact.

Third, we ask whether these "language network" units are similar to human brain responses. Prior work comparing LLMs to brain data typically uses units from a full LLM layers or across the whole LLM, without isolating functional subcomponents. In this work, we first show that the "language network" units reproduce empirical findings about the human language network such as a higher response to Sentences than unconnected words (W), Jabberwocky (J) and non-words in held-out materials (such as in several studies, e.g., Fedorenko et al., 2010). Next, we show that the activation patterns of "language network" units are more aligned with human brain responses than randomly sampled units. This work moves toward mapping the internal organization of LLMs and understanding representational correspondence between brain regions and LLMs in terms of shared, behaviorally relevant function.

Finally, we demonstrate the feasibility of applying the neuroscientific approach to identify relevant LLM units in other domains, such as within structured reasoning and social inference.

We present a speeded version of a widely used localizer (Fedorenko et al., 2010) for the language network (for a review on the language network, see Fedorenko et al., 2024). This localizer uses a reading task contrasting sentences and sequences of nonwords. However, our localizer is faster as we present each word/nonword for 200ms instead of 450ms.

In the paper, we first investigated whether the speeded localizer version can reliably localize language-responsive areas in individual participants (n=24). The brief answer is yes. First, we looked at the voxel-level activation maps, and found that the speeded localizer elicited highly similar activation topographies as the standard localizer (Fedorenko et al., 2010). Next, we investigated the BOLD response magnitudes and found that the speeded localizer elicited a robust response to sentences, with an even greater sentences > nonwords effect size than the standard localizer. Second, we asked whether increased processing difficulty due to speeded reading affects neural responses in the language network and/or elsewhere in the brain. We focused on the Multiple Demand (MD) network, sensitive to cognitive effort across various domains and in some cases of effortful language comprehension. It is an open question how the MD network (and other brain regions) contributes to language processing during different kinds of effortful linguistic processing. We found that the MD network responded ~43% more strongly during speeded sentence reading vs. standard sentence reading (compared to a ~16% increase in the language network to speeded reading). Moreover, we did not observe a nonwords > sentences effect in the speeded localizer, compared to the standard one. Hence, speeded reading was more effortful than slower reading, and this cost loaded largely onto the domain-general Multiple Demand (MD) network (only minimally manifesting in the language network and not affecting the ability to localize language areas).

We hope that this language localizer will be useful to researchers when time is of essence (this localizer was developed when we were trying to cram hundreds of sentences into a single fMRI session a few years ago). It should work well with any proficient readers.

Next, we scaled up our approach, and trained a BERT Transformer model with the topographical motifs, on masked word prediction. The performance of the Topoformer-BERT model was similar to that of a non-topograhical BERT across several linguistic benchmarks. Hence, topography does not impede function (nor enhance it). Finally, we asked whether the topographic organization in the Topoformer-BERT model corresponded to that of the human language system. We found that the language responses in the human brain were topographic ("smooth"/"patchy"), and critically, that dimensions between Topoformer-BERT and the brain could be aligned.
This work provides a critical step towards developing more biologically plausible LLMs.

Next, we lay out arguments as to why, a priori, we would or would not expect LMs to share similarities with the human language system. In brief, we would expect LMs to share similarities with the human language system because of their similar behavioral language performance (formal linguistic competence, e.g., Mahowald, Ivanova et al., 2023), both systems are highly modulated and/or shaped by prediction, both systems are sensitive to multiple levels of linguistic structure, and finally, ‘small’ LMs that do language well, struggle with other tasks (e.g., simple arithmetic), suggesting that near-human language ability does not entail near-human reasoning ability. Conversely, LMs are very different from humans in their learning procedure, access to long-range contexts of prior input, and of course, hardware differences.

We then describe evidence that LMs represent linguistic information similarly enough to humans to enable relatively accurate brain encoding and decoding, and then turn to the question: Which properties of LMs enable them to capture human responses to language? LMs vary along many dimensions, which we separated into 3 categories: model architecture, model behavior and model training. A few take-aways are: For architecture, contextualized LMs provide a big improvement in brain-LM alignment over decontextualized semantic models. However, within contextualized LMs, many architectures fit brain data well. Larger LLMs predict brain data better, but become worse at predicting human language behavior. For behavior, in line with the idea that neural representations are shaped by behavioral demands, LMs’ ability to predict linguistic input is positively correlated with brain alignment. It remains unknown whether prediction per se, or other factors such as representational generality, is the key factor underlying alignment. Lastly, for training, even LMs trained on a developmentally plausible amount of training data align with brain data. Semantic properties of training data seem to matter more for brain-LM alignment than morphosyntactic ones. Fine-tuning can increase brain-LM alignment (task- and model dependent).

The final part of the paper discusses the general use as LMs as in silico language networks, and we describe challenges and future directions associated with the use of LMs in understanding more about language in the mind and brain.

The review focuses on visual and language processing, and we broadly tackle two questions: i) How should we use neuroscientific data in model development (raw experimental data vs. qualitative insights)?, and ii) How should we collect experimental data for model development (model-free vs. model-based)?. We discuss the pros and cons associated with each stance, and moreover, we review how neuroscience data has traditionally been leveraged to advance model-building within the domains of vision and language. Finally, we discuss directions for both experimenters and model developers in the quest to advance artificial models of brain activity and behavior.

We showed that these model-selected new sentences indeed drive and suppress the activity of human language areas in new individuals. Hence, we trained an encoding model to generate predictions about the magnitude of activation in the language network for the new drive/suppress sentences and then closed the loop, so to speak, by collecting brain data for the new sentences from new participants, effectively showing that GPT2-XL captures features related to language processing that enables us to “control” brain activity.

Second, we wanted to understand what kinds of sentences that the language network responds to the most. This general approach is rooted in the work by Hubel and Wiesel in the late 1950’s and 1960’s, who discovered what kinds of visual input (e.g., specific orientations of light) that would make neurons in the visual cortex of cats and monkeys fire the most. The core premise behind Hubel and Wiesel's approach (and ours) is that understanding what kinds of stimuli that maximally engage a given system provides insight into the functional properties of that system. In our study, we characterized all of our sentences (spanning the full range of brain responses from low to high) using 11 linguistic properties (we collected a lot of behavioral ratings!). We discovered that sentences with a certain degree of unusual grammar and/or meaning elicit the highest activity in the language network in the form of an inverted U-shape – for instance, sentences like “I’m progressive and you fall right.” would elicit the highest responses. Conversely, sentences that were highly normal/frequent (e.g., "We were sitting on the couch.") or highly unusual, like a string of words (e.g., "LG'll obj you back suggested Git.") would elicit quite low responses. In other words, the language network responds strongly to sentences that are “normal” (language-like) enough to engage it, but unusual enough to tax it. This means that specific areas of your brain – the language network – will respond selectively to linguistic input that aligns with the statistics of the language and will work really hard to make sense of the words and how they go together.

Taking a step back, we establish LLMs as causal tools to investigate language in the mind and brain, with implications for future basic-science investigations and clinical research.

Other three findings that are worth highlighting: i) A model’s training data significantly influences its similarity to brain responses: models trained to perform word/speaker recognition in the presence of background noise are much better at accounting for auditory cortex responses than models trained in quiet. ii) A model trained on multiple tasks was the best model of the auditory cortex overall, and also accounted for neural tuning to speech and music. Training on particular tasks improved predictions for specific types of tuning, with the MultiTask model getting “best of all worlds”. iii) Finally, in light of recent discussion suggesting that the dimensionality of a model’s representation correlates with regression-based brain predictions, we evaluated how the effective dimensionality (ED) of each network stage correlated with both the regression and RSA metrics. There was a modest correlation between ED and brain-model similarity but significantly less than that between the two datasets or the two similarity measures. Thus ED does not seem to explain most of the variance across DNNs in our datasets.

Finally, our work demonstrates that if the core goal is to obtain the most quantitatively accurate model of the auditory system, machine learning models move us closer to this goal. However, our findings also highlight the explanatory gap that remains (predictions of all models are well below the noise ceiling), as well as the need for better model-brain evaluation metrics and finer-grained neural recordings to better distinguish models.

We used functional data from a well‑validated language localizer task (e.g., Lipkin et al., 2022) for a set of 800 participants, jointly aligning the language contrast (sentences versus strings of non‑words) with cortical folding patterns, demonstrating better alignment in both folding patterns and function compared to two existing methods.

We took an initial stab at this question, and asked i) Do brain and LLM representations contain similar amounts of information?, and ii) Can we leverage an information bottleneck approach to generate compressed representations of brain activity and better unify the representational spaces of humans and LLMs during language processing?
Our prelimary results show that compressing brain activity in frontal language regions improves alignment with LLMs, suggesting that frontal regions in the human brain encode information that LLMs do not. There was no benefit to compressing temporal brain regions, suggesting that these regions might align better with the information present in current LLMs. Broadly, our work establishes the use of information theoretic tools as a "dial" to modify representational complexity and to better unify representational spaces, including those from biological and artificial lanuage processing.

This means that manipulations that remove the content words or alter the meaning of the sentence decrease brain predictivity. Conversely, manipulations that remove function words or perturb the syntactic structure of the sentence (e.g., by shuffling the word order) do not lead to large decreases in brain predictivity. We show that stimulus manipulations that adversely affect brain predictivity have two interpretable causes: i) they lead to more divergent representations in the LLM’s embedding space (relative to the representations of the original stimuli), and decrease the LLM’s ability to predict upcoming tokens in those stimuli. So, stimuli that are less predictable on average lead to larger decreases in brain predictivity performance.

In summary, the critical result—that lexical‑semantic content is the main contributor to the similarity between LLM representations and neural ones—aligns with the idea that the goal of the human language system is to extract meaning from linguistic strings. Finally, this work highlights the strength of systematic experimental manipulations for evaluating how close we are to accurate and generalizable models of the human language network.

We used fMRI methods to establish that the right hemisphere language network is similar to the left hemisphere language network in controls. However, the critical question was whether EG’s intact left lateral frontal lobe contained language-responsive areas. We found no reliable response to language in EG’s intact left frontal lobe, suggesting that the existence of temporal language areas appears to be a prerequisite for the emergence of language areas in the frontal lobe.

The atlas was obtained from >800 individuals based on functional localization (a contrast between processing of sentences and a linguistically/acoustically degraded condition, such as non-word strings). Thus, among these ~800 individuals, we provide a group average map that allows us to quantify and visualize where ‘the average’ language network resides.

Examples of use cases of LanA are: 1) A common reference frame for analyzing group-level activation peaks from past/future fMRI studies, 2) Lesion locations in individual brains, 3) Electrode locations in intracranial ECoG/SEEG investigations, 4) Functional mapping during brain surgery when fMRI is not possible, and others. The atlas is openly available, alongside individual contrast/significance maps and demographic data.

Lexical features operate on individual lexical items (words) and entail features such as concreteness, age of acquisition, lexical decision latency, and contextual diversity. As several properties of a sentence cannot be attributed to individual words, so the contextual module quantifies a sentence as a whole. This module entails features such as syntactic storage and integration cost, center embedding depth, and sentiment. Hence, SentSpace provides an interpretable sentence embedding with features that have been to shown to affect language processing. SentSpace allows for quantification and comparison of different types of text and can be useful for answering questions like: How does text generated by an artificial language model compare to that of humans? How does utterances produced by neurotypicals compare to that of individuals with communication disorders? What psycholinguistic information do high-dimensional vector representations from artificial language models capture?

We implemented an attention training paradigm designed by DeBettencourt et al., (2015) in EEG. In a double-blinded design, we trained twenty-two participants on the attention paradigm within a single neurofeedback session with behavioral pretraining and posttraining sessions.

We demonstrate that we are able to decode covert visual attention in real time. First of all, we report a mean classifier decoding error rate of 34.3% (chance = 50%). Second, we link this decoding performance to behavioral states, and show that within the neurofeedback group, there was a greater level of task-relevant attentional information decoded in the participant's brain before making a correct behavioral response than before an incorrect response (not evident in the control group; interaction p=7.23e−4). This indicates that we were able to achieve a meaningful measure of subjective attentional state in real time and control participants' behavior during the neurofeedback session. Finally, we do not provide conclusive evidence whether a single neurofeedback session per se provided lasting effects in sustained attention abilities.

This finding suggests a heavy reliance on visual information when no sensory feedback is available, possibly as a compensatory mechanism. Conclusively, we demonstrate that closed-loop proprioceptive feedback can enable desired neuroplastic changes toward improved neuroprosthetic capability.

I think these two points open up for multiple exciting research questions: Given that better-performing models are more brain-like, how can we engineer more brain-inspired models? Most state-of-the-art language models are inefficient (requiring billions of parameters and training samples resulting in massive energy expenditure), not robust (can be fooled by adversarial input), and not very interpretable (making it challenging to localize causes of success/unwanted capabilities). How can we exploit principles from the human brain that allows us to processes language efficiently and robustly? Can we modularize or constrain language model representations using human data? In which scenarios do interpretability and performance go hand in hand?