What if the foundation models we use every day are secretly learning the causal structure of the world? We show that transformers trained for prediction are inherently causal learners — their gradient sensitivities reveal the true cause-and-effect relationships in data, without any explicit causal training objectives.
The big picture
Causal discovery — figuring out what causes what — is fundamental to science. Traditional methods require specialized algorithms with strong assumptions. Meanwhile, transformers have become the backbone of modern AI, excelling at prediction tasks across domains.
Our key insight: these two worlds are deeply connected. When a transformer learns to predict the future from the past, it must implicitly learn which past variables actually matter for each prediction. This is exactly what causal discovery aims to find.
From prediction to causation
Consider a p-variate time series X_t = (X_1,t, …, X_p,t) and a lag window L ≥ 1. Each variable follows X_i,t = f_i(Pa(i,t), U_t, N_i,t), where Pa(i,t) are the lagged parents, U_t are unobserved processes and N_i,t are mutually independent noises.
- A1 — Conditional exogeneity. Latent confounders are allowed as long as they don’t create spurious dependencies.
- A2 — No instantaneous effects. All parents occur at lags ℓ ≥ 1.
- A3 — Lag-window coverage. The chosen L includes all true parents.
- A4 — Faithfulness. The distribution is faithful to the causal graph.
Theorem · causal identifiability via prediction. Under A1–A4 and regularity conditions, the lagged causal graph G* is uniquely identifiable via the score gradient energy: edge j → i at lag ℓ exists if and only if
H^ℓ_j,i := 𝔼[(∂_x_j,t−ℓ log p(X_i,t | X_<t))²] > 0.
Unlike classical Granger causality, which tests mean prediction, the score gradient energy captures influence on the entire conditional distribution. Gradient sensitivity = causal relevance.
Why transformers are natural causal learners
Alignment with identifiability. Causal masking enforces temporal precedence (A2); the window L bounds maximum lag (A3); autoregressive training naturally fits conditional distributions.
Scalable sparsity. Finite capacity and weight decay compress observations into generalizable parameters; softmax attention induces competitive selection among candidates; multi-head context supports selecting complementary parents.
Contextual parameters. Attention matrices are input-conditioned, adapting to heterogeneity and non-stationarity — different contexts induce distinct dependency patterns, enabling a mixture-of-graphs view.
Gradient-based extraction. Layer-wise Relevance Propagation computes how much each past variable contributes to each prediction; we aggregate and threshold to recover the causal graph.
Results
We evaluate decoder-only transformers against PCMCI, DYNOTEARS, VAR-LiNGAM, NTS-NOTEARS, TCDF and Granger causality tests. The transformer recovers lagged parents accurately and consistently across settings, reaching 0.62 average F1 — nearly doubling the best baseline (DYNOTEARS at 0.37) — and stays robust as dynamics and dimension grow, without sensitive hyperparameter tuning.
Additional analysis
To validate that gradient attributions truly capture causal relationships, we compare them with intervention-based effects — the gold standard in causal inference — by intervening on inputs by one standard deviation and measuring the average effect on outputs. The strong correlation confirms that the transformer’s internal representation genuinely captures causal structure, not just statistical association.
We also ask whether existing time-series foundation models can discover structure zero-shot. Forecasting accuracy is reasonable but structure recovery is suboptimal; finetuning on domain-specific data significantly improves both, and adding node embeddings — even randomly initialized — helps the model distinguish variables.
Implications
For causal discovery. A new paradigm: instead of hand-crafting discovery algorithms, leverage the representation learning of foundation models. The transformer becomes a universal causal structure extractor that scales with data.
For foundation models. Gradient attributions reveal learned dependencies (interpretability); hallucinations may arise when insufficient data prevents accurate structure learning; causal priors such as sparsity and modularity could guide better architectures.
