Why does' correlation 'not equal' causality '
The sales of ice cream increase in summer, and the number of drowning people also increases. The two are highly correlated, but obviously it is not ice cream that causes drowning - the real common cause is temperature. The situation where the third variable affects both simultaneously is called a confounding factor. It reminds us that statistical correlations may come from coincidences, common causes, or reverse causality.
Even more counterintuitive is Simpson's Paradox: conclusions presented in the overall data may be completely opposite after grouping. If all samples are mixed together without distinction, the model is prone to making incorrect or even harmful judgments.
Two mainstream frameworks: latent outcomes and structural causal models
There are two main technical routes for modern causal inference. The first one is the Rubin Causal Model, which defines the causal effect of each individual as the difference between the "outcome when receiving treatment" and the "outcome when not receiving treatment". The problem is that we can only observe one outcome, and the other is counterfactual and unobservable - the core difficulty of causal inference lies in this.
The second one is the Structural Causal Model (SCM), developed by Judea Pearl et al. It uses a directed acyclic graph (DAG) to describe the causal relationships between variables and introduces the do operator: do (X=x) represents the intervention of "forcing X to be x". So there is a key distinction - the observation probability P (Y | X) is not the same as the intervention probability P (Y | do (X)): the former is "seeing", and the latter is "doing". To support decision-making, AI must know both.
Common methods for estimating causal effects
In practice, there is a complete toolbox for estimating causal effects:
Randomized controlled trials (RCTs): Randomized grouping eliminates confounding at the source and is considered the gold standard, but it is costly and sometimes unethical or impractical.
Double Difference in Differences (DID): Comparing the differences in changes before and after intervention, between the treatment group and the control group, commonly used for policy evaluation.
Propensity score matching (PSM): Find the "most similar" control sample for each processed sample and simulate randomization.
Instrumental variables (IV) and breakpoint regression (RDD): using exogenous shocks or threshold rules to separate confounding.
How to integrate causal inference with AI/machine learning
Traditional machine learning excels at fitting P (Y | X): given input, predict output. But it naturally does not answer 'what would happen if I changed X', so it is prone to failure in situations where decision-making is needed. Causal inference perfectly complements this puzzle:
Causal discovery: Learning causal structures between variables from data, not just related networks.
• Skewness and robustness: Introducing a causal perspective to alleviate the model's dependence on false correlations and enhance out of distribution (OOD) generalization ability.
Decision making and evaluation: In scenarios such as recommendation, healthcare, financial risk control, and personalized pricing, make decisions based on causal effects rather than correlations to avoid errors caused by using correlations as causality.
Combining with large models: Empowering large models with causal reasoning capabilities (such as causal question answering and counterfactual reasoning) is currently one of the cutting-edge directions.
one-sentence summary
Causal inference is not meant to replace machine learning, but to add the necessary element of "decision-making" to predictive models: knowing "what to do will bring about what changes" is often more important than knowing "what will happen next".
[Reference source] Comprehensive compilation of industry information released publicly, involving Judea Pearl's structural causal model and do calculus, Rubin's latent result framework, Simpson's paradox, and other publicly available classic theories and materials.