The debates

Chollet says scale multiplies generalisation rather than replacing it

Francois Chollet, creator of Keras, answered Richard Sutton by arguing that more data and compute multiply the generalisation power of a method, so they cannot rescue a weak one. Yann LeCun separated joint-embedding from joint-embedding predictive architectures and defended the second against critics who point to weaker benchmarks.

Filed under Learning theory, Large language models

Chollet’s point is about ordering. Inefficient methods can look adequate when investment is enormous, but a method that generalises better gains more from each additional unit of scale, so in his view it will eventually overtake. The argument is a claim about the shape of scaling curves, not a reported measurement.

LeCun traced the loss functions of joint-embedding methods to siamese networks, Barlow Twins, VICReg and DINO, and argued that adding a predictor helps when a task is asymmetric, as in predicting the future state of a world model. Critics replied that many papers with JEPA in the title report weaker results than tuned systems such as DINO or V-JEPA. LeCun answered that research on a method and the engineering that later makes it competitive are different stages. The two threads agree that architecture still matters and disagree on how much of it can be learned from scale alone.