#68 DR. WALID SABA 2.0 - Natural Language Understanding [UNPLUGGED]
Patreon: https://www.patreon.com/mlst Discord: https://discord.gg/HNnAwSduud YT version: https://youtu.be/pMtk-iUaEuQ Dr. Walid Saba is an old-school polymath. He has a background in cognitive psychology, linguistics, philosophy, computer science and logic and he’s is now a Senior Scientist at Sorcero. Walid is perhaps the most outspoken critic of BERTOLOGY, which is to say trying to solve the problem of natural language understanding with application of large statistical language models. Walid thinks this approach is cursed to failure because it’s analogous to memorising infinity with a large hashtable. Walid thinks that the various appeals to infinity by some deep learning researchers are risible. [00:00:00] MLST Housekeeping [00:08:03] Dr. Walid Saba Intro [00:11:56] AI Cannot Ignore Symbolic Logic, and Here’s Why [00:23:39] Main show - Proposition: Statistical learning doesn't work [01:04:44] Discovering a sorting algorithm bottom-up is hard [01:17:36] The axioms of nature (universal cognitive templates) [01:31:06] MLPs are locality sensitive hashing tables References; The Missing Text Phenomenon, Again: the case of Compound Nominals https://ontologik.medium.com/the-missing-text-phenomenon-again-the-case-of-compound-nominals-abb6ece3e205 A Spline Theory of Deep Networks https://proceedings.mlr.press/v80/balestriero18b/balestriero18b.pdf The Defeat of the Winograd Schema Challenge https://arxiv.org/pdf/2201.02387.pdf Impact of Pretraining Term Frequencies on Few-Shot Reasoning https://twitter.com/yasaman_razeghi/status/1495112604854882304?s=21 https://arxiv.org/abs/2202.07206 AI Cannot Ignore Symbolic Logic, and Here’s Why https://medium.com/ontologik/ai-cannot-ignore-symbolic-logic-and-heres-why-1f896713525b Learnability can be undecidable http://gtts.ehu.es/German/Docencia/1819/AC/extras/s42256-018-0002-3.pdf Scaling Language Models: Methods, Analysis & Insights from Training Gopher https://arxiv.org/pdf/2112.11446.pdf DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning https://arxiv.org/abs/2006.08381 On the Measure of Intelligence [Chollet] https://arxiv.org/abs/1911.01547 A Formal Theory of Commonsense Psychology: How People Think People Think https://www.amazon.co.uk/Formal-Theory-Commonsense-Psychology-People/dp/1107151007 Continuum hypothesis https://en.wikipedia.org/wiki/Continuum_hypothesis Gödel numbering + completness theorems https://en.wikipedia.org/wiki/G%C3%B6del_numbering https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theorems Concepts: Where Cognitive Science Went Wrong [Jerry A. Fodor] https://oxford.universitypressscholarship.com/view/10.1093/0198236360.001.0001/acprof-9780198236368