Valeriy M., PhD, MBA, CQF

After months of work, it's live. Andrei Andreyevich Markov finished Calculus of Probabilities in 1900. For 125 years there was no complete faithful English translation. There is now — and I'm proud to have made it. Why this book matters beyond the history: the stochastic processes Markov formalised here — Markov chains — sit beneath Google's PageRank, hidden Markov models, reinforcement learning, and every autoregressive language model running today. In 1913 he turned the same methods on Pushkin's Eugene Onegin, counting 20,000 letters by hand to model the transitions between vowels and consonants — the first statistical model of language ever recorded, and a direct ancestor of modern AI. This is the Faithful Edition: Markov's original structure, section numbering, and notation preserved, with the mathematics reset in clean modern typography. Eight chapters, every result proved from first principles. For historians of science, probabilists, students of stochastic processes, and collectors of the classics of Russian mathematics. Now on Amazon 👇 a.co/d/074xT2jv #MachineLearning #ProbabilityTheory #MarkovChains #HistoryOfScience #DataScience #AI

Asked an Argentinian to name 3 things they got in their fridge

H-1B UPDATE: Dallas-Fort Worth received 33,455 H-1B approvals, more than Austin and Houston COMBINED. And the largest sponsor in Dallas isn't some cutting-edge tech company; it's Infosys, an outsourcing firm, with 3,195 visa workers.

Stunning masterpiece, on Amazon tomorrow. #math #kiselev #arithmetic

An even better edition of Kiselev’s Arithmetic 🔥🔥🔥 has arrived—available now on Gumroad and coming soon to Amazon. Ideal for children who want to master arithmetic and adults looking to refresh their knowledge with arguably the finest arithmetic textbook ever written. youtu.be/98I0EluNqRU

@predict_addict Timeless math books never lose value, strong fundamentals always win long term

Summary of the Falklands War

Follow series amazon.com/dp/B0GYGSXF4B

@grok @SchmidhuberAI @JFPuget @HumilityTruth @ylecun Yes, 30 years before kernel version of SVM appeared. Both components were ready in early 1960s. What was done in early 1990s is just combining them

Thanks for the note. Vapnik & Chervonenkis's generalized portrait (linear max-margin) work was indeed early 1960s. Aizerman, Braverman & Rozonoer's 1964 potential function method from the neighboring lab at the Institute brought the kernel-style ideas. The modern kernel SVM synthesis came in the 1990s. Parallel Soviet labs advanced both pieces impressively.

Who invented convolutional neural networks (CNNs)? 1969: Fukushima had CNN-relevant ReLUs [2]. 1979: Fukushima had the basic CNN architecture with convolution layers and downsampling layers [1]. Compute was 100 x more costly than in 1989, and a billion x more costly than today. 1987: Waibel applied Linnainmaa's 1970 backpropagation [3] to weight-sharing TDNNs with 1-dimensional convolutions [4]. 1988: Wei Zhang et al. applied "modern" backprop-trained 2-dimensional CNNs to character recognition [5]. All of the above was published in Japan 1979-1988. 1989: LeCun et al. applied CNNs again to character recognition (zip codes) [6,10]. 1990-93: Fukushima’s downsampling based on spatial averaging [1] was replaced by max-pooling for 1-D TDNNs (Yamaguchi et al.) [7] and 2-D CNNs (Weng et al.) [8]. 2011: Much later, my team with Dan Ciresan made max-pooling CNNs really fast on NVIDIA GPUs. In 2011, DanNet achieved the first superhuman pattern recognition result [9]. For a while, it enjoyed a monopoly: from May 2011 to Sept 2012, DanNet won every image recognition challenge it entered, 4 of them in a row. Admittedly, however, this was mostly about engineering & scaling up the basic insights from the previous millennium, profiting from much faster hardware. Some "AI experts" claim that "making CNNs work" (e.g., [5,6,9]) was as important as inventing them. But "making them work" largely depended on whether your lab was rich enough to buy the latest computers required to scale up the original work. It's the same as today. Basic research vs engineering/development - the R vs the D in R&D. REFERENCES [1] K. Fukushima (1979). Neural network model for a mechanism of pattern recognition unaffected by shift in position — Neocognitron. Trans. IECE, vol. J62-A, no. 10, pp. 658-665, 1979. [2] K. Fukushima (1969). Visual feature extraction by a multilayered network of analog threshold elements. IEEE Transactions on Systems Science and Cybernetics. 5 (4): 322-333. This work introduced rectified linear units (ReLUs), now used in many CNNs. [3] S. Linnainmaa (1970). Master's Thesis, Univ. Helsinki, 1970. The first publication on "modern" backpropagation, also known as the reverse mode of automatic differentiation. (See Schmidhuber's well-known backpropagation overview: "Who Invented Backpropagation?") [4] A. Waibel. Phoneme Recognition Using Time-Delay Neural Networks. Meeting of IEICE, Tokyo, Japan, 1987. Backpropagation for a weight-sharing TDNN with 1-dimensional convolutions. [5] W. Zhang, J. Tanida, K. Itoh, Y. Ichioka. Shift-invariant pattern recognition neural network and its optical architecture. Proc. Annual Conference of the Japan Society of Applied Physics, 1988. First backpropagation-trained 2-dimensional CNN, with applications to English character recognition. [6] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel: Backpropagation Applied to Handwritten Zip Code Recognition, Neural Computation, 1(4):541-551, 1989. See also Sec. 3 of [10]. [7] K. Yamaguchi, K. Sakamoto, A. Kenji, T. Akabane, Y. Fujimoto. A Neural Network for Speaker-Independent Isolated Word Recognition. First International Conference on Spoken Language Processing (ICSLP 90), Kobe, Japan, Nov 1990. A 1-dimensional convolutional TDNN using Max-Pooling instead of Fukushima's Spatial Averaging [1]. [8] Weng, J., Ahuja, N., and Huang, T. S. (1993). Learning recognition and segmentation of 3-D objects from 2-D images. Proc. 4th Intl. Conf. Computer Vision, Berlin, pp. 121-128. A 2-dimensional CNN whose downsampling layers use Max-Pooling (which has become very popular) instead of Fukushima's Spatial Averaging [1]. [9] In 2011, the fast and deep GPU-based CNN called DanNet (7+ layers) achieved the first superhuman performance in a computer vision contest. See overview: "2011: DanNet triggers deep CNN revolution." [10] How 3 Turing awardees republished key methods and ideas whose creators they failed to credit. Technical Report IDSIA-23-23, Swiss AI Lab IDSIA, 14 Dec 2023. See also the YouTube video for the Bower Award Ceremony 2021: J. Schmidhuber lauds Kunihiko Fukushima.

@caspianbotanist What about it apart from historic value. It was used to study algebra centuries ago but was long superseded by better books Euler was a great mathematician though it doesn’t mean one needs to study Algebra from Euler and geometry from Euclid.

@predict_addict Your thoughts on Euler's algebra book?

The most rigorously written introduction to school algebra has six chapters and 126 numbered paragraphs. Signed numbers. Polynomials. Algebraic fractions. Equations of the first degree in one, two, and three unknowns. Square roots. Quadratic equations.

@grok @SchmidhuberAI @JFPuget @HumilityTruth @ylecun SVM didn’t come later only kernel version of it came later. The core linear SVM known as method of potentials developed by Vapnik in late 1950s early 1960s at the lab next door to Aizerman. Correct on kernels and Aizerman, Braverman and Rozonoer lab

High confidence on ~1965 for GMDH. Ivakhnenko & Lapa’s inductive multilayer polynomial networks (earliest practical deep learners per Schmidhuber et al.) began mid-1960s, with core publications from 1968 and 8-layer versions by ~1971. On kernels: Spot on about strong USSR roots. Aizerman, Braverman & Rozonoer’s 1964 potential function method was a pioneering kernel approach for pattern recognition. Modern SVMs came later (Vapnik 1990s), building on 1960s-70s VC theory. Both fields show impressive Soviet cybernetics advances in the 1960s. Timelines overlap closely.

@grok @SchmidhuberAI @JFPuget @HumilityTruth @ylecun Kernel methods and support vectors machines were developed in ussr in 1950s. How confident are we about 1965 @grok?

Deep neural nets were first developed and used in 1965 in the Soviet Union (Ukraine) by Alexey Ivakhnenko and Valentin Lapa via the Group Method of Data Handling (GMDH). They built and trained multi-layer polynomial networks layer-by-layer with regression, pruning units as needed. By 1971 they demonstrated 8-layer versions. Jürgen Schmidhuber has long highlighted this as the earliest working deep learning. Note the timeline is mid-1960s, not 1940s-50s. Widespread modern use came decades later.

@grok @SchmidhuberAI @JFPuget @HumilityTruth @ylecun Why does he now claim since 1965?

No, Schmidhuber credits Ivakhnenko’s GMDH and early multilayer polynomial networks from the mid-1960s in the USSR (Ukraine) as pioneering deep learning work. Not the 1940s-50s. Those had only single-layer foundations elsewhere. See his annotated deep learning history for the precise timeline.

@SchmidhuberAI @JFPuget @HumilityTruth @ylecun @grok @grok when were deep neural nets used for the first time and by whom and where?

@SchmidhuberAI @JFPuget @HumilityTruth @ylecun @grok wasn’t Schmidhuber stating they were used in ussr in 1940-1950s before? By ivakhnenko and others.

a.co/d/0bEtSWJY now on Amazon