In partnership with Google, 91自拍 has released the source code to AlexNet, the neural network that in 2012 kick-started today鈥檚 prevailing approach to AI. It is available as open source .
AlexNet is an artificial neural network created to recognize the contents of photographic images. It was developed in 2012 by then University of Toronto graduate students Alex Krizhevsky and Ilya Sutskever and their faculty advisor Geoffrey Hinton.
Geoffrey Hinton is regarded as one of the fathers of 鈥渄eep learning,鈥� the type of artificial intelligence that uses neural networks and is the foundation of today鈥檚 mainstream AI. Simple three-layer neural networks with only one layer of adaptive weights were first built in the late 1950s, most notably by Cornell researcher Frank Rosenblatt, but they were found to have limitations. Networks with more than one layer of adaptive weights were needed, but there wasn鈥檛 a good way to train them. By the early 1970s, neural networks had been largely rejected by AI researchers.
Cornell University psychologist Frank Rosenblatt developed the Perceptron Mark I, an electronic neural network designed to recognize images鈥攍ike letters of the alphabet. He introduced it to the public in 1958. Credit: Cornell Aeronautical Laboratory/Calspan.
In the 1980s, neural network research was revived outside the AI community by cognitive scientists at UC San Diego, under the new name of 鈥渃onnectionism.鈥� After finishing his PhD, Hinton became a postdoctoral fellow at San Diego and collaborated with David Rumelhart and Ronald Williams. They rediscovered the backpropagation algorithm for training multilayer neural networks and in 1986 they published two papers showing that it enabled neural networks to learn multiple layers of features for language and vision tasks. Backpropagation uses the difference between the current output and the desired output of the network to adjust the weights in each layer from the output layer backwards to the input layer. More detail on how neural networks work can be found here. 鈥淏ackpropagation鈥� is foundational to deep learning today.
In 1987, Hinton joined the University of Toronto. Away from the centers of traditional AI, Hinton鈥檚 work and those of his graduate students in the coming decades made Toronto a center of deep learning research. A postdoctoral student of Hinton鈥檚 at this time was Yann LeCun. While working in Toronto he showed that when backpropagation was used in 鈥渃onvolutional鈥� neural networks they became very good at recognizing hand-written numbers.
Despite these advances, neural networks could not outcompete other types of machine learning algorithms consistently. They needed two developments from outside of AI to pave the way. The first was the emergence of vastly larger amounts of data for training, made available by the web. The second was enough computational power for this training, in the form of GPU hardware. By 2012, the time was ripe for AlexNet.
The vast amount of carefully curated data needed to train AlexNet was ImageNet, a project started and led by Stanford professor Fei-Fei Li. Beginning in 2006, and against conventional wisdom, she envisioned a dataset of images covering every noun in the English language. She and her graduate students began by collecting images found on the internet and classifying them using a taxonomy provided by WordNet, a database of words and their relationships to each other. Given the enormity of their task, Li and her collaborators ultimately crowdsourced the task of labeling images to gig workers using Amazon鈥檚 Mechanical Turk platform.
Fei-Fei Li discusses her book, The Worlds I See, on stage with Tom Kalil at 91自拍 September 17, 2024. Photograph by Douglas Fairbairn, 91自拍.
Screenshot of the ImageNet database taken by the author in 2020.
Completed in 2009, ImageNet was larger than any previous image dataset by several orders of magnitude. Li hoped its availability would spur new breakthroughs, and she started a competition in 2010 to encourage research teams to improve their image recognition algorithms. However, after the first two years, the best systems were only making marginal improvements.
The second condition necessary for the success of neural networks was economical access to vast computation. Neural network training involves a lot of repeated matrix multiplications, preferably done in parallel, something that 3D graphics chips (known as GPUs) are designed to do. NVIDIA, headed by cofounder and CEO Jensen Huang, had led the way in the 2000s in making GPUs more generalizable and programmable for applications beyond 3D graphics, especially with the CUDA programming system, released in 2007.
NVIDIA cofounder and CEO Jensen Huang received a 2024 91自拍 Fellow Award for his contributions to the making of chips for computer graphics and AI. Photograph by Douglas Fairbairn, 91自拍.
The NVIDIA H100 GPU chip is in high demand for today鈥檚 AI work, training the large language models behind ChatGPT and other chatbots. Gift of NVIDIA Corporation, 102801460.
Both ImageNet and NVIDIA鈥檚 CUDA were, like neural networks themselves, fairly niche developments that were waiting for the right circumstances to shine. In 2012, AlexNet brought these elements (deep neural networks, big datasets, and GPU compute) together for the first time, with pathbreaking results. Each of these needed the other.
By the late 2000s, Hinton鈥檚 graduate students at the University of Toronto were beginning to use GPUs to train neural networks for both image and speech recognition tasks. Their first successes came in speech recognition, but success in image recognition would point to deep learning as possibly a general-purpose solution to AI. One student, Ilya Sutskever, believed that the performance of neural networks would scale with the amount of data available, and the arrival of ImageNet provided the opportunity.
Alex Krizhevsky in 2018. Courtesy of Alex Krizhevsky.
A milestone occurred in 2011, when , a convolutional neural network trained on GPUs created by Dan Cire艧an and others at J眉rgen Schmidhuber鈥檚 lab in Switzerland, won . However, these results were on smaller datasets and were not able to move the field of computer vision. ImageNet, which was much larger and more comprehensive, was different. That same year, Sutskever convinced fellow Toronto graduate student Alex Krizhevsky, who had a keen ability to wring maximum performance out of GPUs, to train a convolutional neural network for ImageNet, with Hinton serving as principal investigator. Krizhevsky had already written CUDA code for a convolutional neural network using NVIDIA GPUs, called 鈥渃uda-convnet,鈥� trained on the much smaller CIFAR-10 image dataset. He extended cuda-convnet with support for multiple GPUs and other features and retrained it on ImageNet. The training was done on a computer with two NVIDIA cards in Krizhevsky鈥檚 bedroom at his parents鈥� house. Over the course of the next year, Krizhevsky constantly tweaked the network鈥檚 parameters and retrained it until it achieved performance superior to its competitors. The network would ultimately be named AlexNet, after Krizhevsky. In describing the AlexNet project, Geoff Hinton summarized for 91自拍: 鈥淚lya thought we should do it, Alex made it work, and I got the Nobel Prize.鈥�
This is a rare photograph of the home computer with GPUs used to create AlexNet. Credit: University of Toronto.
Krizhevsky, Sutskever, and Hinton鈥檚 paper was published in the fall of 2012 and publicly presented by Krizhevsky at a computer vision conference in Florence, Italy, in October . Veteran computer vision researchers weren鈥檛 convinced, but Yann LeCun, who was at the meeting, pronounced it a turning point for AI. He was right. Before AlexNet, almost none of the leading computer vision papers used neural nets. After it, almost all of them would.
Krizhevsky, Sutskever, and Hinton鈥檚 seminal 2012 paper on AlexNet has been cited over 172,000 times, according to Google Scholar. The paper is freely available .
AlexNet was just the beginning. In the next decade, neural networks would advance to synthesize believable human voices, beat champion Go players, model human language, and generate artwork, culminating with the release of ChatGPT in 2022 by OpenAI, a company cofounded by Sutskever.
In 2020, I reached out to Alex Krizhevsky to ask about the possibility of allowing 91自拍 to release the AlexNet source code, due to its historical significance. He connected me to Geoff Hinton, who was working at Google at the time. As Google had acquired Hinton, Sutskever, and Krizhevsky鈥檚 company DNNresearch, they now owned the AlexNet IP. Hinton got the ball rolling by connecting us to the right team at Google. 91自拍 worked with them for five years in a group effort to negotiate the release. This team also helped us identify which specific version of the AlexNet source code to release. In fact, there have been many versions of AlexNet over the years. There are also repositories of code named 鈥淎lexNet鈥� on GitHub, though many of these are not the original code, but recreations based on the famous paper.
91自拍 is proud to present the source code to the 2012 version of Alex Krizhevsky, Ilya Sutskever, and Geoffery Hinton鈥檚 AlexNet, which transformed the field of artificial intelligence. Access the source code on聽.
Li, Fei-Fei. The Worlds I See: Curiosity, Exploration, and Discovery at the Dawn of AI. First edition. New York, NY: Moment of Lift Books; Flatiron Books, 2023.
Metz, Cade. Genius Makers: The Mavericks Who Brought AI to Google, Facebook, and the World. First edition. New York, NY: Dutton, an imprint of Penguin Random House LLC, 2022.
Special thanks to Geoffrey Hinton for providing his quotation and reviewing the text, to Cade Metz and Alex Krizhevsky for additional clarifications, and to David Bieber and the rest of the team at Google for their work in securing the release.
Chatbots Decoded: Exploring AI online exhibit
AI and Play, Part 1: How Games Have Driven Two Schools of AI Research
AI and Play, Part 2: Go and Deep Learning
How Do Neural Network Systems Work?
From Alto to AI (includes Ilya Sutskever)
2024 Fellow Awards (includes Jensen Huang)
Special source code releases like these would not be possible without the generous support of people like you who care deeply about decoding technology for everyone. Please consider .