What is neural architecture search
Neural Architecture Search (NAS) refers to the use of algorithms to automatically find the "structure" of a neural network, rather than relying on manual trial and error. It often determines how many layers to stack, what operator to use for each layer (convolution, attention, or pooling), how many channels to use, and how layers are connected.
Why do we need NAS
Manually designing a high-performance network typically relies on the researcher's experience and a large amount of computational power for trial and error. NAS transforms "designing networks" from a human intuition problem to a search problem with a goal: given a task and data, let the algorithm find a set of structures that perform best on the validation set.
Three core elements
Search space: The range of candidate network structures determines what can be found.
Search strategy: How to find in space, commonly including reinforcement learning, evolutionary algorithms, and differentiable search (such as DARTS).
Performance evaluation: How to score candidate networks? Common methods include evaluating them on a validation set after training, or using proxy metrics and weight sharing to save computational power.
Classic methods and representative achievements
In 2016, Zoph and Le used reinforcement learning to "generate" network descriptions for a controller, which was a pioneering work in NAS; In 2018, DARTS relaxed the selection of discrete structures into differentiable problems, significantly reducing search costs; In 2019, EfficientNet used a composite scaling system to balance depth, width, and resolution, becoming a representative of balancing accuracy and efficiency. In addition, there are routes based on evolutionary algorithms and weight sharing (such as ENAS).
Cost and Current Situation
Early NAS often required hundreds or thousands of GPU days, which was difficult for ordinary teams to afford. Later on, ideas such as weight sharing, proxy tasks, and one shot search reduced costs, but whether the discovered structures can be stably migrated to new tasks remains a research hotspot. For most teams, directly using existing efficient architectures is often more cost-effective than searching from scratch.
Reference source
Comprehensive compilation of publicly published academic papers, including Zoph and Le (2016, Neural Architecture Search with Reinforcement Learning), Liu et al. (2018, DARTS), Tan and Le (2019, EfficientNet).