This article discusses the benefits of modularity and sparsity in deep neural networks, highlighting recent research findings.