LogIn / RegistrationArticles for sale
- Head office address: Moscow, 123317, Moscow City, 8th Floor, Presnenskaya Embankment, 6, bldg. 2
- article@123mi.ru
- Сортировка:
- по горящим
- по цене
- по дате
#5849. Momentum-driven adaptive synchronization model for distributed DNN training on HPC clusters
| July 2026 | publication date |
| Proposal available till | 17-05-2025 |
| 4 total number of authors per manuscript | 0 $ |
| The title of the journal is available only for the authors who have already paid for |
| Journal’s subject area: |
| Theoretical Computer Science; Computer Networks and Communications; Hardware and Architecture; Software; Artificial Intelligence; |
| place 1 | place 2 | place 3 | place 4 |
| Free | Free | Free | Free |
| 2350 $ | 1200 $ | 1050 $ | 900 $ |
Contract №5849.1   | Contract №5849.2 | Contract №5849.3 | Contract №5849.4 |
1 place - free (for sale)
2 place - free (for sale)
3 place - free (for sale)
4 place - free (for sale)
Abstract:
Building a distributed deep learning (DDL) system on HPC clusters that guarantees convergence speed and scalability for the training of DNNs is challenging. The HPC cluster, which consists of multiple high-density multi-GPU servers connected by the Infiniband network (HDGib), compresses the computing and communication time for distributed DNNs training but brings new challenges. The convergence time is far from linear scalability (with respect to the number of workers) for parallel DNNs training. We thus analyze the optimization process and identify three key issues that cause scalability degradation. First, the high-frequency update for parameters due to the compression of the computing and communication times exacerbates the stale gradient problem, which slows down the convergence. Second, the previous methods used to constrain the gradient noise (stochastic error) of the SGD are outdated, as HDGib clusters can support more strict constraints due to the Infiniband network connections, which can further constrain the stochastic error. Third, the same learning rate for all workers is inefficient due to the different training stages of each worker. We thus propose a momentum-driven adaptive synchronization model that focuses on solving the above issues and accelerating the training procedure on HDGib clusters.
Keywords:
Adaptive; Distributed deep learning system; High-performance computing; Momentum-driven; Synchronization model
Contacts :
2 place - free (for sale)
3 place - free (for sale)
4 place - free (for sale)
Abstract:
Building a distributed deep learning (DDL) system on HPC clusters that guarantees convergence speed and scalability for the training of DNNs is challenging. The HPC cluster, which consists of multiple high-density multi-GPU servers connected by the Infiniband network (HDGib), compresses the computing and communication time for distributed DNNs training but brings new challenges. The convergence time is far from linear scalability (with respect to the number of workers) for parallel DNNs training. We thus analyze the optimization process and identify three key issues that cause scalability degradation. First, the high-frequency update for parameters due to the compression of the computing and communication times exacerbates the stale gradient problem, which slows down the convergence. Second, the previous methods used to constrain the gradient noise (stochastic error) of the SGD are outdated, as HDGib clusters can support more strict constraints due to the Infiniband network connections, which can further constrain the stochastic error. Third, the same learning rate for all workers is inefficient due to the different training stages of each worker. We thus propose a momentum-driven adaptive synchronization model that focuses on solving the above issues and accelerating the training procedure on HDGib clusters.
Keywords:
Adaptive; Distributed deep learning system; High-performance computing; Momentum-driven; Synchronization model
Contacts :
