DSD:Dense-Sparse-Dense training for deep neural networks

标签：image   int   imp   from   ssi   ram   regular   dsd   show   

ICLR 2017会议论文。

摘要：

神经网络因为参数很多，所以很难训练。

Modern deep neural networks have a large number of parameters, making them very hard to train.

所以，分步骤训练参数。

We propose DSD, a dense-sparse-dense training ?ow, for regularizing deep neural networks and achieving better optimization performance. In the ?rst D (Dense) step, we train a dense network to learn connection weights and importance. In the S (Sparse) step, we regularize the network by pruning the unimportant connections with small weights and retraining the network given the sparsity constraint. In the ?nal D (re-Dense) step, we increase the model capacity by removing the sparsity constraint, re-initialize the pruned parameters from zero and retrain the whole dense network.

实验结果好。

Experiments show that DSD training can improve the performance for a wide range of CNNs, RNNs and LSTMs on the tasks of image classi?cation, caption generation and speech recognition

DSD:Dense-Sparse-Dense training for deep neural networks

标签：image   int   imp   from   ssi   ram   regular   dsd   show   

原文地址：http://www.cnblogs.com/mengmengmiaomiao/p/7652779.html