分享丨深度学习理论与架构最新进展综述论文，66页pdf，333篇参考文献

转自 专知 

【导读】本文章从深度神经网络(DNN)入手，对深度学习(DL)领域的研究进展进行了简要的综述。内容包括：卷积神经网络(CNN)、循环神经网络(RNN)、长时记忆(LSTM)和门控递归单元(GRU)、自动编码器(AE)、深度信念网络(DBN)、生成对抗性网络(GAN)和深度强化学习(DRL)。

近年来，深度学习在各个应用领域都取得了巨大的成功。这个机器学习的新领域发展迅速，已经应用于大多数传统的应用领域，以及一些提供更多机会的新领域。针对不同类型的学习，提出了不同的学习方法，包括监督学习、半监督学习和非监督学习。

实验结果表明，与传统机器学习方法相比，深度学习在图像处理、计算机视觉、语音识别、机器翻译、艺术、医学成像、医学信息处理、机器人与控制、生物信息学、自然语言处理、网络安全等领域具有最先进的性能。 

本研究从深度神经网络(DNN)入手，对深度学习(DL)领域的研究进展进行了简要的综述。研究内容包括：卷积神经网络(CNN)、循环神经网络(RNN)、长时记忆(LSTM)和门控递归单元(GRU)、自动编码器(AE)、深度信念网络(DBN)、生成对抗性网络(GAN)和深度强化学习(DRL)。

此外，我们还讨论了最近的发展，例如基于这些DL方法的高级变体DL技术。这项工作考虑了2012年以后发表的大部分论文，当时深度学习的历史开始了。此外，本文中还包括了在不同应用领域探索和评价的DL方法。我们还包括最近开发的框架、SDKs和基准数据集，用于实施和评估深度学习方法。目前有一些研究已经发表，例如使用神经网络和一个关于强化学习(RL)的综述。然而，这些论文还没有讨论大规模深度学习模型的个别高级训练技术和最近发展起来的生成模型的方法。

关键词：卷积神经网络(CNN)；循环神经网络(RNN)；自动编码器(AE)；受限Boltzmann机器(RBM)；深度信念网络(DBN)；生成对抗性网络(GAN)；深度强化学习(DRL)；迁移学习。

• Tensorflow: https://www.tensorflow.org/
• Caffe: http://caffe.berkeleyvision.org/
• KERAS: https://keras.io/
• Theano: http://deeplearning.net/software/theano/
• Torch: http://torch.ch/
• PyTorch: http://pytorch.org/
• Lasagne: https://lasagne.readthedocs.io/en/latest/
• DL4J (DeepLearning4J): https://deeplearning4j.org/
• Chainer: http://chainer.org/
• DIGITS: https://developer.nvidia.com/digits
• CNTK (Microsoft): https://github.com/Microsoft/CNTK
• MatConvNet: http://www.vlfeat.org/matconvnet/
• MINERVA: https://github.com/dmlc/minerva
• MXNET: https://github.com/dmlc/mxnet
• OpenDeep: http://www.opendeep.org/
• PuRine: https://github.com/purine/purine2
• PyLerarn2: http://deeplearning.net/software/pylearn2/
• TensorLayer: https://github.com/zsdonghao/tensorlayer
• LBANN: https://github.com/LLNL/lbann

• cuDNN: https://developer.nvidia.com/cudnn
• TensorRT: https://developer.nvidia.com/tensorrt
• DeepStreamSDK: https://developer.nvidia.com/deepstream-sdk
• cuBLAS: https://developer.nvidia.com/cublas
• cuSPARSE: http://docs.nvidia.com/cuda/cusparse/
• NCCL: https://devblogs.nvidia.com/parallelforall/fast-multi-gpu-collectives-nccl/

A.3.1. Image Classification or Detection or Segmentation
List of datasets are used in the field of image processing and computer vision:
• MNIST: http://yann.lecun.com/exdb/mnist/
• CIFAR 10/100: https://www.cs.toronto.edu/~{}kriz/cifar.html
• SVHN/ SVHN2: http://ufldl.stanford.edu/housenumbers/

• CalTech 101/256: http://www.vision.caltech.edu/Image_Datasets/Caltech101/

• STL-10:https://cs.stanford.edu/~{}acoates/stl10/
• NORB: http://www.cs.nyu.edu/~{}ylclab/data/norb-v1.0/
• SUN-dataset: http://groups.csail.mit.edu/vision/SUN/
• ImageNet: http://www.image-net.org/
• National Data Science Bowl Competition: http://www.datasciencebowl.com/
• COIL 20/100: http://www.cs.columbia.edu/CAVE/software/softlib/coil-20.php
• MS COCO DATASET: http://mscoco.org/
• MIT-67 scene dataset: http://web.mit.edu/torralba/www/indoor.html
• Caltech-UCSD Birds-200 dataset: http://www.vision.caltech.edu/visipedia/CUB-200-2011.html
• Pascal VOC 2007 dataset: http://host.robots.ox.ac.uk/pascal/VOC/voc2007/
• H3D Human Attributes dataset: https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/
shape/poselets/
• Face recognition dataset: http://vis-www.cs.umass.edu/lfw/
• For more data-set visit: https://www.kaggle.com/
• http://homepages.inf.ed.ac.uk/rbf/CVonline/Imagedbase.htm
• Recently Introduced Datasets in Sept. 2016:
• Google Open Images (~9M images)—https://github.com/openimages/dataset
• Youtube-8M (8M videos: https://research.google.com/youtube8m/

A.3.2. Text Classification
• Reuters-21578 Text Categorization Collection: http://kdd.ics.uci.edu/databases/reuters21578/
reuters21578.html
• Sentiment analysis from Stanford: http://ai.stanford.edu/~{}amaas/data/sentiment/
• Movie sentiment analysis from Cornel: http://www.cs.cornell.edu/people/pabo/movie-review-data/
A.3.3. Language Modeling
• Free eBooks: https://www.gutenberg.org/
• Brown and stanford corpus on present americal english: https://en.wikipedia.org/wiki/Brown_
Corpus
• Google 1Billion word corpus: https://github.com/ciprian-chelba/1-billion-word-languagemodeling-benchmark
A.3.4. Image Captioning
Flickr-8k: http://nlp.cs.illinois.edu/HockenmaierGroup/8k-pictures.html
• Flickr-30k
• Common Objects in Context (COCO): http://cocodataset.org/#overview, http://sidgan.me/
technical/2016/01/09/Exploring-Datasets
A.3.5. Machine Translation
• Pairs of sentences in English and French: https://www.isi.edu/natural-language/download/
hansard/
• European Parliament Proceedings parallel Corpus 196-2011: http://www.statmt.org/europarl/
• The statistics for machine translation: http://www.statmt.org/
A.3.6. Question Answering
• Stanford Question Answering Dataset (SQuAD): https://rajpurkar.github.io/SQuAD-explorer/

• Dataset from DeepMind: https://github.com/deepmind/rc-data
• Amazon dataset: http://jmcauley.ucsd.edu/data/amazon/qa/, http://trec.nist.gov/data/qamain..., http://www.ark.cs.cmu.edu/QA-data/, http://webscope.sandbox.yahoo.co...,
http://blog.stackoverflow.com/20..

A.3.7. Speech Recognition
• TIMIT: https://catalog.ldc.upenn.edu/LDC93S1
• Voxforge: http://voxforge.org/
• Open Speech and Language Resources: http://www.openslr.org/12/
A.3.8. Document Summarization
• https://archive.ics.uci.edu/ml/datasets/Legal+Case+Reports
• http://www-nlpir.nist.gov/related_projects/tipster_summac/cmp_lg.html
• https://catalog.ldc.upenn.edu/LDC2002T31
A.3.9. Sentiment Analysis:
• IMDB dataset: http://www.imdb.com/
A.3.10. Hyperspectral Image Analysis
•http://www.ehu.eus/ccwintco/index.php/Hyperspectral_Remote_Sensing_Scenes
• https://engineering.purdue.edu/~{}biehl/MultiSpec/hyperspectral.html
• http://www2.isprs.org/commissions/comm3/wg4/HyRANK.html

A.4.1. Conferences
• Neural Information Processing System (NIPS)
• International Conference on Learning Representation (ICLR): What are you doing for Deep Learning?
• International Conference on Machine Learning (ICML)
• Computer Vision and Pattern Recognition (CVPR): What are you doing with Deep Learning?
• International Conference on Computer Vision (ICCV)
• European Conference on Computer Vision (ECCV)
• British Machine Vision Conference (BMVC)
A.4.2. Journal
• Journal of Machine Learning Research (JMLR)
• IEEE Transaction of Neural Network and Learning System (ITNNLS)
• IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
• Computer Vision and Image Understanding (CVIU)
• Pattern Recognition Letter

• Neural Computing and Application
• International Journal of Computer Vision
• IEEE Transactions on Image Processing
• IEEE Computational Intelligence Magazine
• Proceedings of IEEE
• IEEE Signal Processing Magazine
• Neural Processing Letter
• Pattern Recognition
• Neural Networks
• ISPPRS Journal of Photogrammetry and Remote Sensing

A.4.3. Tutorials on Deep Learning
• http://deeplearning.net/tutorial/
• http://deeplearning.stanford.edu/tutorial/
• http://deeplearning.net/tutorial/deeplearning.pdf
• Courses on Reinforcement Learning: http://rll.berkeley.edu/deeprlcourse/
A.4.4. Books on Deep Learning• https://github.com/HFTrader/DeepLearningBook
• https://github.com/janishar/mit-deep-learning-book-pdf
• http://www.deeplearningbook.org/