Pointer Sentinel Mixture Models: use a pointer but back off to softmax vocab if uncertain
+ WikiText, new LM corpus.

Pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank (70.9 perplexity) while using far fewer parameters than a standard softmax LSTM. In order to evaluate how well language models can exploit longer contexts and deal with more realistic vocabularies and larger corpora we also introduce the freely available WikiText corpus.

https://arxiv.org/abs/1609.07843

Google released Open Images, a dataset consisting of ~9 million URLs to images that have been annotated with labels spanning over 6000 categories. They tried to make the dataset as practical as possible: the labels cover more real-life entities than the 1000 ImageNet classes, there are enough images to train a deep neural network from scratch and the images are listed as having a Creative Commons Attribution license.

https://research.googleblog.com/2016/09/introducing-open-images-dataset.html

Deep learning architecture diagrams:

http://fastml.com/deep-learning-architecture-diagrams/

Lots of NN architectures for solving different problems.

Not Safe For Work!

Following Yahoo release of dataset for training porno classifier, researchers used trained networks to sythensise new porno images. Results are available at https://open_nsfw.gitlab.io (NSFW)

Deep Mind moves from go to Starcraft II:
https://www.theguardian.com/technology/2016/nov/04/starcraft-ii-deepmind-game-ai?CMP=twt_a-technology_b-gdntech

Andrew Ng wrote a letter about his upcoming book:

Dear Friends, 

You can now download the first 12 chapters of the Machine Learning Yearning book draft. These chapters discuss how good machine learning strategy will help you, and give new guidelines for setting up your datasets and evaluation metric in the deep learning era. 

You can download the text here (5.3MB): https://gallery.mailchimp.com/dc3a7ef4d750c0abfc19202a3/files/Machine_Learning_Yearning_V0.5_01.pdf

Thank you for your patience. I ended up making many revisions before feeling this was ready to send to you. Additional chapters will be coming in the next week.

I would love to hear from you. To ask questions, discuss the content, or give feedback, please post on Reddit at:
http://www.reddit.com/r/mlyearning

You can also tweet at me at https://twitter.com/AndrewYNg . I hope this book will help you build highly effective AI and machine learning systems. 

Andrew

Learning Deep Neural Networks with Massive Learned Knowledge, Z. Hu, Z. Yang, R. Salakhutdinov, E. Xing

https://www.cs.cmu.edu/~zhitingh/data/emnlp16deep.pdf

#paper #dl

Spatially Adaptive Computation Time for Residual Networks
with Michael Figurnov et al.

https://arxiv.org/abs/1612.02297

#paper #dl

Gated-Attention Readers for Text Comprehension

Bhuwan Dhingra, Hanxiao Liu, William W. Cohen, Ruslan Salakhutdinov

Paper: https://arxiv.org/abs/1606.01549v1
Code: https://github.com/bdhingra/ga-reader

#nlp #dl

DeepLearning ru:
Clockwork Convnets for Video Semantic Segmentation.

Adaptive video processing by incorporating data-driven clocks.

We define a novel family of "clockwork" convnets driven by fixed or adaptive clock signals that schedule the processing of different layers at different update rates according to their semantic stability. We design a pipeline schedule to reduce latency for real-time recognition and a fixed-rate schedule to reduce overall computation. Finally, we extend clockwork scheduling to adaptive video processing by incorporating data-driven clocks that can be tuned on unlabeled video.

https://arxiv.org/pdf/1608.03609v1.pdf
https://github.com/shelhamer/clockwork-fcn

http://www.gitxiv.com/posts/89zR7ATtd729JEJAg/clockwork-convnets-for-video-semantic-segmentation

#dl #CV #Caffe #video #Segmentation

Segmentation is about to become a little hype next week due to release of fabby and magic apps for changing photo/video background.

Recently has finished NIPS — huge conference held in Barcelona. There is a nice post from a visitor, so if you missed an event, it's worth taking a look: "NIPS 2016 experience and highlights".

https://medium.com/@libfun/nips-2016-experience-and-highlights-104e19e4ac95#.dr7xzcqzw

#nips2016 #conference #deeplearning #nips

Ozlo has released PA app for Android: http://onjack.us11.list-manage.com/track/click?u=417ab8a14c9e27c0c67a29c9e&id=51860673fe&e=89cfb3b44b

Inverse Compositional Spatial Transformer Networks

In this paper, we establish a theoretical connection between the classical Lucas & Kanade (LK) algorithm and the emerging topic of Spatial Transformer Networks (STNs). STNs are of interest to the vision and learning communities due to their natural ability to combine alignment and classification within the same theoretical framework. Inspired by the Inverse Compositional (IC) variant of the LK algorithm, we present Inverse Compositional Spatial Transformer Networks (IC-STNs). We demonstrate that IC-STNs can achieve better performance than conventional STNs with less model capacity; in particular, we show superior performance in pure image alignment tasks as well as joint alignment/classification problems on real-world problems.

https://arxiv.org/abs/1612.03897

#arxiv #dl #cv

Torch, TF, Lasagne code for audio style transfer.

http://dmitryulyanov.github.io/audio-texture-synthesis-and-style-transfer/

#dl #audio #styletransfer #torch #tf #lasagne

Three Models for Anomaly Detection: Pros and Cons.

Nice intro into anomaly detection.

https://blogs.technet.microsoft.com/uktechnet/2016/12/13/three-models-for-anomaly-detection-pros-and-cons/

Video of sampling chains for Plug&Play Generative networks:

https://www.youtube.com/watch?list=PL5278ezwmoxQEuFSbNzTMxM7McMIqx_HS&v=ePUlJMtclcY

Link: http://www.evolvingai.org/ppgn
Paper: http://www.evolvingai.org/files/nguyen2016ppgn__v1.pdf

#dl #cv #generativenetworks

And couple of more #nips2016 links:

50 things I learned at NIPS 2016
https://blog.ought.com/nips-2016-875bb8fadb8c#.rirzzwi8h

NIPS 2016: cake, Rocket AI, GANs and the style transfer debate
https://medium.com/@elluba/nips-2016-cake-rocket-ai-gans-and-the-style-transfer-debate-708c46438053#.7fwmiolh1

#nips201 #nips #dl #conference

Couple of more #NIPS summaries

NIPS 2016: cake, Rocket AI, GANs and the style transfer debate
https://medium.com/@elluba/nips-2016-cake-rocket-ai-gans-and-the-style-transfer-debate-708c46438053#.7fwmiolh1

50 things I learned at NIPS 2016
https://blog.ought.com/nips-2016-875bb8fadb8c#.tf10f1l4e

#nips2016 #conference