Open Source Deep Learning Project: CXXNET

CXXNET is a fast, concise, distributed deep learning framework

Project Website: CXXNet now move forward to MXNet

Github Link:


CXXNET is a fast, concise, distributed deep learning framework.


Learning to use cxxnet by examples
Note on Code
User Group(TODO)
Feature Highlights

Lightweight: small but sharp knife
cxxnet contains concise implementation of state-of-art deep learning models
The project maintains a minimum dependency that makes it portable and easy to build
Scale beyond single GPU and single machine
The library works on multiple GPUs, with nearly linear speedup
THe library works distributedly backed by disrtibuted parameter server
Easy extensibility with no requirement on GPU programming
cxxnet is build on mshadow
developer can write numpy-style template expressions to extend the library only once
mshadow will generate high performance CUDA and CPU code for users
It brings concise and readable code, with performance matching hand crafted kernels
Convenient interface for other languages
Python interface for training from numpy array, and prediction/extraction to numpy array
Matlab interface

Open Source Deep Learning Project: mshadow

mshadow: Matrix Shadow

Project Website: None

Github Link:


MShadow is a lightweight CPU/GPU Matrix/Tensor Template Library in C++/CUDA. The goal of mshadow is to support efficient, device invariant and simple tensor library for machine learning project that aims for maximum performance and control, while also emphasize simplicty.

MShadow also provides interface that allows writing Multi-GPU and distributed deep learning programs in an easy and unified way.


Efficient: all the expression you write will be lazily evaluated and compiled into optimized code
No temporal memory allocation will happen for expression you write
mshadow will generate specific kernel for every expression you write in compile time.
Device invariant: you can write one code and it will run on both CPU and GPU
Simple: mshadow allows you to write machine learning code using expressions.
Whitebox: put a float* into the Tensor struct and take the benefit of the package, no memory allocation is happened unless explicitly called
Lightweight library: light amount of code to support frequently used functions in machine learning
Extendable: user can write simple functions that plugs into mshadow and run on GPU/CPU, no experience in CUDA is required.
MultiGPU and Distributed ML: mshadow-ps interface allows user to write efficient MultiGPU and distributed programs in an unified way.

Open Source Deep Learning Project: deepmat


Project Website: None

Github Link:


= Generative Stochastic Network =

A simple implementation of GSN according to (Bengio et al., 2013)
= Convolutional Neural Network =

A naive implementation (purely using Matlab)
Pooling: max (Jonathan Masci’s code) and average
Not for serious use!
= Restricted Boltzmann Machine & Deep Belief Networks =

Binary/Gaussian Visible Units + Binary Hidden Units
Enhanced Gradient, Adaptive Learning Rate
Adadelta for RBM
Contrastive Divergence
(Fast) Persistent Contrastive Divergence
Parallel Tempering
DBN: Up-down Learning Algorithm
= Deep Boltzmann Machine =

Binary/Gaussian Visible Units + Binary Hidden Units
(Persistent) Contrastive Divergence
Enhanced Gradient, Adaptive Learning Rate
Two-stage Pretraining Algorithm (example)
Centering Trick (fixed center variables only)
= Denoising Autoencoder (Tied Weights) =

Binary/Gaussian Visible Units + Binary(Sigmoid)/Gaussian Hidden Units
tanh/sigm/relu nonlinearities
Shallow: sparsity, contractive, soft-sparsity (log-cosh) regularization
Deep: stochastic backprop
Adagrad, Adadelta
= Multi-layer Perceptron =

Stochastic Backpropagation, Dropout
tanh/sigm/relu nonlinearities
Adagrad, Adadelta
Balanced minibatches using crossvalind()

Open Source Deep Learning Project: Deeplearning4J

Deeplearning4J: Neural Net Platform

Project Website:

Github Link:


Deeplearning4j is the first commercial-grade, open-source, distributed deep-learning library written for Java and Scala. Integrated with Hadoop and Spark, DL4J is designed to be used in business environments on distributed GPUs and CPUs. Skymind is its commercial support arm.

Deeplearning4J is an Apache 2.0-licensed, open-source, distributed neural net library written in Java and Scala.

Deeplearning4J integrates with Hadoop and Spark and runs on several backends that enable use of CPUs and GPUs. The aim is to create a plug-and-play solution that is more convention than configuration, and which allows for fast prototyping.

The most recent stable release in Maven Central is 0.4-rc3.9, and the current master is 0.4-rc3.10-SNAPSHOT. For more on working with snapshots, see this page.

Open Source Deep Learning Project: RNNLIB


Project Website:

Github Link: None


RNNLIB is a recurrent neural network library for sequence learning problems. Applicable to most types of spatiotemporal data, it has proven particularly effective for speech and handwriting recognition.

full installation and usage instructions given at

Multidimensional recurrent neural networks
Connectionist temporal classification
Adaptive weight noise (stochastic variational inference)

Open Source Deep Learning Project: RNNLM

RNNLM – Tomas Mikolov’s Recurrent Neural Network based Language models Toolkit

Project Website:

Github Link: None


RNNLM Toolkit

Neural network based language models are nowdays among the most successful techniques for statistical language modeling. They can be easily applied in wide range of tasks, including automatic speech recognition and machine translation, and provide significant improvements over classic backoff n-gram models. The ‘rnnlm’ toolkit can be used to train, evaluate and use such models.

The goal of this toolkit is to speed up research progress in the language modeling field. First, by providing useful implementation that can demonstrate some of the principles. Second, for the empirical experiments when used in speech recognition and other applications. And finally third, by providing a strong state of the art baseline results, to which future research that aims to “beat state of the art techniques” should compare to.

Open Source Deep Learning Project: DeepLearnToolbox

DeepLearnToolbox: Matlab/Octave toolbox for deep learning

Project Website: None

Github Link:


Matlab/Octave toolbox for deep learning. Includes Deep Belief Nets, Stacked Autoencoders, Convolutional Neural Nets, Convolutional Autoencoders and vanilla Neural Nets. Each method has examples to get you started.

A Matlab toolbox for Deep Learning.

Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data. It is inspired by the human brain’s apparent deep (layered, hierarchical) architecture. A good overview of the theory of Deep Learning theory is Learning Deep Architectures for AI

For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.

The Next Generation of Neural Networks (Hinton, 2007)
Recent Developments in Deep Learning (Hinton, 2010)
Unsupervised Feature Learning and Deep Learning (Ng, 2011)
If you use this toolbox in your research please cite Prediction as a candidate for learning deep hierarchical models of data

author = “R. B. Palm”,
title = “Prediction as a candidate for learning deep hierarchical models of data”,
year = “2012”,
Contact: rasmusbergpalm at gmail dot com

Open Source Deep Learning Project: Chainer

Chainer: A Powerful, Flexible, and Intuitive Framework of Neural Networks

Project Website:

Github Link:


Chainer is a flexible framework for neural networks. One major goal is flexibility, so it must enable us to write complex architectures simply and intuitively.

Most existing deep learning frameworks are based on the “Define-and-Run” scheme. That is, first a network is defined and fixed, and then the user periodically feeds it with mini-batches. Since the network is statically defined before any forward/backward computation, all the logic must be embedded into the network architecture as data. Consequently, defining a network architecture in such systems (e.g. Caffe) follows a declarative approach. Note that one can still produce such a static network definition using imperative languages (e.g. Torch7 and Theano-based frameworks).

In contrast, Chainer adopts a “Define-by-Run” scheme, i.e., the network is defined on-the-fly via the actual forward computation. More precisely, Chainer stores the history of computation instead of programming logic. This strategy enables to fully leverage the power of programming logic in Python. For example, Chainer does not need any magic to introduce conditionals and loops into the network definitions. The Define-by-Run scheme is the core concept of Chainer. We will show in this tutorial how to define networks dynamically.

Open Source Deep Learning Project: Keras

Keras: Deep Learning library for Theano and TensorFlow

Project Website:

Github Link:


Keras is a minimalist, highly modular neural networks library, written in Python and capable of running on top of either TensorFlow or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.

Use Keras if you need a deep learning library that:

allows for easy and fast prototyping (through total modularity, minimalism, and extensibility).
supports both convolutional networks and recurrent networks, as well as combinations of the two.
supports arbitrary connectivity schemes (including multi-input and multi-output training).
runs seamlessly on CPU and GPU.
Read the documentation at

Keras is compatible with: Python 2.7-3.5.

Open Source Text Processing Project: CMU Flite

CMU Flite: a small, fast run time synthesis engine

Project Website:
Github Link: None


CMU Flite (festival-lite) is a small, fast run-time open source text to speech synthesis engine developed at CMU and primarily designed for small embedded machines and/or large servers. Flite is designed as an alternative text to speech synthesis engine to Festival for voices built using the FestVox suite of voice building tools.
CMU Flite 2.0.0-release is now released as open source. Flite offers:

C only, very portable source
Thread safe (really this time)
Multi-voice, multi-language
Support for converting the latest Clustergen Random Forest voices for CMU Flite
Demos in English and Indic
New in this version

Loadable voices (13 US English, in various accents)
Android support
INDIC language support ( Hindi, Tamil, Telugu and Marathi)
Here are slides about Flite from an early talk given by AWB slides.pdf

Here is a recent publication at the 4th ISCA Speech Synthesis Workshop by Alan W Black and Kevin A. Lenzo html, postscript