Open Source Deep Learning Project: CUDAMat

CUDAMat: Python module for performing basic dense linear algebra computations on the GPU using CUDA

Project Website: None

Github Link: https://github.com/cudamat/cudamat

Description

The aim of the cudamat project is to make it easy to perform basic matrix calculations on CUDA-enabled GPUs from Python. cudamat provides a Python matrix class that performs calculations on a GPU. At present, some of the operations our GPU matrix class supports include:

Easy conversion to and from instances of numpy.ndarray.
Limited slicing support.
Matrix multiplication and transpose.
Elementwise addition, subtraction, multiplication, and division.
Elementwise application of exp, log, pow, sqrt.
Summation, maximum and minimum along rows or columns.
Conversion of CUDA errors into Python exceptions.
The current feature set of cudamat is biased towards features needed for implementing some common machine learning algorithms. We have included implementations of feedforward neural networks and restricted Boltzmann machines in the examples that come with cudamat.

Example:

import numpy as np
import cudamat as cm

cm.cublas_init()

# create two random matrices and copy them to the GPU
a = cm.CUDAMatrix(np.random.rand(32, 256))
b = cm.CUDAMatrix(np.random.rand(256, 32))

# perform calculations on the GPU
c = cm.dot(a, b)
d = c.sum(axis = 0)

# copy d back to the host (CPU) and print
print(d.asarray())

Open Source Deep Learning Project: EBLearn

EBLearn: Open Source C++ Machine Learning Library

Project Website: http://eblearn.sourceforge.net/

Github Link: None

Description

Eblearn is an object-oriented C++ library that implements various machine learning models, including energy-based learning, gradient-based learning for machine composed of multiple heterogeneous modules. In particular, the library provides a complete set of tools for building, training, and running convolutional networks.

Open Source Deep Learning Project: Nengo

The Nengo Neural Simulator

Project Website: http://nengo.ca/

Github Link: None

Description

Nengo is a graphical and scripting based software package for simulating large-scale neural systems. The book How to build a brain, which includes Nengo tutorials, is now available. This website also has additional information on the book.

To use Nengo, you define groups of neurons in terms of what they represent, and then form connections between neural groups in terms of what computation should be performed on those representations. Nengo then uses the Neural Engineering Framework (NEF) to solve for the appropriate synaptic connection weights to achieve this desired computation. Nengo also supports various kinds of learning. Nengo helps make detailed spiking neuron models that implement complex high-level cognitive algorithms.

Among other things, Nengo has been used to implement motor control, visual attention, serial recall, action selection, working memory, attractor networks, inductive reasoning, path integration, and planning with problem solving (see the model archives and publications for details).

Open Source Deep Learning Project: CXXNET

CXXNET is a fast, concise, distributed deep learning framework

Project Website: CXXNet now move forward to MXNet

Github Link: https://github.com/dmlc/cxxnet

Description

CXXNET is a fast, concise, distributed deep learning framework.

Contributors: https://github.com/antinucleon/cxxnet/graphs/contributors

Documentation
Learning to use cxxnet by examples
Note on Code
User Group(TODO)
Feature Highlights

Lightweight: small but sharp knife
cxxnet contains concise implementation of state-of-art deep learning models
The project maintains a minimum dependency that makes it portable and easy to build
Scale beyond single GPU and single machine
The library works on multiple GPUs, with nearly linear speedup
THe library works distributedly backed by disrtibuted parameter server
Easy extensibility with no requirement on GPU programming
cxxnet is build on mshadow
developer can write numpy-style template expressions to extend the library only once
mshadow will generate high performance CUDA and CPU code for users
It brings concise and readable code, with performance matching hand crafted kernels
Convenient interface for other languages
Python interface for training from numpy array, and prediction/extraction to numpy array
Matlab interface

Open Source Deep Learning Project: mshadow

mshadow: Matrix Shadow

Project Website: None

Github Link: https://github.com/dmlc/mshadow

Description

MShadow is a lightweight CPU/GPU Matrix/Tensor Template Library in C++/CUDA. The goal of mshadow is to support efficient, device invariant and simple tensor library for machine learning project that aims for maximum performance and control, while also emphasize simplicty.

MShadow also provides interface that allows writing Multi-GPU and distributed deep learning programs in an easy and unified way.

Features

Efficient: all the expression you write will be lazily evaluated and compiled into optimized code
No temporal memory allocation will happen for expression you write
mshadow will generate specific kernel for every expression you write in compile time.
Device invariant: you can write one code and it will run on both CPU and GPU
Simple: mshadow allows you to write machine learning code using expressions.
Whitebox: put a float* into the Tensor struct and take the benefit of the package, no memory allocation is happened unless explicitly called
Lightweight library: light amount of code to support frequently used functions in machine learning
Extendable: user can write simple functions that plugs into mshadow and run on GPU/CPU, no experience in CUDA is required.
MultiGPU and Distributed ML: mshadow-ps interface allows user to write efficient MultiGPU and distributed programs in an unified way.

Open Source Deep Learning Project: deepmat

deepmat

Project Website: None

Github Link: https://github.com/kyunghyuncho/deepmat

Description

= Generative Stochastic Network =

A simple implementation of GSN according to (Bengio et al., 2013)
= Convolutional Neural Network =

A naive implementation (purely using Matlab)
Pooling: max (Jonathan Masci’s code) and average
Not for serious use!
= Restricted Boltzmann Machine & Deep Belief Networks =

Binary/Gaussian Visible Units + Binary Hidden Units
Enhanced Gradient, Adaptive Learning Rate
Adadelta for RBM
Contrastive Divergence
(Fast) Persistent Contrastive Divergence
Parallel Tempering
DBN: Up-down Learning Algorithm
= Deep Boltzmann Machine =

Binary/Gaussian Visible Units + Binary Hidden Units
(Persistent) Contrastive Divergence
Enhanced Gradient, Adaptive Learning Rate
Two-stage Pretraining Algorithm (example)
Centering Trick (fixed center variables only)
= Denoising Autoencoder (Tied Weights) =

Binary/Gaussian Visible Units + Binary(Sigmoid)/Gaussian Hidden Units
tanh/sigm/relu nonlinearities
Shallow: sparsity, contractive, soft-sparsity (log-cosh) regularization
Deep: stochastic backprop
Adagrad, Adadelta
= Multi-layer Perceptron =

Stochastic Backpropagation, Dropout
tanh/sigm/relu nonlinearities
Adagrad, Adadelta
Balanced minibatches using crossvalind()

Open Source Deep Learning Project: Deeplearning4J

Deeplearning4J: Neural Net Platform

Project Website: http://deeplearning4j.org/

Github Link: https://github.com/deeplearning4j/deeplearning4j

Description

Deeplearning4j is the first commercial-grade, open-source, distributed deep-learning library written for Java and Scala. Integrated with Hadoop and Spark, DL4J is designed to be used in business environments on distributed GPUs and CPUs. Skymind is its commercial support arm.

Deeplearning4J is an Apache 2.0-licensed, open-source, distributed neural net library written in Java and Scala.

Deeplearning4J integrates with Hadoop and Spark and runs on several backends that enable use of CPUs and GPUs. The aim is to create a plug-and-play solution that is more convention than configuration, and which allows for fast prototyping.

The most recent stable release in Maven Central is 0.4-rc3.9, and the current master is 0.4-rc3.10-SNAPSHOT. For more on working with snapshots, see this page.

Open Source Deep Learning Project: RNNLIB

RNNLIB

Project Website: https://sourceforge.net/projects/rnnl/

Github Link: None

Description

RNNLIB is a recurrent neural network library for sequence learning problems. Applicable to most types of spatiotemporal data, it has proven particularly effective for speech and handwriting recognition.

full installation and usage instructions given at
http://sourceforge.net/p/rnnl/wiki/Home/

Features
LSTM
Multidimensional recurrent neural networks
Connectionist temporal classification
Adaptive weight noise (stochastic variational inference)

Open Source Deep Learning Project: RNNLM

RNNLM – Tomas Mikolov’s Recurrent Neural Network based Language models Toolkit

Project Website: http://rnnlm.org/

Github Link: None

Description

RNNLM Toolkit

Neural network based language models are nowdays among the most successful techniques for statistical language modeling. They can be easily applied in wide range of tasks, including automatic speech recognition and machine translation, and provide significant improvements over classic backoff n-gram models. The ‘rnnlm’ toolkit can be used to train, evaluate and use such models.

The goal of this toolkit is to speed up research progress in the language modeling field. First, by providing useful implementation that can demonstrate some of the principles. Second, for the empirical experiments when used in speech recognition and other applications. And finally third, by providing a strong state of the art baseline results, to which future research that aims to “beat state of the art techniques” should compare to.

Open Source Deep Learning Project: DeepLearnToolbox

DeepLearnToolbox: Matlab/Octave toolbox for deep learning

Project Website: None

Github Link: https://github.com/rasmusbergpalm/DeepLearnToolbox

Description

Matlab/Octave toolbox for deep learning. Includes Deep Belief Nets, Stacked Autoencoders, Convolutional Neural Nets, Convolutional Autoencoders and vanilla Neural Nets. Each method has examples to get you started.

A Matlab toolbox for Deep Learning.

Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data. It is inspired by the human brain’s apparent deep (layered, hierarchical) architecture. A good overview of the theory of Deep Learning theory is Learning Deep Architectures for AI

For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.

The Next Generation of Neural Networks (Hinton, 2007)
Recent Developments in Deep Learning (Hinton, 2010)
Unsupervised Feature Learning and Deep Learning (Ng, 2011)
If you use this toolbox in your research please cite Prediction as a candidate for learning deep hierarchical models of data

@MASTERSTHESIS\{IMM2012-06284,
author = “R. B. Palm”,
title = “Prediction as a candidate for learning deep hierarchical models of data”,
year = “2012”,
}
Contact: rasmusbergpalm at gmail dot com