KenLM: Faster and Smaller Language Model Queries

Project Website:

Github Link:

Description

KenLM Language Model Toolkit

benchmark | dependencies | developers | estimation | filter | moses | structures
Ken Models with Computer Engineer Barbie KenLM estimates, filters, and queries language models. Estimation is fast and scalable due to streaming algorithms explained in the paper
Scalable Modified Kneser-Ney Language Model Estimation
Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. ACL, Sofia, Bulgaria, 4—9 August, 2013.
[Paper] [Slides] [BibTeX]
Querying is fast and low-memory, as shown in the paper
KenLM: Faster and Smaller Language Model Queries
Kenneth Heafield. WMT at EMNLP, Edinburgh, Scotland, United Kingdom, 30—31 July, 2011.
[Paper] [Slides] [BibTeX]