Getting started with topia.termextract

Deep Learning Specialization on Coursera

About topia.termextract

Open Source Text Processing Project: topia.termextract

Install topia.termextract

Also topia.termextract has a pip site, but cannot install it by “pip install” method, you should download the source code first:

https://pypi.python.org/packages/d1/b9/452257976ebee91d07c74bc4b34cfce416f45b94af1d62902ae39bf902cf/topia.termextract-1.1.0.tar.gz

Then “tar -zxvf topia.termextract-1.1.0.tar.gz” and “cd topia.termextract-1.1.0” and “sudo python setup.py install” it.

How to Use topia.termextract for term extraction

topia.termextract is based on english pos tagger, so it can be used for Word Tokenize and POS Tagging:

IPython 3.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
 
In [1]: from topia.termextract import tag
 
In [2]: dir(tag)
Out[2]: 
['DATA_DIRECTORY',
 'TERM_SPEC',
 'Tagger',
 '__builtins__',
 '__doc__',
 '__file__',
 '__name__',
 '__package__',
 'correctDefaultNounTag',
 'determineVerbAfterModal',
 'interfaces',
 'normalizePluralForms',
 'os',
 're',
 'verifyProperNounAtSentenceStart',
 'zope']
 
In [3]: tagger = tag.Tagger()
 
In [4]: tagger
Out[4]: <Tagger for english>
 
In [5]: tagger.initialize()
 
In [7]: tagger.tokenize("this's topia.termextarct word tokenize test.")
Out[7]: ['this', "'s", 'topia.termextarct', 'word', 'tokenize', 'test', '.']
 
 
In [9]: test_text = "Terminology extraction (also known as term extraction, glossary extraction, term recognition, or terminology mining) is a subtask of information extraction. The goal of terminology extraction is to automatically extract relevant terms from a given corpus."
 
In [11]: tagger.tokenize("test_text")
Out[11]: ['test', '_text']
 
In [12]: tagger.tokenize(test_text)
Out[12]: 
['Terminology',
 'extraction',
 '(',
 'also',
 'known',
 'as',
 'term',
 'extraction',
 ',',
 'glossary',
 'extraction',
 ',',
 'term',
 'recognition',
 ',',
 'or',
 'terminology',
 'mining',
 ')',
 'is',
 'a',
 'subtask',
 'of',
 'information',
 'extraction',
 '.',
 'The',
 'goal',
 'of',
 'terminology',
 'extraction',
 'is',
 'to',
 'automatically',
 'extract',
 'relevant',
 'terms',
 'from',
 'a',
 'given',
 'corpus',
 '.']
 
In [13]: tagger("this's topia.termextarct word tokenize test.")
Out[13]: 
[['this', 'DT', 'this'],
 ["'s", 'POS', "'s"],
 ['topia.termextarct', 'NN', 'topia.termextarct'],
 ['word', 'NN', 'word'],
 ['tokenize', 'NN', 'tokenize'],
 ['test', 'NN', 'test'],
 ['.', '.', '.']]
 
In [14]: tagger("these examples are more better")
Out[14]: 
[['these', 'DT', 'these'],
 ['examples', 'NNS', 'example'],
 ['are', 'VBP', 'are'],
 ['more', 'JJR', 'more'],
 ['better', 'JJR', 'better']]
 
In [15]: tagger(test_text)
Out[15]: 
[['Terminology', 'NN', 'Terminology'],
 ['extraction', 'NN', 'extraction'],
 ['(', '(', '('],
 ['also', 'RB', 'also'],
 ['known', 'VBN', 'known'],
 ['as', 'IN', 'as'],
 ['term', 'NN', 'term'],
 ['extraction', 'NN', 'extraction'],
 [',', ',', ','],
 ['glossary', 'NN', 'glossary'],
 ['extraction', 'NN', 'extraction'],
 [',', ',', ','],
 ['term', 'NN', 'term'],
 ['recognition', 'NN', 'recognition'],
 [',', ',', ','],
 ['or', 'CC', 'or'],
 ['terminology', 'NN', 'terminology'],
 ['mining', 'NN', 'mining'],
 [')', ')', ')'],
 ['is', 'VBZ', 'is'],
 ['a', 'DT', 'a'],
 ['subtask', 'NN', 'subtask'],
 ['of', 'IN', 'of'],
 ['information', 'NN', 'information'],
 ['extraction', 'NN', 'extraction'],
 ['.', '.', '.'],
 ['The', 'DT', 'The'],
 ['goal', 'NN', 'goal'],
 ['of', 'IN', 'of'],
 ['terminology', 'NN', 'terminology'],
 ['extraction', 'NN', 'extraction'],
 ['is', 'VBZ', 'is'],
 ['to', 'TO', 'to'],
 ['automatically', 'RB', 'automatically'],
 ['extract', 'VB', 'extract'],
 ['relevant', 'JJ', 'relevant'],
 ['terms', 'NNS', 'term'],
 ['from', 'IN', 'from'],
 ['a', 'DT', 'a'],
 ['given', 'VBN', 'given'],
 ['corpus', 'NN', 'corpus'],
 ['.', '.', '.']]

Now let’s getting started with Term Extractor by topia.termextract:

 
In [16]: from topia.termextract import extract
 
In [17]: extractor = extract.TermExtractor()
 
In [18]: extractor
Out[18]: <TermExtractor using <Tagger for english>>
 
 
In [20]: extractor.tagger
Out[20]: <Tagger for english>
 
In [21]: test_sample = """
   ....: Terminology extraction (also known as term extraction, glossary extraction, term recognition, or terminology mining) is a subtask of information extraction. The goal of terminology extraction is to automatically extract relevant terms from a given corpus. In the semantic web era, a growing number of communities and networked enterprises started to access and interoperate through the internet. Modeling these communities and their information needs is important for several web applications, like topic-driven web crawlers,[1] web services,[2] recommender systems,[3] etc. The development of terminology extraction is essential to the language industry. One of the first steps to model the knowledge domain of a virtual community is to collect a vocabulary of domain-relevant terms, constituting the linguistic surface manifestation of domain concepts. Several methods to automatically extract technical terms from domain-specific document warehouses have been described in the literature.Typically, approaches to automatic term extraction make use of linguistic processors (part of speech tagging, phrase chunking) to extract terminological candidates, i.e. syntactically plausible terminological noun phrases, NPs (e.g. compounds "credit card", adjective-NPs "local tourist information office", and prepositional-NPs "board of directors" - in English, the first two constructs are the most frequent[citation needed]). Terminological entries are then filtered from the candidate list using statistical and machine learning methods. Once filtered, because of their low ambiguity and high specificity, these terms are particularly useful for conceptualizing a knowledge domain or for supporting the creation of a domain ontology or a terminology base. Furthermore, terminology extraction is a very useful starting point for semantic similarity, knowledge management, human translation and machine translation, etc.
   ....: """
 
 
In [22]: extractor(test_sample)
Out[22]: 
[('web applications', 1, 2),
 ('domain concepts', 1, 2),
 ('domain-relevant terms', 1, 2),
 ('terminology base', 1, 2),
 ('web', 4, 1),
 ('knowledge', 3, 1),
 ('tourist information office ",', 1, 4),
 ('Terminology extraction', 1, 2),
 ('candidate list', 1, 2),
 ('term', 7, 1),
 ('domain ontology', 1, 2),
 ('knowledge domain', 2, 2),
 ('terminology', 5, 1),
 ('domain', 4, 1),
 ('knowledge management', 1, 2),
 ('information extraction', 1, 2),
 ('glossary extraction', 1, 2),
 ('terminological candidates', 1, 2),
 ('Several methods', 1, 2),
 ('terminology mining', 1, 2),
 ('networked enterprises', 1, 2),
 ('machine translation', 1, 2),
 ('term extraction', 2, 2),
 ('domain-specific document warehouses', 1, 3),
 ('community', 3, 1),
 ('credit card ", adjective-NPs', 1, 4),
 ('extraction', 8, 1),
 ('term recognition', 1, 2),
 ('terminological noun phrases', 1, 3),
 ('language industry', 1, 2),
 ('surface manifestation', 1, 2),
 ('information', 3, 1),
 ('topic-driven web crawlers ,[1] web services ,[2] recommender systems ,[3]',
  1,
  10),
 (']). Terminological entries', 1, 3),
 ('terminology extraction', 3, 2),
 ('web era', 1, 2),
 ('phrase chunking', 1, 2)]
 
In [23]: term_result = extractor(test_sample)
 
In [24]: term_result
Out[24]: 
[('web applications', 1, 2),
 ('domain concepts', 1, 2),
 ('domain-relevant terms', 1, 2),
 ('terminology base', 1, 2),
 ('web', 4, 1),
 ('knowledge', 3, 1),
 ('tourist information office ",', 1, 4),
 ('Terminology extraction', 1, 2),
 ('candidate list', 1, 2),
 ('term', 7, 1),
 ('domain ontology', 1, 2),
 ('knowledge domain', 2, 2),
 ('terminology', 5, 1),
 ('domain', 4, 1),
 ('knowledge management', 1, 2),
 ('information extraction', 1, 2),
 ('glossary extraction', 1, 2),
 ('terminological candidates', 1, 2),
 ('Several methods', 1, 2),
 ('terminology mining', 1, 2),
 ('networked enterprises', 1, 2),
 ('machine translation', 1, 2),
 ('term extraction', 2, 2),
 ('domain-specific document warehouses', 1, 3),
 ('community', 3, 1),
 ('credit card ", adjective-NPs', 1, 4),
 ('extraction', 8, 1),
 ('term recognition', 1, 2),
 ('terminological noun phrases', 1, 3),
 ('language industry', 1, 2),
 ('surface manifestation', 1, 2),
 ('information', 3, 1),
 ('topic-driven web crawlers ,[1] web services ,[2] recommender systems ,[3]',
  1,
  10),
 (']). Terminological entries', 1, 3),
 ('terminology extraction', 3, 2),
 ('web era', 1, 2),
 ('phrase chunking', 1, 2)]
 
In [25]: sorted_term_result = sorted(term_result, key=lambda x: x[1] * x[2], reverse=True)
 
In [26]: sorted_term_result
Out[26]: 
[('topic-driven web crawlers ,[1] web services ,[2] recommender systems ,[3]',
  1,
  10),
 ('extraction', 8, 1),
 ('term', 7, 1),
 ('terminology extraction', 3, 2),
 ('terminology', 5, 1),
 ('web', 4, 1),
 ('tourist information office ",', 1, 4),
 ('knowledge domain', 2, 2),
 ('domain', 4, 1),
 ('term extraction', 2, 2),
 ('credit card ", adjective-NPs', 1, 4),
 ('knowledge', 3, 1),
 ('domain-specific document warehouses', 1, 3),
 ('community', 3, 1),
 ('terminological noun phrases', 1, 3),
 ('information', 3, 1),
 (']). Terminological entries', 1, 3),
 ('web applications', 1, 2),
 ('domain concepts', 1, 2),
 ('domain-relevant terms', 1, 2),
 ('terminology base', 1, 2),
 ('Terminology extraction', 1, 2),
 ('candidate list', 1, 2),
 ('domain ontology', 1, 2),
 ('knowledge management', 1, 2),
 ('information extraction', 1, 2),
 ('glossary extraction', 1, 2),
 ('terminological candidates', 1, 2),
 ('Several methods', 1, 2),
 ('terminology mining', 1, 2),
 ('networked enterprises', 1, 2),
 ('machine translation', 1, 2),
 ('term recognition', 1, 2),
 ('language industry', 1, 2),
 ('surface manifestation', 1, 2),
 ('web era', 1, 2),
 ('phrase chunking', 1, 2)]

Posted by TextProcessing


Leave a Reply

Your email address will not be published. Required fields are marked *