LingPipe Blog | Natural Language Processing and Text Analytics

Time 2021-08-20 11:06:33

Web Name: LingPipe Blog | Natural Language Processing and Text Analytics

WebSite: http://lingpipe-blog.com

ID:188051

Keywords:

Natural,Language,LingPipe,

Description:

Andrew Gelman and David Madigan wrote a paper on why 0-1 loss is so problematic:Gelman and Madigan. 2015. How is Ethics Like Logistic Regression? Chance 28(12).This is related to the issue of whether one should be training on an artificial gold standard. Suppose we have a bunch of annotators and we don t have perfect agreement on items. What do we do? Well, in practice, machine learning evals tend to either (1) throw away the examples without agreement (e.g., the RTE evals, some biocreative named entity evals, etc.), or (2) go with the majority label (everything else I know of). Either way, we are throwing away a huge amount of information by reducing the label to artificial certainty. You can see this pretty easily with simulations, and Raykar et al. showed it with real data. Yet 0/1 corpora and evaluation remain the gold standard (pun intended) in most machine learning evals. Kaggle has gone largely to log loss, but even that s very flat around the middle of the range, as Andrew discusses in this blog post:Gelman. 2016. TOP SECRET: Newly declassified documents on evaluating models based on predictive accuracy. Blog post.The problem is that it s very hard to train a model that s well calibrated if you reduce the data to an artificial gold standard. If you don t know what I mean by calibration, check out this paper:Gneiting, Balabdaoui, and Raftery. 2007. Probabilistic forecasts, calibration and sharpness. J. R. Statist. Soc. B 69.It s one of my favorites. Once you ve understood it, it s hard not to think of evaluating models in terms of calibration and sharpness. Rebecca J. Passonneau and Bob Carpenter. 2014. The Benefits of a Model of Annotation. Transactions of the Association for Comptuational Linguistics (TACL) 2(Oct):311−326. [pdf] (Presented at EMNLP 2014.)Becky just presented it at EMNLP this week. I love the TACL concept and lobbied Michael Collins for short reviewer deadlines based on the fact that (a) I saw the bioinformatics journals could do it and (b) I either do reviews right away or wait until I m nagged, then do them right away they never take even three weeks of real work. The TACL editors/reviewers really were quick, getting responses to us on the initial paper and revisions in under a month. What you GetThere s nothing new in the paper technically that I haven t discussed before on this blog. In fact, the model s a step back from what I consider best practices, mainly to avoid having to justify Bayesian hierarchical models to reviewers and hence avoid reviews like the last one I got. I d highly recommend using a proper hierarchical model (that s proper in the proper English breakfast sense) and using full Bayesian inference. One of the main points of the paper is that high kappa values are not necessary for high quality corpora and even more surprisingly, high kappa values are not even sufficient. Another focus is on why the Dawid and Skene type models that estimate a full categorical response matrix per annotator are preferable to anything that relies on weighted voting (hint: voting can t adjust for bias).We also make a pitch for using expected information gain (aka mutual information) to evaluate quality of annotators. The DataThe paper s very focused on the Mechanical Turk data we collected for word sense. That data s freely available as part of the manually annotated subcorpus (MASC) of the American national corpus (ANC). That was 1000 instances of roughly 50 words (nouns, adjectives and verbs) and roughly 25 annotations per instance, for a total of around 1M labels. You can get it in easily digestible form from the repo linked below.Open-Source Software and DataI wrote a simple EM-based optimizer in R to perform maximum penalized likelihood (equivalently max a posteriori [MAP]) estimates. It s not super fast, but it can deal with 25K labels over 200 annotators and 1000 items in a minute or so. The code is available from aGitHub Repo: bob-carpenter/anno. It includes a CSV-formatted version of all the data we collected in case you want to do your own analysis.We collected so many labels per item so that we could try to get a handle on item difficulty. We just haven t had time to do the analysis. So if you re interested or want to collaborate, let me know (carp@alias-i.com). As promised in my last post, this post shows you how to use Lucene s ranked search results and document store to build a simple classifier. Most of this post is excerpted from Text Processing in Java, Chapter 7, Text Search with Lucene. The data and source code for this example are contained in the source bundle distributed with this book, which can be downloaded from https://github.com/colloquial/javabook.A classifier takes some input, which could be just about anything, and returns a classification of the input over a finite number of discrete categories. The goal of text classification is to assign some text to some category. Given a set of texts that contain discrete category labels, we can use that can assign these labels to new documents based on the similarity of the new documents to the labeled documents.Example Dataset: The 20 Newsgroups CorpusFor this example, we use the data from the 20 Newsgroups corpus, a set of roughly 20,000 messages posted to 20 different newsgroups.  The classification task is to build a system that can assign a newsgroup name (the category label) to a newsgroup message (the text). Since all the messages in the corpus are already labeled, we can use one set of newsgroup posts to train our classifier and use the other set to test the classifier. To test the system, we compare the newsgroup name generated by the classifier against the name of the newsgroup that the message was posted to.To use Lucene to classify a newsgroup post from the test set, we tokenize the message subject and body and create a Lucene query over these tokens. We do a search on the index over the training data and use the newsgroup name of the top-scoring document as the classifier response.  The names of the newsgroups in this corpus are:This dataset is maintained by Jason Rennie. There are no licensing terms listed and the data may be downloaded directly from: http://people.csail.mit.edu/jrennie/20Newsgroups/.  The bundle 20news-bydate.tar.gz is divided into two sets of posts, a training set comprising roughly 60% of the posts to each group and a test set containing the remaining 40% of the posts. This version of the corpus doesn t include cross-posts (messages posted to more than one newsgroup) and the newsgroup-identifying header lines have been removed.  20news-bydate.tar.gz unpacks into two top-level directories: 20news-bydate-train and 20news-bydate-test. Both of these contain 20 subdirectories of newsgroup data, where the directory name is the same as the newsgroup name, and each of these contains a set of newsgroup messages, roughly 400—600 posts for the training sets and roughly 250—400 posts for the test sets.The Program ClassifyNewsgroups.javaThe program ClassifyNewsgroups.java constructs the Lucene index from the training data and then uses this index to do a first-best classification of the test data. The classification results are presented as a confusion matrix that shows the full response profile of the classifiers. This is a 20-by-20 matrix where both the rows and columns are newsgroup names. Each row represents the performance of the classifier on posts from a given newsgroup, and each column represents the classifier response, so that reading across a row, we can see the number of posts given a particular category label. The classifier also reports the percentage of posts in the test set that were correctly labeled.The source bundle contains the source code and the Lucene jar files as well as the file 20news-bydate.tar.gz.  The source code is in directory src/applucene which contains an Ant build.xml file. The ClasifyNewsgroups.java program is in subdirectory src/applucene/src/com/colloquial/applucene. The Ant target classify runs the classification program over the 20 newsgroups data.Building the IndexWe ve already discussed the method buildIndex in previous post on Lucene 4. This method builds a Lucene index over all the documents in the training directory. It takes two java.io.File arguments: the directory for the Lucene index and a the directory containing the training data newsgroup posts.void buildIndex(File indexDir, File trainDir) throws IOException, FileNotFoundException { Directory fsDir = FSDirectory.open(indexDir); IndexWriterConfig iwConf = new IndexWriterConfig(VERSION,mAnalyzer); iwConf.setOpenMode(IndexWriterConfig.OpenMode.CREATE); IndexWriter indexWriter = new IndexWriter(fsDir,iwConf); File[] groupsDir = trainDir.listFiles(); for (File group : groupsDir) { String groupName = group.getName(); File[] posts = group.listFiles(); for (File postFile : posts) { String number = postFile.getName(); NewsPost post = parse(postFile, groupName, number); Document d = new Document(); d.add(new StringField("category", post.group(),Store.YES)); d.add(new TextField("text", post.subject(),Store.NO)); d.add(new TextField("text", post.body(),Store.NO)); indexWriter.addDocument(d); indexWriter.commit(); indexWriter.close();The first four statements in buildIndex create a oal.store.FSDirectory that stores the index as a file-system directory and an IndexWriter which does the work of analyzing documents and adding them to the index. The behavior of the IndexWriter is specified by an IndexWriterConfig object. Documents are indexed via the IndexWriter s default Analyzer. At the outset of the program we define a static final Lucene Version enum constant named VERSION. In order to ensure that the tokenization applied to the query matches the tokenization used to build the index, we use the member variable mAnalyzer to hold the reference to a LuceneAnalyzer. We supply these as arguments to the IndexWriterConfig constructor. We call the setOpenMode method with the enum constant IndexWriterConfig.OpenMode.CREATE, which causes the index writer to create a new index or overwrite an existing one.For each newsgroup sub-directory, all posts in that directory are parsed into a NewsPost object, which is a simple domain model of a newsgroup post that holds the newsgroup name, subject, body, and filename (a string of numbers). The NewsPost object is processed into a Lucene Document that has two fields: a StringField named category for the newsgroup name and a TextField named text that holds the message subject and message body.Using the Index for ClassificationTo use Lucene to classify a newsgroup post from the test set, we tokenize the message subject and body and create a Lucene query over these tokens. We do a search on the index over the training data and use the newsgroup name of the top-scoring document as the classifier response.The method buildQuery creates a Lucene query from the subject and body text of the test message by tokenizing the message subject and body. The Lucene query ORs together all terms (up to a maximum of 1,024) into a BooleanQuery using the BooleanClause enum value SHOULD.BooleanQuery buildQuery(String text) throws IOException { BooleanQuery termsQuery = new BooleanQuery(); Reader textReader = new StringReader(text); TokenStream tokStream = mAnalyzer.tokenStream("text",textReader); try { tokStream.reset(); CharTermAttribute terms = tokStream.addAttribute(CharTermAttribute.class); int ct = 0; while (tokStream.incrementToken() ct++ 1024) { termsQuery. add(new TermQuery(new Term("text", terms.toString())), Occur.SHOULD); tokStream.end(); } finally { tokStream.close(); textReader.close(); return termsQuery;As we process each newsgroup in the test set, we fill in the confusion matrix, one row at a time. To index the rows and columns of the matrix by newsgroup name, we define a String array of newsgroup names called NEWSGROUPS:public static final String[] NEWSGROUPS = { "alt.atheism", "comp.graphics", "comp.os.ms-windows.misc",The method testIndex iterates over the sets of newsgroup posts in the test set and tabulates the results in a 20-by-20 confusion matrix.void testIndex(File indexDir, File testDir) throws IOException, FileNotFoundException { Directory fsDir = FSDirectory.open(indexDir); DirectoryReader reader = DirectoryReader.open(fsDir); IndexSearcher searcher = new IndexSearcher(reader); int[][] confusionMatrix = new int[NEWSGROUPS.length][NEWSGROUPS.length]; File[] groupsDir = testDir.listFiles(); for (File group : groupsDir) { String groupName = group.getName(); int rowIdx = Arrays.binarySearch(NEWSGROUPS,groupName); File[] posts = group.listFiles(); for (File postFile : posts) { String number = postFile.getName(); NewsPost post = parse(postFile, groupName, number); BooleanQuery termsQuery = buildQuery(post.subject() + " " + post.body()); // only get first-best result TopDocs hits = searcher.search(termsQuery,1); ScoreDoc[] scoreDocs = hits.scoreDocs; for (int n = 0; n scoreDocs.length; n++) { ScoreDoc sd = scoreDocs[n]; int docId = sd.doc; Document d = searcher.doc(docId); String category = d.get("category"); // record result in confusion matrix int colIdx = Arrays.binarySearch(NEWSGROUPS,category); confusionMatrix[rowIdx][colIdx]++; System.out.print(groupName); for (int i=0; i NEWSGROUPS.length; i++) System.out.printf("| %4d ", confusionMatrix[rowIdx][i]); System.out.println("|");Evaluating the Classifier and Comparing Tokenization StrategiesThe ClassifyNewsgroups program allows the user to specify the analyzer used. This allows us to see how different tokenization strategies affect classification. The three analyzers available are: the Lucene StandardAnalyzer, which chains together a StandardTokenizer that breaks the text into a stream of words, discarding whitespace and punctuation, followed by a StandardFilter, a LowerCaseFilter, and then a StopFilter, which uses a list of English stop words; an analyzer that chains together just a StandardTokenizer and a LowerCaseFilter; and an Analyzer that uses only a Lucene NGramTokenizer (from package oal.analysis.ngram) to index the data using 4-grams.Here is the top-left quadrant of the confusion matrix, with the results for the first 10 newsgroups of running the classifier using the Lucene StandardAnalyzer:The cells on the diagonal show that in general, this classifier performs well on the data. Posts to alt.atheism are correctly classified in 247 out of 319 cases, at a rate of roughly 75 percent, whereas a classifier which always guessed at random, choosing uniformly among the newsgroups, would be correct only 5 percent of the time. In comparing the classification rates for the posts to the different computer newsgroups, we see a certain amount of confusion between the different kinds of computers and operating systems, as well as confusion with the listings in the misc.forsale newsgroup however, there is little confusion between the posts to the computer newsgroup and alt.atheism or with rec.sports.baseball.To evaluate how well this program is able to identify the newsgroup to which a post belongs, we did several runs over the 20 Newsgroups dataset using different tokenization strategies and tabulated the results. The table below compares the correct classification rates for the three different analyzers for all newsgroups.All classifiers work pretty well. No one classifier works best for all categories, reflecting the difference in the kinds of vocabulary and writing styles used across the different newsgroups.N-Grams can be very effective for both regular search and classification, however the storage requirements for n-grams are considerably greater than for word-based tokenization. The size of index over the 20 Newsgroups training data is 7 MB when the Lucene StandardAnalyzer is used, 8 MB when the lowercase only analyzer is used, and 24 MB for the length-4 n-grams. Here s a short-ish introduction to the Lucene search engine which shows you how to use the current API to develop search over a collection of texts. Most of this post is excerpted from Text Processing in Java, Chapter 7, Text Search with Lucene.Lucene OverviewApache Lucene is a search library written in Java. It’s popular in both academic and commercial settings due to its performance, configurability, and generous licensing terms. The Lucene home page is http://lucene.apache.org.Lucene provides search over documents. A document is essentially a collection of fields. A field consists of a field name that is a string and one or more field values. Lucene does not in any way constrain document structures. Fields are constrained to store only one kind of data, either binary, numeric, or text data. There are two ways to store text data: string fields store the entire item as one string; text fields store the data as a series of tokens. Lucene provides many ways to break a piece of text into tokens as well as hooks that allow you to write custom tokenizers. Lucene has a highly expressive search API that takes a search query and returns a set of documents ranked by relevancy with documents most similar to the query having the highest score.The Lucene API consists of a core library and many contributed libraries. The top-level package is org.apache.lucene, which is abbreviated as oal in this article. As of Lucene 4, the Lucene distribution contains approximately two dozen package-specific jars, e.g.: lucene-core-4.7.0.jar, lucene-analyzers-common-4.7.0.jar, lucene-misc-4.7.0.jar. This cuts down on the size of an application at a small cost to the complexity of the build file.A Lucene Index is an Inverted IndexLucene manages an index over a dynamic collection of documents and provides very rapid updates to the index as documents are added to and deleted from the collection. An index may store a heterogeneous set of documents, with any number of different fields that may vary by document in arbitrary ways. Lucene indexes terms, which means that Lucene search is search over terms. A term combines a field name with a token. The terms created from the non-text fields in the document are pairs consisting of the field name and field value. The terms created from text fields are pairs of field name and token.The Lucene index provides a mapping from terms to documents.  This is called an inverted index because it reverses the usual mapping of a document to the terms it contains.  The inverted index provides the mechanism for scoring search results: if a number of search terms all map to the same document, then that document is likely to be relevant.Here are three entries from an index over part of the The Federalist Papers, a collection of 85 political essays which contains roughly 190,000 word instances over a vocabulary of about 8,000 words.  A field called text holds the contents of each essay, which have been tokenized into words, all lowercase, no punctuation.  The inverted index entries for the terms consisting of field name text and tokens abilities, able, and abolish are:1, 3, 4, 7, 8, 9, 10, 11, 12, 15, 16, 21, 23, 24,25, 27, 29, 30, 33, 34, 35, 36, 37, 38, 41, 43, 46, 49,51, 52, 58, 62, 63, 64, 67, 68, 70, 71, 75, 78, 85Note that the document numbers here are Lucene s internal references to the document.  These ids are not stable; Lucene manages the document id as it manages the index and the internal numbering may change as documents are added to and deleted from the index.Lucene Versions and Version NumbersThe current Apache Lucene Java release is version 4.7, where 4 is the major version number and 7 is the minor version number.  The Apache odometer rolled over to 4.6.0 in November, 2013 and just hit 4.7.0 on February 26, 2014.  At the time that I wrote the Lucene chapter of Text Processing in Java, the current version was 4.5.  Minor version updates maintain backwards compatibility for the given major version therefore, all the example programs in the book compile and run under version 4.7 as well.The behavior of many Lucene components has changed over time.  In particular, the index file format is subject to change from release to release as different methods of indexing and compressing the data are implemented.   To address this, the Enum class oal.util.Version was introduced in Lucene 3.  A Version instance identifies the major and minor versions of Lucene. For example, LUCENE_45 identifies version 4.5.  The Lucene version is supplied to the constructor of the components in an application.  As of Lucene 4.7, older versions have been deprecated, so the compiler issues a warning when older versions are specified.  This means that the examples from the book, which specify version 4.5, generate a compiler warning when compiled under version 4.7.There s no requirement that all components in an application be of the same version however, for components used for both search and indexing, it is critical that the Lucene version is the same in the code that is called at indexing time and the code that is called at search time.  A Lucene index knows what version of Lucene was used to create it (by using the Lucene Version enum constant).  Lucene is backward compatible with respect to searching and maintaining old index versions because it includes classes that can read and write all versions of the index up through the current release.Lucene Indexes FieldsConceptually, Lucene provides indexing and search over documents, but implementation-wise, all indexing and search is carried out over fields.  A document is a collection of fields.  Each field has three parts: name, type, and value.  At search time, the supplied field name restricts the search to particular fields.For example, a MEDLINE citation can be represented as a series of fields: one field for the name of the article, another field for name of the journal in which it was published, another field for the authors of the article, a pub-date field for the date of publication, a field for the text of the article s abstract, and another field for the list of topic keywords drawn from Medical Subject Headings (MeSH).  Each of these fields is given a different name, and at search time, the client could specify that it was searching for authors or titles or both, potentially restricting to a date range and set of journals by constructing search terms for the appropriate fields and values.The Lucene API for fields has changed across the major versions of Lucene as the functionality and organization of the underlying Lucene index have evolved.  Lucene 4 introduces a new interface oal.index.IndexableField, which is implemented by class oal.document.Field.  Lucene 4 also introduces datatype-specific subclasses of Field that encapsulate indexing and storage details for common use cases.  For example, to index integer values, use class oal.document.IntField, and to index simple unanalyzed strings (keywords), use oal.document.StringField.  These so-called sugar subclasses are all final subclasses.The field type is an object that implements oal.index.IndexableFieldType.  Values may be text, binary, or numeric.  The value of a field can be indexed for search or stored for retrieval or both.  The value of an indexed field is processed into terms that are stored in the inverted index.  The raw value of a stored field is stored in the index in a non-inverted manner.  Storing the raw values allows you to retrieve them at search time but may consume substantial space.Indexing and storage options are specified via setter methods on oal.document.FieldType.  These include the method setIndexed(boolean), which specifies whether or not to index a field, and the method setTokenized(boolean), which specifies whether or not the value should be tokenized.  The method setOmitNorms(boolean) controls how Lucene computes term frequency.  Lucene s default behavior is to represent term frequency as a proportion by computing the ratio of the number of times a term occurs to the total number of terms in the document, instead of storing a simple term frequency count.  To do this calculation it stores a normalizing factor for each field that is indexed.  Calling method setOmitNorms with value true turns this off and the raw term frequency is used instead.Some indexing choices are interdependent.  Lucene checks the values on a FieldType object at Field construction time and throws an IllegalArgumentException if the FieldType has inconsistent values.  For example, a field must be either indexed or stored or both, so indexed and/or stored must be true.  If indexed is false, then stored must be true and all other indexing options should be set to false.  The following code fragment defines a custom FieldType and then creates a Field of this type:FieldType myFieldType = new FieldType();myFieldType.setIndexed(true);myFieldType.setOmitNorms(true);myFieldType.setIndexOptions(IndexOptions.DOCS_AND_FREQS);myFieldType.setStored(false);myFieldType.setTokenized(true);myFieldType.freeze();Field myField = new Field("field name", "field value", myFieldType);The source code of the subclasses of Field provides good examples of how to define a FieldType.Analyzing Text into TokensSearch and indexing over text fields require processing text data into tokens.  The package oal.analysis contains the base classes for tokenizing and indexing text.  Processing may consist of a sequence  of transformations , e.g., whitespace tokenization, case normalization, stop-listing, and stemming.The abstract class oal.analysis.TokenStream breaks the incoming text into a sequence of tokens that are retrieved using an iterator-like pattern.  TokenStream has two subclasses: oal.analysis.Tokenizer and oal.analysis.TokenFilter.   A Tokenizer takes a java.io.Reader as input whereas a TokenFilter takes another oal.analysis.TokenStream as input.  This allows us to chain together tokenizers such that the initial tokenizer gets its input from a reader and the others operate on tokens from the preceding TokenStream in the chain.An oal.analysis.Analyzer supplies the indexing and searching processes with TokenStreams on a per-field basis.  It maps field names to tokenizers and may also supply a default analyzer for unknown field names.  Lucene includes many analysis modules that provide concrete implementations of different kinds of analyzers.  As of Lucene 4, these modules are bundled into separate jarfiles.  There are several dozen language-specific analysis packages, from oal.analysis.ar for Arabic to oal.analysis.tr for Turkish.  The package oal.analysis.core provides several general-purpose analyzers, tokenizers, and tokenizer factory classes.The abstract class oal.analysis.Analyzer contains methods used to extract terms from input text.  Concrete subclasses of Analyzer must override the method createComponents, which returns an object of the nested class TokenStreamComponents that defines the tokenization process and provides access to initial and file components of the processing pipeline.  The initial component is a Tokenizer that handles the input source.  The final component is an instance of TokenFilter and it is the TokenStream returned by the method Analyzer.tokenStream(String,Reader).  Here is an example of a custom Analyzer that tokenizes its inputs into individual words with all letters lowercase.Analyzer analyzer = new Analyzer() { @Override protected TokenStreamComponents createComponents(String fieldName, Reader reader) { Tokenizer source = new StandardTokenizer(VERSION,reader); TokenStream filter = new LowerCaseFilter(VERSION,source); return new TokenStreamComponents(source, filter);Note that the constructors for the oal.analysis.standard.StandardTokenizer and oal.analysis.core.LowerCaseFilter objects require a Version argument.  Further note that package oal.analysis.standard is distributed in the jarfile lucene-analyzers-common-4.x.y.jar, where x and y are the minor version and release number.Indexing DocumentsDocument indexing consists of first constructing a document that contains the fields to be indexed or stored, then adding that document to the index.  The key classes involved in indexing are oal.index.IndexWriter, which is responsible for adding documents to an index, and oal.store.Directory, which is the storage abstraction used for the index itself.  Directories provide an interface that s similar to an operating system s file system.  A Directory contains any number of sub-indexes called segments.  Maintaining the index as a set of segments allows Lucene to rapidly update and delete documents from the index.The following example shows how to create a Lucene index given a directory containing a set of data files.  The data in this example is taking from the 20 Newsgroups corpus a set of roughly 20,000 messages posted to 20 different newsgroups.  This code is excerpted from the example program in section 7.11 of Text Processing in Java that shows how to do document classification using Lucene.  We ll get to document classification in a later post.  Right now we just want to go over how to build the index.void buildIndex(File indexDir, File dataDir) throws IOException, FileNotFoundException { Directory fsDir = FSDirectory.open(indexDir); IndexWriterConfig iwConf = new IndexWriterConfig(VERSION,mAnalyzer); iwConf.setOpenMode(IndexWriterConfig.OpenMode.CREATE); IndexWriter indexWriter = new IndexWriter(fsDir,iwConf); File[] groupsDir = dataDir.listFiles(); for (File group : groupsDir) { String groupName = group.getName(); File[] posts = group.listFiles(); for (File postFile : posts) { String number = postFile.getName(); NewsPost post = parse(postFile, groupName, number); Document d = new Document(); d.add(new StringField("category", post.group(),Store.YES)); d.add(new TextField("text", post.subject(),Store.NO)); d.add(new TextField("text", post.body(),Store.NO)); indexWriter.addDocument(d); indexWriter.commit(); indexWriter.close();The method buildIndex walks over the training data directory and parses the newsgroup messages into a NewsPost object, which is a simple domain model of a newsgroup post, consisting of the newsgroup name, subject, body, and filename (a string of numbers).  We treat each post as a Lucene document consisting of two fields: a StringField named category for the newsgroup name and a TextField named text that holds the message subject and message body.  Method buildIndex takes two java.io.File arguments: the directory for the Lucene index and a directory for one set of newsgroup posts where the directory name is the same as the Usenet newsgroup name, e.g.: rec.sport.baseball.The first four statements in buildIndex create a oal.store.FSDirectory that stores the index as a file-system directory and an IndexWriter which does the work of analyzing documents and adding them to the index.  The behavior of the IndexWriter is specified by an IndexWriterConfig object.  Documents are indexed via the IndexWriter s default Analyzer.  At the outset of the program we define a static final Lucene Version enum constant named VERSION. and a Lucene Analyzer named mAnalyzer.  We supply these as arguments to the IndexWriterConfig constructor.  We call the setOpenMode method with the enum constant IndexWriterConfig.OpenMode.CREATE, which causes the index writer to create a new index or overwrite an existing one.A for loop iterates over all files in the data directory.  Each file is first parsed into a Java NewsPost and a corresponding Lucene document is created and fields are added to the document.  This document has two fields: category and text.  Newsgroup names are stored as simple unanalyzed strings.  The field named text stores both the message subject and message body.  A document may have multiple values for a given field.  Search over that field will be over all values for that field, however phrase search over that field will not match across different field values.  When theIndexWriter.addDocument(Document) method is called, the document is added to the index and the constituent fields are indexed and stored accordingly.The final two statements invoke the IndexWriter s commit and close methods, respectively.  The commit method writes all pending changes to the directory and syncs all referenced index files so that the changes are visible to index readers.  Lucene now implements a two-phase commit so that if the commit succeeds, the changes to the index will survive a crash or power loss.  A commit may be expensive, and part of performance tuning is determining when to commit as well as how often and how aggressively to merge the segments of the index.Document Search and Search RankingThe Lucene search API takes a search query and returns a set of documents ranked by relevancy with documents most similar to the query having the highest score.  Lucene provides a highly configurable hybrid form of search that combines exact boolean searches with softer, more relevance-ranking-oriented vector-space search methods.  All searches are field specific, because Lucene indexes terms and a term is composed of a field name and a token.Search QueriesLucene specifies a language in which queries may be expressed.  For instance, computer NOT java produces a query that specifies the term computer must appear in the default field and the term java must not appear.  Queries may specify fields, as in text:java, which requires the term java to appear in the text field of a document.The query syntax includes basic term and field specifications, modifiers for wildcard, fuzzy, proximity, or range searches, and boolean operators for requiring a term to be present, absent, or for combining queries with logical operators.  Finally, sub-queries may be boosted by providing numeric values to raise or lower their prominence relative to other parts of the query. The full syntax specification is in the package level javadoc for package oal.queryparser.classic.  A bonus feature of Text Processing in Java is a quick one-page reference guide to Lucene s search query syntax (see Figure 7.2).Queries may be constructed programmatically using the dozen or so built-in implementations of the oal.search.Query abstract base class.  The most basic kind of query is a search for a single token on a single field, i.e., a single term.  This query is implemented in Lucene s oal.search.TermQuery class.  A term query is constructed from a Term object, which is constructed from a field name and text for the term, both specified as strings.Search ScoringLucene s default search scoring algorithm weights results using TF—IDF, term frequency—inverse document frequency.  Term frequency means that high-frequency terms within a document have higher weight than do low-frequency terms.  Inverse document frequency means that terms that occur frequently across many documents in a collection of documents are less likely to be meaningful descriptors of any given document in a corpus and are therefore down-weighted.  This filters out common words.As of Lucene 4, the API provides alternative scoring algorithms and a pluggable architecture that allows developers to build their own custom scoring models.Search ExampleThe following program illustrates the basic sequence of search operations.  This program is a simplified version of the Lucene search example program included with the book.  When run from the command line, this program takes three arguments: the path to a Lucene index; a query string; the maximum number of results to return.  It runs the specified query against the specified index and prints out the rank, score, and internal document id for all documents that match the query.import org.apache.lucene.analysis.Analyzer;import org.apache.lucene.analysis.standard.StandardAnalyzer;import org.apache.lucene.index.CorruptIndexException;import org.apache.lucene.index.DirectoryReader;import org.apache.lucene.queryparser.classic.ParseException;import org.apache.lucene.queryparser.classic.QueryParser;import org.apache.lucene.search.IndexSearcher;import org.apache.lucene.search.Query;import org.apache.lucene.search.ScoreDoc;import org.apache.lucene.search.TopDocs;import org.apache.lucene.store.Directory;import org.apache.lucene.store.FSDirectory;import org.apache.lucene.util.Version;import java.io.File;import java.io.IOException;public class LuceneSearch { public static void main(String[] args) throws ParseException, CorruptIndexException, IOException { File indexDir = new File(args[0]); String query = args[1]; int maxHits = Integer.parseInt(args[2]); Directory fsDir = FSDirectory.open(indexDir); DirectoryReader reader = DirectoryReader.open(fsDir); IndexSearcher searcher = new IndexSearcher(reader); Analyzer stdAn = new StandardAnalyzer(Version.LUCENE_45); QueryParser parser = new QueryParser(Version.LUCENE_45,"text",stdAn); Query q= parser.parse(query); TopDocs hits = searcher.search(q,maxHits); ScoreDoc[] scoreDocs = hits.scoreDocs; System.out.println("hits=" + scoreDocs.length); System.out.println("Hits (rank,score,docId)"); for (int n = 0; n scoreDocs.length; ++n) { ScoreDoc sd = scoreDocs[n]; float score = sd.score; int docId = sd.doc; System.out.printf("%3d %4.2f %d\n", n, score, docId); reader.close();The main method is declared to throw a Lucene corrupt index exception if the index isn t well formed, a Lucene parse exception if the query isn t well formed, and a general Java I/O exception if there is a problem reading from or writing to the index directory.  It starts off by reading in command-line arguments, then it creates a Lucene directory, index reader, index searcher, and query parser, and then it uses the query parser to parse the query.Lucene uses instances of the aptly named IndexReader to read data from an index, in this example, we use an instance of class oal.index.DirectoryReader.  The oal.search.IndexSearcher class performs the actual search.  Every index searcher wraps an index reader to get a handle on the indexed data.  Once we have an index searcher, we can supply queries to it and enumerate results in order of their score.The class oal.queryparser.classic.QueryParserBase provides methods to parse a query string into a oal.search.Query object or throw a oal.queryparser.classic.ParseException if it is not.  All tokens are analyzed using the specified Analyzer.  This example is designed to search an index for Lucene version 4.5 that was created using an analyzer of class oal.analysis.standard.StandardAnalyzer which contains text a field named text.It is important to use the same analyzer in the query parser as is used in the creation of the index.  If they don t match, queries that should succeed will fail should the different analyzers produce differing tokenizations given the same input.  For instance, if we apply stemming to the contents of text field text during indexing and reduce the word codes to code, but we don t apply stemming to the query, then a search for the word codes in field text will fail. The search term is text:codes but the index contains only the term text:code.  Note that package oal.queryparser.classic is distributed in the jarfile lucene-queryparser-4.x.y.jar.The actual search is done via a call to the IndexSearcher s search method, which takes two arguments: the query and an upper bound on the number of hits to return.  It returns an instance of the Lucene class oal.search.TopDocs.  The TopDocs result provides access to an array of search results.  Each result is an instance of the Lucene class oal.search.ScoreDoc, which encapsulates a document reference with a floating point score.  The array of search results is sorted in decreasing order of score, with higher scores representing better matches.  For each ScoreDoc object, we get the score from the public member variable score.  We get the document reference number (Lucene s internal identifier for the doc) from the public member variable doc.  The document identifier is used to retrieve the document from the searcher.  Here, we re just using the id as a diagnostic to illustrate the basics of search and so we just print the number, even though it is only an internal reference.  Finally, we close the IndexReader.Discussion and Further ReadingThis post tries to cover the essentials of Lucene 4 in a very short amount of space.  In order to do this, this post contains minimal amounts of examples, and links to the Lucene javadocs have been substituted for detailed explanation of how classes and methods behave.  Putting these code fragments together into a full application is left as an exercise to the reader.Back when Lucene was at version 2.4 I wrote my first blog post titled Lucene 2.4 in 60 Seconds.   A more accurate title would have been Lucene 2.4 in 60 Seconds or More.   Likewise, a more accurate title for this post would be The Essential Essentials of Text Search and Indexing with Lucene 4 but that s just not very snappy.  For more on the essentials of search and indexing with Lucene, please check out Chapter Seven of Text Processing in Java, (and all the other chapters as well).To cover all of Lucene would take an entire book, and there are many good ones out there. My favorite is Lucene in Action, Second Edition, which, alas, only covers version 3. I m pleased to announce the publication of Text Processing in Java !This book teaches you how to master the subtle art of multilingual text processing and prevent text data corruption.  Data corruption is the all-too-common problem of words that are garbled into strings of question marks, black diamonds, or random glyphs.  In Japanese this is called mojibake (“character change”), written 文字化け, but on your browser it might look like this: �����  or this: 文字化ã‘. When this happens, pinpointing the source of error can be surprisingly difficult and time-consuming. The information and example programs in this book make it easy.This book also provides an introduction to natural language processing using Lucene and Solr. It covers the tools and techniques necessary for managing large collections of text data, whether they come from news feeds, databases, or legacy documents. Each chapter contains executable programs that can also be used for text data forensics.Topics covered:Unicode code pointsCharacter encodings from ASCII and Big5 to UTF-8 and UTF-32LECharacter normalization using International Components for Unicode (ICU)Java I/O, including working directly with zip, gzip, and tar filesRegular expressions in JavaTransporting text data via HTTPParsing and generating XML, HTML, and JSONUsing Lucene 4 for natural language search and text classificationSearch, spelling correction, and clustering with Solr 4Other books on text processing presuppose much of the material covered in this book. They gloss over the details of transforming text from one format to another and assume perfect input data. The messy reality of raw text will have you reaching for this book again and again.Buy Text Processing in Java on Amazon Code Camp is canceled. We are late on delivering a LingPipe recipes book to our publisher and that will have to be our project for March. But we could have testers/reviewers come and hang out. Less fun I think.Apologies.Breck Dates: To be absolutely clear: Dates are 3/2/2014 to 3/31/14 in Driggs Idaho. You can work remotely, we will be doing stuff before as well for setup.New: We have setup a github repository. URL is https://github.com/watsonclone/jeopardy-.git Every year we go out west for a month of coding and skiing. Last year it was Salt Lake City Utah, this year it is Driggs Idaho for access to Grand Targhee and Jackson Hole. The house is rented for the month of March and the project selected. This year we will create a Watson clone.I have blogged about Watson before and I totally respect what they have done. But the coverage is getting a bit breathless with the latest billion dollar effort. So how about assembling a scrappy bunch of developers and see how close we can come to recreating the Jeopardy beast?How this works:We have a month of focused development. We tend to code mornings, ski afternoons. House has 3 bedrooms if you want to come stay. Prior arrangements must be made. House is paid for. Nothing else is.The code base will be open source. We are moving LingPipe to the AGPL so maybe that license or we could just apache license it. We want folks to be comfortable contributing.You don t have to be present to contribute.We have a very fun time last year. We worked very hard on an email app that didn t quite get launched but the lesson learned was to start with the project defined.If you are interested in participating let us know at watsonclone@lingpipe.com. Let us know your background, what you want to do, do you expect to stay with us etc . No visa situations please and we don t have any funds to support folks. Obviously we have limited space physically and mentally so we may say no but we will do our best to be inclusive. Step 1 transcribe some Jeopardy shows.Ask questions in the comments so all can benefit. Check comments before asking questions pls. I ll answer the first one that is on everyone s mind:Q: Are you frigging crazy???A: Why, yes, yes we are. But we are also really good computational linguists .Breck If we are representing probabilities, we are interested in numbers between 0 and 1. It turns out these have very different properties in floating-point arithmetic. And it s not as easy to solve as just working on the log scale.Smallest Gaps between Floating Point Numbers of Different MagnitudesThe difference between 0 and the next biggest number representable in double-precision floating point (Java or C++ double) is on the order of 1e-300. In constrast, the difference between 1 and the next smallest number representable is around 1e-15. The reason is that to exploit the maximum number of significant digits, the mantissa for the number near 1 has to be scaled roughly like 0.999999999999999 or 1.0000000000001 and the exponent has to be scaled like 0.CDFs and Complementary CDFsThis is why there are two differently translated error functions in math libs, erf() and erfc(), which are rescaled cumulative and complementary cumulative distribution functions. That is, you can use erf() to calculate the cumulative unit normal distribution function (cdf), written Phi(x); erfc() can be used to calculate the complementary cumulative unit normal distribution function (ccdf), written (1 Phi(x)).Log Scale no PanaceaSwitching to the log scale doesn t sove the problem. If you have a number very close to 1, you need to be careful in taking its log. If you write log(1-x), you run into the problem that x can only be so close to 1. That s why standard math libraries provide a log1p() function defined by log1p(x) = log(1 + x). This gives you back the precision you lose by subtraction two numbers close to each other (what s called catastrophic cancellation in the arithmetic processing literature).A Little QuizI ll end with a little quiz to get you thinking about floating point a little more closely. Suppose you set the bits in a floating point number randomly, say by flipping a coin for each bit. Junior Varsity: What s the approximate probability that the random floating point number has an absolute value less than 1? Varsity: What is the event probability of drawing a number in the range (L,U). Or, equivalently, what s the density function to which the draws are proportional?Antske Fokkens, Marieke van Erp, Marten Postma, Ted Pedersen, Piek Vossen, Nuno Freire. 2013. Offspring from Reproduction Problems: What Replication Failure Teaches Us. ACL Proceedings.You should, too. Or if you d rather watch, there s a video of their ACL presentation.The GistThe gist of the paper is that the authors tried to reproduce some results found in the literature for word sense identification and named-entity detection, and had a rather rough go of it. In particular, they found that every little decision they made impacted the evaluation error they got, including how to evaluate the error. As the narrative of their paper unfolded, all I could do was nod along. I loved that their paper was in narrative form it made it very easy to follow.The authors stress that we need more information for reproducibility and that reproducing known results should get more respect. No quibble there. A secondary motivation (after the pedagogical one) of the LingPipe tutorials was to show that we could reproduce existing results in the literature.The Minor FallAlthough they re riding one of my favorite hobby horses, they don t quite lead it in the direction I would have. In fact, they stray into serious discussion of the point estimates of performance that their experiments yielded, despite the variation they see under cross-validation folds. So while they note that rankings of approaches changes based on what seem like minor details, they stop short of calling into question the whole idea of trying to assign a simple ranking. Instead, they stress getting to the bottom of why the rankings differ. A Slightly Different Take on the IssueWhat I would ve liked to have seen in this paper is more emphasis on two key statistical concepts: the underlying sample variation problem, andthe overfitting problem with evaluations.Sample VariationThe authors talk about sample variation a bit when they consider different folds, etc., in cross-validation. For reasons I don t understand they call it experimental setup. But they don t put it in terms of estimation variance. That is, independently of getting the features to all line up, the performance and hence rankings of various approaches vary to a large extent because of sample variation. The easiest way to see this is to run cross-validation.I would have stressed that the bootstrap method should be used to get at sample variation. The bootstrap differs from cross-validation in that it samples training and test subsets with replacement. The bootstrap effectively evaluates within-sample sampling variation, even in the face of non-i.i.d. samples (usually there is correlation among the items in a corpus due to choosing multiple sentences from a single document or even multiple words from the same sentence). The picture is even worse in that the data set at hand is rarely truly representative of the intended out-of-sample application, which is typically to new text. The authors touch on these issues with discussions of robustness. OverfittingThe authors don t really address the overfitting problem, though it is tangential to what they call the versioning problem (when mismatched versions of corpora such as WordNet produce different results). Overfitting is a huge problem in the current approach to evaluation in NLP and machine learning research. I don t know why anyone cares about the umpteenth evaluation on the same fold of the Penn Treebank that produces a fraction of a percentage gain. When the variation across folds is higher than your estimated gain on a single fold, it s time to pack the papers in the file drawer, not in the proceedings of ACL. At least until we also get a journal of negative results to go with our journal of reproducibility. The Major LiftPerhaps hill-climbing error on existing data sets is not the best way forward methdologically. Maybe, just maybe, we can make the data better. It s certainly what Breck and I tell customers who want to build an application.If making the data better sounds appealing, check out my favorite NLP paper in recent memory, which is about improving data quality, and one of my all-time favorite NLP papers, which is about improving data quantity.I find the two ideas so compelling that it s the only area of NLP I m actively researching. More on that later (or sooner if you want to read ahead). (My apologies to Leonard Cohen for stealing his lyrics.) We have started an incubator program for start-ups with natural language processing (NLP) needs. Seer is our first egg. The company creates productivity apps based on unstructured data sources that is where LingPIpe fits in. The guys (Conall and Joe) fit a pattern that we see quite often smart people with a good idea that presupposes NLP but they don t have much experience with it.The GetSeer guys were on the ball right away because they had a bunch of data annotated and had cobbled together a regular expression based classifier that served as an excellent starting point. We had machine learning based classifiers up and running within two hours of starting. That included an evaluation harness. Nice.The great thing about our working setup is that I get to teach/advise which keeps my time commitment manageable and they get to do most of the heavy lifting which is how they learn to build and maintain NLP systems. I have had to learn some Scala since I co-code most of the time when we work together and that is their implementation language. Scala is a hip extension of Java with a less verbose syntax and stronger type inference.I ll keep the blog updated with developments. Current status:There is a small amount of training data.Current results are awful (no surprise).We have been roughing in the solution with Naive Bayes. Next steps will be opening a can of logistic regression for more interesting feature extraction. One of the perennial problems in tokenization for search, classification, or clustering of natural language data is which words and phrases are spelled as compounds and which are separate words. For instance consider dodgeball (right) vs. dodge ball (wrong) and golfball (wrong) vs. golf ball (right)? It s a classic lumpers vs. splitters problem. Alas, the right answer can change over time and some words are listed as both lumped and split in some dictionaries. This point was first brought home to me and Breck when we were browsing Comcast s query logs for TV search queries, where users seemed to arbitrarily insert or delete spaces in queries. Who can blame them with the inconsistencies of English in this regard? This token lumping vs. splitting problem is one of the reasons LingPipe s spelling correction is not token-based like Lucene s (though Lucene has other ways to handle combining and splitting compounds more on that from Mitzi in the near future). And it s one of the reasons (along with morphological affixes) that character n-grams provide a better basis for classification than hard tokenization.My favorite examples are comic book superhero names. It turns out that the editors at Marvel Comics are splitters, whereas those at DC Comics are lumpers. Hence we have on the Marvel roster, Ant-Man, Iron Man, and Spider-Man.They re not consistent on hyphenation, which is another issue if you don t throw away punctuation.DC, on the other hand, employs men, women, and children, but doesn t have the budget for spaces:Aquaman, Batman, Catwoman, Hawkman, Martian Manhunter, Superboy, Supergirl, and Superman.The exception that proves the rule (I m open-minded enough to use a phrase that s a pet peeve of mine), DC gives us:Wonder Woman

TAGS:Natural Language LingPipe 

<<< Thank you for your visit >>>

Natural Language Processing and Text Analytics

Hot Websites