public class EnglishUnknownWordModel extends BaseUnknownWordModel
getSignature(String, int).
Implementation note: the contents of this class tend to overlap somewhat
with ArabicUnknownWordModel and were originally included in BaseLexicon.| Modifier and Type | Field and Description |
|---|---|
protected boolean |
smartMutation |
protected int |
unknownPrefixSize |
protected int |
unknownSuffixSize |
protected java.lang.String |
wordClassesFile |
NULL_ITW, nullTag, nullWord, tagHash, tagIndex, trainOptions, unknown, unknownLevel, unSeenCounter, useFirst, useGT, VERBOSE, wordIndex| Constructor and Description |
|---|
EnglishUnknownWordModel(Options op,
Lexicon lex,
Index<java.lang.String> wordIndex,
Index<java.lang.String> tagIndex)
This constructor creates an UWM with empty data structures.
|
EnglishUnknownWordModel(Options op,
Lexicon lex,
Index<java.lang.String> wordIndex,
Index<java.lang.String> tagIndex,
ClassicCounter<IntTaggedWord> unSeenCounter) |
| Modifier and Type | Method and Description |
|---|---|
java.lang.String |
getSignature(java.lang.String word,
int loc)
This routine returns a String that is the "signature" of the class of a
word.
|
int |
getSignatureIndex(int index,
int sentencePosition,
java.lang.String word)
Returns the index of the signature of the word numbered wordIndex, where
the signature is the String representation of unknown word features.
|
float |
score(IntTaggedWord iTW,
int loc,
double c_Tseen,
double total,
double smooth,
java.lang.String word)
Currently we don't consider loc or the other parameters in determining
score in the default implementation; only English uses them.
|
double |
scoreProbTagGivenWordSignature(IntTaggedWord iTW,
int loc,
double smooth,
java.lang.String word)
Calculate P(Tag|Signature) with Bayesian smoothing via just P(Tag|Unknown)
|
addTagging, getLexicon, getUnknownLevel, score, scoreGT, unSeenCounterprotected final boolean smartMutation
protected final int unknownSuffixSize
protected final int unknownPrefixSize
protected final java.lang.String wordClassesFile
public EnglishUnknownWordModel(Options op, Lexicon lex, Index<java.lang.String> wordIndex, Index<java.lang.String> tagIndex, ClassicCounter<IntTaggedWord> unSeenCounter)
public EnglishUnknownWordModel(Options op, Lexicon lex, Index<java.lang.String> wordIndex, Index<java.lang.String> tagIndex)
public float score(IntTaggedWord iTW, int loc, double c_Tseen, double total, double smooth, java.lang.String word)
BaseUnknownWordModelscore in interface UnknownWordModelscore in class BaseUnknownWordModeliTW - An IntTaggedWord pairing a word and POS tagloc - The position in the sentence. In the default implementation
this is used only for unknown words to change their
probability distribution when sentence initial. Now,
a negative value c_Tseen - Total count of this tag (on seen words) in trainingtotal - Total count of word tokens in trainingsmooth - Weighting on prior P(T|U) in estimateword - The word itself; useful so we don't look it up in the indexpublic double scoreProbTagGivenWordSignature(IntTaggedWord iTW, int loc, double smooth, java.lang.String word)
scoreProbTagGivenWordSignature in interface UnknownWordModelscoreProbTagGivenWordSignature in class BaseUnknownWordModelpublic int getSignatureIndex(int index,
int sentencePosition,
java.lang.String word)
getSignatureIndex in interface UnknownWordModelgetSignatureIndex in class BaseUnknownWordModelpublic java.lang.String getSignature(java.lang.String word,
int loc)
getSignature in interface UnknownWordModelgetSignature in class BaseUnknownWordModelword - The word to make a signature forloc - Its position in the sentence (mainly so sentence-initial
capitalized words can be treated differently)