xapian-core  1.5.0
Public Types | Public Member Functions | List of all members
Xapian::TfIdfWeight Class Reference

Xapian::Weight subclass implementing the tf-idf weighting scheme. More...

+ Inheritance diagram for Xapian::TfIdfWeight:

Public Types

enum  wdf_norm : unsigned char {
  wdf_norm::NONE = 1, wdf_norm::BOOLEAN = 2, wdf_norm::SQUARE = 3, wdf_norm::LOG = 4,
  wdf_norm::PIVOTED = 5, wdf_norm::LOG_AVERAGE = 6, wdf_norm::AUG_LOG = 7, wdf_norm::SQRT = 8,
  wdf_norm::AUG_AVERAGE = 9, wdf_norm::MAX = 10, wdf_norm::AUG = 11
}
 Wdf normalizations. More...
 
enum  idf_norm : unsigned char {
  idf_norm::NONE = 1, idf_norm::TFIDF = 2, idf_norm::SQUARE = 3, idf_norm::FREQ = 4,
  idf_norm::PROB = 5, idf_norm::PIVOTED = 6, idf_norm::GLOBAL_FREQ = 7, idf_norm::LOG_GLOBAL_FREQ = 8,
  idf_norm::INCREMENTED_GLOBAL_FREQ = 9, idf_norm::SQRT_GLOBAL_FREQ = 10
}
 Idf normalizations. More...
 
enum  wt_norm : unsigned char { wt_norm::NONE = 1 }
 Weight normalizations. More...
 
- Public Types inherited from Xapian::Weight
enum  type_smoothing {
  TWO_STAGE_SMOOTHING = 1, DIRICHLET_SMOOTHING = 2, ABSOLUTE_DISCOUNT_SMOOTHING = 3, JELINEK_MERCER_SMOOTHING = 4,
  DIRICHLET_PLUS_SMOOTHING = 5
}
 Type of smoothing to use with the Language Model Weighting scheme. More...
 

Public Member Functions

 TfIdfWeight (const std::string &normalizations)
 Construct a TfIdfWeight. More...
 
 TfIdfWeight (const std::string &normalizations, double slope, double delta)
 Construct a TfIdfWeight. More...
 
 TfIdfWeight (wdf_norm wdf_normalization, idf_norm idf_normalization, wt_norm wt_normalization)
 Construct a TfIdfWeight. More...
 
 TfIdfWeight (wdf_norm wdf_norm_, idf_norm idf_norm_, wt_norm wt_norm_, double slope, double delta)
 Construct a TfIdfWeight. More...
 
 TfIdfWeight ()
 Construct a TfIdfWeight using the default normalizations ("ntn").
 
std::string name () const
 Return the name of this weighting scheme. More...
 
std::string short_name () const
 Return the short name of the weighting scheme. More...
 
std::string serialise () const
 Return this object's parameters serialised as a single string. More...
 
TfIdfWeightunserialise (const std::string &serialised) const
 Unserialise parameters. More...
 
double get_sumpart (Xapian::termcount wdf, Xapian::termcount doclen, Xapian::termcount uniqterm, Xapian::termcount wdfdocmax) const
 Calculate the weight contribution for this object's term to a document. More...
 
double get_maxpart () const
 Return an upper bound on what get_sumpart() can return for any document. More...
 
double get_sumextra (Xapian::termcount doclen, Xapian::termcount uniqterms, Xapian::termcount wdfdocmax) const
 Calculate the term-independent weight component for a document. More...
 
double get_maxextra () const
 Return an upper bound on what get_sumextra() can return for any document. More...
 
TfIdfWeightcreate_from_parameters (const char *params) const
 Return the parameterised weighting scheme object. More...
 
- Public Member Functions inherited from Xapian::Weight
 Weight ()
 Default constructor, needed by subclass constructors.
 
virtual ~Weight ()
 Virtual destructor, because we have virtual methods.
 

Additional Inherited Members

- Static Public Member Functions inherited from Xapian::Weight
static const Weightcreate (const std::string &scheme, const Registry &reg=Registry())
 Return the appropriate weighting scheme object. More...
 
- Protected Types inherited from Xapian::Weight
enum  stat_flags {
  COLLECTION_SIZE = 1, RSET_SIZE = 2, AVERAGE_LENGTH = 4, TERMFREQ = 8,
  RELTERMFREQ = 16, QUERY_LENGTH = 32, WQF = 64, WDF = 128,
  DOC_LENGTH = 256, DOC_LENGTH_MIN = 512, DOC_LENGTH_MAX = 1024, WDF_MAX = 2048,
  COLLECTION_FREQ = 4096, UNIQUE_TERMS = 8192, TOTAL_LENGTH = 16384, WDF_DOC_MAX = 32768
}
 Stats which the weighting scheme can use (see need_stat()). More...
 
- Protected Member Functions inherited from Xapian::Weight
void need_stat (stat_flags flag)
 Tell Xapian that your subclass will want a particular statistic. More...
 
 Weight (const Weight &)
 Don't allow copying. More...
 
Xapian::doccount get_collection_size () const
 The number of documents in the collection.
 
Xapian::doccount get_rset_size () const
 The number of documents marked as relevant.
 
Xapian::doclength get_average_length () const
 The average length of a document in the collection.
 
Xapian::doccount get_termfreq () const
 The number of documents which this term indexes.
 
Xapian::doccount get_reltermfreq () const
 The number of relevant documents which this term indexes.
 
Xapian::termcount get_collection_freq () const
 The collection frequency of the term.
 
Xapian::termcount get_query_length () const
 The length of the query.
 
Xapian::termcount get_wqf () const
 The within-query-frequency of this term.
 
Xapian::termcount get_doclength_upper_bound () const
 An upper bound on the maximum length of any document in the database. More...
 
Xapian::termcount get_doclength_lower_bound () const
 A lower bound on the minimum length of any document in the database. More...
 
Xapian::termcount get_wdf_upper_bound () const
 An upper bound on the wdf of this term. More...
 
Xapian::totallength get_total_length () const
 Total length of all documents in the collection.
 

Detailed Description

Xapian::Weight subclass implementing the tf-idf weighting scheme.

Member Enumeration Documentation

◆ idf_norm

enum Xapian::TfIdfWeight::idf_norm : unsigned char
strong

Idf normalizations.

Enumerator
NONE 

None.

    idfn=1
TFIDF 

TfIdf.

    idfn=log(N/Termfreq) where N is the number of documents
    in collection and Termfreq is the number of documents which are
    indexed by the term t.
SQUARE 

Square.

    idfn=log(N/Termfreq)^2
FREQ 

Frequency.

    idfn=1/Termfreq
PROB 

Probability.

    idfn=log((N-Termfreq)/Termfreq)
PIVOTED 

Pivoted.

    idfn=log((N+1)/Termfreq)
GLOBAL_FREQ 

Global frequency IDF.

    idfn=Collfreq/Termfreq
LOG_GLOBAL_FREQ 

Log global frequency IDF.

    idfn=log(Collfreq/Termfreq+1)
INCREMENTED_GLOBAL_FREQ 

Incremented global frequency IDF.

    idfn=Collfreq/Termfreq+1
SQRT_GLOBAL_FREQ 

Square root global frequency IDF.

    idfn=sqrt(Collfreq/Termfreq-0.9)

◆ wdf_norm

enum Xapian::TfIdfWeight::wdf_norm : unsigned char
strong

Wdf normalizations.

Enumerator
NONE 

None.

    wdfn=wdf
BOOLEAN 

Boolean.

    wdfn=1 if term in document else wdfn=0
SQUARE 

Square.

    wdfn=wdf*wdf
LOG 

Logarithmic.

    wdfn=1+log<sub>e</sub>(wdf)
PIVOTED 

Pivoted.

    wdfn=(1+log(1+log(wdf)))*
     (1/(1-slope+(slope*doclen/avg_len)))+delta
LOG_AVERAGE 

Log average.

    wdfn=(1+log(wdf))/
     (1+log(doclen/unique_terms))
AUG_LOG 

Augmented Log.

    wdfn=0.2+0.8*log(wdf+1)
SQRT 

Square Root.

    wdfn=sqrt(wdf-0.5)+1 if(wdf>0), else wdfn=0
AUG_AVERAGE 

Augmented average term frequency.

    wdfn=0.9+0.1*(wdf/(doclen/unique_terms)) if(wdf>0), else wdfn=0
MAX 

Max wdf.

    wdfn=wdf/wdfdocmax
AUG 

Augmented max wdf.

    wdfn=0.5+0.5*wdf/wdfdocmax if(wdf>0), else wdfn=0

◆ wt_norm

enum Xapian::TfIdfWeight::wt_norm : unsigned char
strong

Weight normalizations.

Enumerator
NONE 

None.

    wtn=tfn*idfn

Constructor & Destructor Documentation

◆ TfIdfWeight() [1/4]

Xapian::TfIdfWeight::TfIdfWeight ( const std::string &  normalizations)
inlineexplicit

Construct a TfIdfWeight.

Parameters
normalizationsA three character string indicating the normalizations to be used for the tf(wdf), idf and document weight. (default: "ntn")

The normalizations string works like so:

  • The first character specifies the normalization for the wdf. The following normalizations are currently supported:
  • 'n': None. wdfn=wdf
  • 'b': Boolean wdfn=1 if term in document else wdfn=0
  • 's': Square wdfn=wdf*wdf
  • 'l': Logarithmic wdfn=1+loge(wdf)
  • 'P': Pivoted wdfn=(1+log(1+log(wdf)))*(1/(1-slope+(slope*doclen/avg_len)))+delta
  • 'L': Log average wdfn=(1+log(wdf))/(1+log(doclen/unique_terms))
  • 'm': Max-wdf wdfn=wdf/wdfdocmax
  • 'a': Augmented max-wdf wdfn=0.5+0.5*wdf/wdfdocmax
  • The second character indicates the normalization for the idf. The following normalizations are currently supported:
  • 'n': None idfn=1
  • 't': TfIdf idfn=log(N/Termfreq) where N is the number of documents in collection and Termfreq is the number of documents which are indexed by the term t.
  • 'p': Prob idfn=log((N-Termfreq)/Termfreq)
  • 'f': Freq idfn=1/Termfreq
  • 's': Squared idfn=log(N/Termfreq)^2
  • 'P': Pivoted idfn=log((N+1)/Termfreq)
  • The third and the final character indicates the normalization for the document weight. The following normalizations are currently supported:
  • 'n': None wtn=tfn*idfn

Implementing support for more normalizations of each type would require extending the backend to track more statistics.

◆ TfIdfWeight() [2/4]

Xapian::TfIdfWeight::TfIdfWeight ( const std::string &  normalizations,
double  slope,
double  delta 
)

Construct a TfIdfWeight.

Parameters
normalizationsA three character string indicating the normalizations to be used for the tf(wdf), idf and document weight. (default: "ntn")
slopeExtra parameter for "Pivoted" tf normalization. (default: 0.2)
deltaExtra parameter for "Pivoted" tf normalization. (default: 1.0)

The normalizations string works like so:

  • The first character specifies the normalization for the wdf. The following normalizations are currently supported:
  • 'n': None. wdfn=wdf
  • 'b': Boolean wdfn=1 if term in document else wdfn=0
  • 's': Square wdfn=wdf*wdf
  • 'l': Logarithmic wdfn=1+loge(wdf)
  • 'P': Pivoted wdfn=(1+log(1+log(wdf)))*(1/(1-slope+(slope*doclen/avg_len)))+delta
  • 'm': Max-wdf wdfn=wdf/wdfdocmax
  • 'a': Augmented max-wdf wdfn=0.5+0.5*wdf/wdfdocmax
  • The second character indicates the normalization for the idf. The following normalizations are currently supported:
  • 'n': None idfn=1
  • 't': TfIdf idfn=log(N/Termfreq) where N is the number of documents in collection and Termfreq is the number of documents which are indexed by the term t.
  • 'p': Prob idfn=log((N-Termfreq)/Termfreq)
  • 'f': Freq idfn=1/Termfreq
  • 's': Squared idfn=log(N/Termfreq)^2
  • 'P': Pivoted idfn=log((N+1)/Termfreq)
  • The third and the final character indicates the normalization for the document weight. The following normalizations are currently supported:
  • 'n': None wtn=tfn*idfn

Implementing support for more normalizations of each type would require extending the backend to track more statistics.

◆ TfIdfWeight() [3/4]

Xapian::TfIdfWeight::TfIdfWeight ( wdf_norm  wdf_normalization,
idf_norm  idf_normalization,
wt_norm  wt_normalization 
)
inline

Construct a TfIdfWeight.

Parameters
wdf_norm_The normalization for the wdf.
idf_norm_The normalization for the idf.
wt_norm_The normalization for the document weight.

Implementing support for more normalizations of each type would require extending the backend to track more statistics.

◆ TfIdfWeight() [4/4]

Xapian::TfIdfWeight::TfIdfWeight ( wdf_norm  wdf_norm_,
idf_norm  idf_norm_,
wt_norm  wt_norm_,
double  slope,
double  delta 
)

Construct a TfIdfWeight.

Parameters
wdf_norm_The normalization for the wdf.
idf_norm_The normalization for the idf.
wt_norm_The normalization for the document weight.
slopeExtra parameter for "Pivoted" tf normalization. (default: 0.2)
deltaExtra parameter for "Pivoted" tf normalization. (default: 1.0)

Implementing support for more normalizations of each type would require extending the backend to track more statistics.

Member Function Documentation

◆ create_from_parameters()

TfIdfWeight* Xapian::TfIdfWeight::create_from_parameters ( const char *  params) const
virtual

Return the parameterised weighting scheme object.

Parameters
paramsthe pointer to the string containing parameter values for a weighting scheme

Reimplemented from Xapian::Weight.

◆ get_maxextra()

double Xapian::TfIdfWeight::get_maxextra ( ) const
virtual

Return an upper bound on what get_sumextra() can return for any document.

This information is used by the matcher to perform various optimisations, so strive to make the bound as tight as possible.

Implements Xapian::Weight.

◆ get_maxpart()

double Xapian::TfIdfWeight::get_maxpart ( ) const
virtual

Return an upper bound on what get_sumpart() can return for any document.

This information is used by the matcher to perform various optimisations, so strive to make the bound as tight as possible.

Implements Xapian::Weight.

◆ get_sumextra()

double Xapian::TfIdfWeight::get_sumextra ( Xapian::termcount  doclen,
Xapian::termcount  uniqterms,
Xapian::termcount  wdfdocmax 
) const
virtual

Calculate the term-independent weight component for a document.

The parameter gives information about the document which may be used in the calculations:

Parameters
doclenThe document's length (unnormalised).
uniqtermsThe number of unique terms in the document.

Implements Xapian::Weight.

◆ get_sumpart()

double Xapian::TfIdfWeight::get_sumpart ( Xapian::termcount  wdf,
Xapian::termcount  doclen,
Xapian::termcount  uniqterms,
Xapian::termcount  wdfdocmax 
) const
virtual

Calculate the weight contribution for this object's term to a document.

The parameters give information about the document which may be used in the calculations:

Parameters
wdfThe within document frequency of the term in the document.
doclenThe document's length (unnormalised).
uniqtermsNumber of unique terms in the document (used for absolute smoothing).
wdfdocmaxMaximum wdf value in the document.

Implements Xapian::Weight.

◆ name()

std::string Xapian::TfIdfWeight::name ( ) const
virtual

Return the name of this weighting scheme.

This name is used by the remote backend. It is passed along with the serialised parameters to the remote server so that it knows which class to create.

Return the full namespace-qualified name of your class here - if your class is called FooWeight, return "FooWeight" from this method (Xapian::BM25Weight returns "Xapian::BM25Weight" here).

If you don't want to support the remote backend, you can use the default implementation which simply returns an empty string.

Reimplemented from Xapian::Weight.

◆ serialise()

std::string Xapian::TfIdfWeight::serialise ( ) const
virtual

Return this object's parameters serialised as a single string.

If you don't want to support the remote backend, you can use the default implementation which simply throws Xapian::UnimplementedError.

Reimplemented from Xapian::Weight.

◆ short_name()

std::string Xapian::TfIdfWeight::short_name ( ) const
virtual

Return the short name of the weighting scheme.

E.g. "bm25".

Reimplemented from Xapian::Weight.

◆ unserialise()

TfIdfWeight* Xapian::TfIdfWeight::unserialise ( const std::string &  serialised) const
virtual

Unserialise parameters.

This method unserialises parameters serialised by the serialise() method and allocates and returns a new object initialised with them.

If you don't want to support the remote backend, you can use the default implementation which simply throws Xapian::UnimplementedError.

Note that the returned object will be deallocated by Xapian after use with "delete". If you want to handle the deletion in a special way (for example when wrapping the Xapian API for use from another language) then you can define a static operator delete method in your subclass as shown here: https://trac.xapian.org/ticket/554#comment:1

Parameters
serialisedA string containing the serialised parameters.

Reimplemented from Xapian::Weight.


The documentation for this class was generated from the following file: