Vector Space Model - Example: Tf-idf Weights

Example: Tf-idf Weights

In the classic vector space model proposed by Salton, Wong and Yang the term specific weights in the document vectors are products of local and global parameters. The model is known as term frequency-inverse document frequency model. The weight vector for document d is, where

w_{t,d} = \mathrm{tf}_{t,d} \cdot \log{\frac{|D|}{|\{d' \in D \, | \, t \in d'\}|}}

and

  • is term frequency of term t in document d (a local parameter)
  • is inverse document frequency (a global parameter). is the total number of documents in the document set; is the number of documents containing the term t.

Using the cosine the similarity between document dj and query q can be calculated as:

In a simpler Term Count Model the term specific weights do not include the global parameter. Instead the weights are just the counts of term occurrences: .

Read more about this topic:  Vector Space Model

Famous quotes containing the word weights:

    “What do you believe in?—In this, that the weights of all things must be determined anew.”
    —Friedrich Nietzsche (1844–1900)