Vector Space Model - Example: Tf-idf Weights

Example: Tf-idf Weights

In the classic vector space model proposed by Salton, Wong and Yang the term specific weights in the document vectors are products of local and global parameters. The model is known as term frequency-inverse document frequency model. The weight vector for document d is, where

w_{t,d} = \mathrm{tf}_{t,d} \cdot \log{\frac{|D|}{|\{d' \in D \, | \, t \in d'\}|}}

and

  • is term frequency of term t in document d (a local parameter)
  • is inverse document frequency (a global parameter). is the total number of documents in the document set; is the number of documents containing the term t.

Using the cosine the similarity between document dj and query q can be calculated as:

In a simpler Term Count Model the term specific weights do not include the global parameter. Instead the weights are just the counts of term occurrences: .

Read more about this topic:  Vector Space Model

Famous quotes containing the word weights:

    “This is essentially a People’s contest. On the side of the Union, it is a struggle for maintaining in the world, that form, and substance of government, whose leading object is, to elevate the condition of men—to lift artificial weights from all shoulders—to clear the paths of laudable pursuit for all—to afford all, an unfettered start, and a fair chance, in the race of life.”
    —Abraham Lincoln (1809–1865)