The Porter Stemming Algorithm
Documentation: The Porter Stemming Algorithm
The Porter stemming algorithm (or ‘Porter stemmer’) is a process for removing the commoner morphological and inflexional endings from words in English. Its main use is a part of a term normalization process that is usually done when setting up Information Retrieval systems.
The SQLite FTS5 porter tokenizer #
The SQLite FTS5 module uses The Porter stemming algorithm as one of its built-in tokenizer modules: https://www.sqlite.org/fts5.html#tokenizers . fts5vocab virtual tables can be used to demonstrate what the algorithm does to words.
sqlite> CREATE VIRTUAL TABLE temp.stemming_test
...> USING fts5(text, tokenize='porter unicode61');
sqlite> INSERT INTO temp.stemming_test
...> VALUES ('The runner runs and was running every day.');
sqlite> CREATE VIRTUAL TABLE temp.stemming_vocab
...> USING fts5vocab(stemming_test, instance);
sqlite> SELECT * FROM temp.stemming_vocab;
╭────────┬─────┬──────┬────────╮
│ term │ doc │ col │ offset │
╞════════╪═════╪══════╪════════╡
│ and │ 1 │ text │ 3 │
│ dai │ 1 │ text │ 7 │
│ everi │ 1 │ text │ 6 │
│ run │ 1 │ text │ 2 │
│ run │ 1 │ text │ 5 │
│ runner │ 1 │ text │ 1 │
│ the │ 1 │ text │ 0 │
│ wa │ 1 │ text │ 4 │
╰────────┴─────┴──────┴────────╯
The stems produced from “The runner runs and was running every day.” are not all valid English words:
- runs → run
- running → run
- every → everi
- day → dai
- was → wa
The normalization works because queries are also normalized.
Another example:
sqlite> CREATE VIRTUAL TABLE temp.stemming_test
...> USING fts5(text, tokenize='porter unicode61');
sqlite> INSERT INTO temp.stemming_test
...> VALUES ('connect connected connecting connection');
sqlite> CREATE VIRTUAL TABLE temp.stemming_vocab
...> USING fts5vocab(stemming_test, instance);
sqlite> SELECT * FROM temp.stemming_vocab;
╭─────────┬─────┬──────┬────────╮
│ term │ doc │ col │ offset │
╞═════════╪═════╪══════╪════════╡
│ connect │ 1 │ text │ 0 │
│ connect │ 1 │ text │ 1 │
│ connect │ 1 │ text │ 2 │
│ connect │ 1 │ text │ 3 │
╰─────────┴─────┴──────┴────────╯