quanteda - dfm (by dfm_lookup) doesn't show the columns of terms as example

Question

I am following this tutorial to create a document-feature matrix with features defined by my dictionary. What I have now is an output of two columns giving my document ID and the frequency of all features in my dictionary.

library(lubridate)
library(quanteda)

## subset data 
item7_corpus_subset <- item_7_corpus |> 
    filter(year(filing_date) == year_data) |> 
    head(100) ## edit here, comment if codes work well
    
# tokenize
item7_tokens <- tokens(item7_corpus_subset, 
                       what = "word", 
                       remove_punct = TRUE,
                       remove_symbols = TRUE,
                       remove_numbers = TRUE,
                       remove_url = TRUE) |> 
    tokens_ngrams(n = 1:3)
       
## count words from dictionary
item7_doc_dict <- item7_tokens |> 
    dfm(tolower = TRUE) |> 
    dfm_lookup(dictionary = cyber_dict, levels = 1:3)
print(item7_doc_dict)

## Document-feature matrix of: 100 documents, 1 feature (94.00% sparse) and 13 docvars.
## features
## docs                                        cyber_dict
## 1000015_10K_1999_0000912057-00-014793.txt          0
## 1000112_10K_1999_0000930661-00-000704.txt          0
## 1000181_10K_1999_0001000181-00-000001.txt          0
## 1000227_10K_1999_0000950116-00-000643.txt          0
## 1000228_10K_1999_0000889812-00-001326.txt          0
## 1000230_10K_1999_0001005150-00-000103.txt          0
## [ reached max_ndoc ... 94 more documents ]

I want to see the frequency of each keyword rather than the total frequency of all keywords that I have. I am trying to emulate the tutorial which generated that as:

dfmat_irish_lg <- dfm_lookup(dfmat_irish, dictionary = dict_lg, levels = 1)
print(dfmat_irish_lg)

## Document-feature matrix of: 14 documents, 9 features (19.84% sparse) and 6 docvars.
##                       features
## docs                   CULTURE ECONOMY ENVIRONMENT GROUPS INSTITUTIONS LAW_AND_ORDER RURAL URBAN VALUES
##   Lenihan, Brian (FF)        9     583          21      0           93            11     9     0     19
##   Bruton, Richard (FG)      35     201           5      0           95            14     0     0     14
##   Burton, Joan (LAB)        33     400           6      3           84             6     2     3      6
##   Morgan, Arthur (SF)       56     427          10      0           63            22     2     1     18
##   Cowen, Brian (FF)         16     416          24      0           63             4     8     1     13
##   Kenny, Enda (FG)          26     211           8      1           53            18     0     2      8
## [ reached max_ndoc ... 8 more documents ] 

Answer 1

There were three mistakes:

• tokens_ngrams() is used before dictionary analysis
• there is only one group (key) in the dictionary
• if exclusive = FALSE, all other words are included

Your code should be

require(quanteda)
#> Loading required package: quanteda
#> Package version: 3.2.3
#> Unicode version: 13.0
#> ICU version: 69.1
#> Parallel computing: 8 of 8 threads used.
#> See https://quanteda.io for tutorials and examples.
# create a sample corpus
texts <- c("This is a sample text mentioning cyber attack.",
           "Cybersecurity is important to protect against cyber threats.",
           "The company experienced a data breach due to a cyber attack.",
           "Cyber criminals are becoming increasingly sophisticated.",
           "Protecting against cyber attacks requires a multi-layered approach.")

corpus <- corpus(texts)

dict <- dictionary(list(cyber = c("cyber", "cybersecurity", "cybercriminals"), 
                        attack = c("cyberattack", "data breach", "protect")))

toks <- quanteda::tokens(corpus)

dfmt <- dfm(toks)
dfmt
#> Document-feature matrix of: 5 documents, 31 features (70.97% sparse) and 0 docvars.
#>        features
#> docs    this is a sample text mentioning cyber attack . cybersecurity
#>   text1    1  1 1      1    1          1     1      1 1             0
#>   text2    0  1 0      0    0          0     1      0 1             1
#>   text3    0  0 2      0    0          0     1      1 1             0
#>   text4    0  0 0      0    0          0     1      0 1             0
#>   text5    0  0 1      0    0          0     1      0 1             0
#> [ reached max_nfeat ... 21 more features ]

dfmt_dict <- dfm_lookup(dfmt, dictionary = dict, levels = 1, 
                         exclusive = TRUE, capkeys = FALSE)
dfmt_dict
#> Document-feature matrix of: 5 documents, 2 features (40.00% sparse) and 0 docvars.
#>        features
#> docs    cyber attack
#>   text1     1      0
#>   text2     2      1
#>   text3     1      0
#>   text4     1      0
#>   text5     1      0