How to use DBpedia properties to build a topic hierarchy?

Question

I am trying to build a topic hierarchy by following the below mentioned two DBpedia properties.

1. skos:broader property
2. dcterms:subject property

My intention is to given the word identify the topic of it. For example, given the word; 'suport vector machine', I want to identify topics from it such as classification algorithm, machine learning etc.

However, sometimes I am bit confused as how to build a topic hierarchy as I am getting more than 5 URIs for subject and many URIs for broader properties. Is there a way to measure strength or something and reduce the additional URIs that I get from DBpedia and to assign only the highest probable URI?

It seems there are two questions there.

1. How to limit the number of DBpedia Spotlight results.
2. How to limit the number of subjects and categories for a particular result.

My current code is as follows.

from SPARQLWrapper import SPARQLWrapper, JSON
import requests
import urllib.parse

## initial consts
BASE_URL = 'http://api.dbpedia-spotlight.org/en/annotate?text={text}&confidence={confidence}&support={support}'
TEXT = 'First documented in the 13th century, Berlin was the capital of the Kingdom of Prussia (1701–1918), the German Empire (1871–1918), the Weimar Republic (1919–33) and the Third Reich (1933–45). Berlin in the 1920s was the third largest municipality in the world. After World War II, the city became divided into East Berlin -- the capital of East Germany -- and West Berlin, a West German exclave surrounded by the Berlin Wall from 1961–89. Following German reunification in 1990, the city regained its status as the capital of Germany, hosting 147 foreign embassies.'
CONFIDENCE = '0.5'
SUPPORT = '120'
REQUEST = BASE_URL.format(
    text=urllib.parse.quote_plus(TEXT), 
    confidence=CONFIDENCE, 
    support=SUPPORT
)
HEADERS = {'Accept': 'application/json'}
sparql = SPARQLWrapper("http://dbpedia.org/sparql")
all_urls = []

r = requests.get(url=REQUEST, headers=HEADERS)
response = r.json()
resources = response['Resources']

for res in resources:
    all_urls.append(res['@URI'])

for url in all_urls:
    sparql.setQuery("""
        SELECT * WHERE {<"""
             +url+
            """>skos:broader|dct:subject ?resource 
            }
    """)

    sparql.setReturnFormat(JSON)
    results = sparql.query().convert()

    for result in results["results"]["bindings"]:
        print('resource ---- ', result['resource']['value'])

I am happy to provide more examples if needed.

Answer 1

It seems you are trying to retrieve Wikipedia categories relevant to a given paragraph.

Minor suggestions

First, I'd suggest you to perform a single request, collecting DBpedia Spotlight results into VALUES, for example, in this way:

values = '(<{0}>)'.format('>) (<'.join(all_urls))

Second, if you are talking about topic hierarchy, you should use SPARQL 1.1 property paths.

These two suggestions are slightly incompatible. Virtuoso is very inefficient, when a query contains both multiple starting points (i. e. VALUES) and arbitrary length paths (i. e. * and + operators).

Here below I'm using the dct:subject/skos:broader property path, i.e. retrieving the 'next-level' categories.

Approach 1

The first way is to order resources by their general popularity, e. g. their PageRank:

values = '(<{0}>)'.format('>) (<'.join(all_urls))

sparql.setQuery(
    """PREFIX vrank:<http://purl.org/voc/vrank#>
       SELECT DISTINCT ?resource ?rank
       FROM <http://dbpedia.org> 
       FROM <http://people.aifb.kit.edu/ath/#DBpedia_PageRank>
       WHERE {
           VALUES (?s) {""" + values + 
    """    }
       ?s dct:subject/skos:broader ?resource .
       ?resource vrank:hasRank/vrank:rankValue ?rank.
       } ORDER BY DESC(?rank)
         LIMIT 10
    """)

Results are:

dbc:Member_states_of_the_United_Nations
dbc:Country_subdivisions_of_Europe
dbc:Republics
dbc:Demography
dbc:Population
dbc:Countries_in_Europe
dbc:Third-level_administrative_country_subdivisions
dbc:International_law
dbc:Former_countries_in_Europe
dbc:History_of_the_Soviet_Union_and_Soviet_Russia

Approach 2

The second way is to calculate category frequency a given text...

values = '(<{0}>)'.format('>) (<'.join(all_urls))

sparql.setQuery(
    """SELECT ?resource count(?resource) AS ?count WHERE {
           VALUES (?s) {""" + values + 
    """    }
       ?s dct:subject ?resource
       } GROUP BY ?resource
         # https://github.com/openlink/virtuoso-opensource/issues/254
         HAVING (count(?resource) > 1)
         ORDER BY DESC(count(?resource))
         LIMIT 10
    """)

Results are:

dbc:Wars_by_country
dbc:Wars_involving_the_states_and_peoples_of_Europe
dbc:Wars_involving_the_states_and_peoples_of_Asia
dbc:Wars_involving_the_states_and_peoples_of_North_America
dbc:20th_century_in_Germany
dbc:Modern_history_of_Germany
dbc:Wars_involving_the_Balkans
dbc:Decades_in_Germany
dbc:Modern_Europe
dbc:Wars_involving_the_states_and_peoples_of_South_America

With dct:subject instead of dct:subject/skos:broader, results are better:

dbc:Former_polities_of_the_Cold_War
dbc:Former_republics
dbc:States_and_territories_established_in_1949
dbc:20th_century_in_Germany_by_period
dbc:1930s_in_Germany
dbc:Modern_history_of_Germany
dbc:1990_disestablishments_in_West_Germany
dbc:1933_disestablishments_in_Germany
dbc:1949_establishments_in_West_Germany
dbc:1949_establishments_in_Germany

Conclusion

Results are not very good. I see two reasons: DBpedia categories are quite random, tools are quite primitive. Perhaps it is possible to achieve better results, combining approaches 1 and 2. Anyway, experiments with a large corpus are needed.