The TagRec Framework as a Toolkit for the Development of Tag-Based Recommender Systems

Recommender systems have become important tools to support users in identifying relevant content in an overloaded information space. To ease the development of recommender systems, a number of recommender frameworks have been proposed that serve a wide range of application domains. Our TagRec framework is one of the few examples of an open-source framework tailored towards developing and evaluating tag-based recommender systems. In this paper, we present the current, updated state of TagRec, and we summarize and reflect on four use cases that have been implemented with TagRec: (i) tag recommendations, (ii) resource recommendations, (iii) recommendation evaluation, and (iv) hashtag recommendations. To date, TagRec served the development and/or evaluation process of tag-based recommender systems in two large scale European research projects, which have been described in 17 research papers. Thus, we believe that this work is of interest for both researchers and practitioners of tag-based recommender systems.


INTRODUCTION
Recommender systems aim to predict the probability that a speci c user will like a speci c resource. erefore, recommender systems utilize the past user behavior (e.g., resources previously consumed by this user) in order to generate a personalized list of potentially relevant resources [32]. Popular application domains of recommender systems include online marketplaces (e.g., Amazon and Zalando), movie and music streaming services (e.g., Net ix and Spotify), job portals (e.g., LinkedIn and Xing), and social tagging systems (e.g., BibSonomy and CiteULike).
Social tagging systems bear particularly great potential for recommender systems as, by nature, they produce a vast amount of user-generated resource-annotations (i.e., tags). us, possible use cases of these tag-based recommender systems include the suggestion of resources to extend a user's set of bookmarks [4,38] and the Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for pro t or commercial advantage and that copies bear this notice and the full citation on the rst page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permi ed. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speci c permission and/or a fee. Request permissions from permissions@acm.org. Conference'17, Washington, DC, USA © 2016 ACM. 978-x-xxxx-xxxx-x/YY/MM. . . $15.00 DOI: 10.1145/nnnnnnn.nnnnnnn suggestion of tags to assist in the annotation of these bookmarks. e la er one is known as the eld of tag recommendations [13]. Over the past years, various recommendation frameworks and libraries have been developed in order to support the development and evaluation of recommender systems (see Section 4). While these frameworks cover a wide range of application domains, to the best of our knowledge, an open-source recommendation framework to design and evaluate tag-based recommender systems was still lacking.
erefore, in 2014, we have started developing TagRec, a standardized tag recommender benchmarking framework [19]. In the initial development phase of the framework, we mainly focused on evaluating tag recommendation algorithms. In 2015, the framework was extended by including resource recommendation algorithms that are based on social tagging data [36]. e aim of this paper, however, is to present the current, updated state of TagRec. is includes the extension of the framework for (i) the analysis of tag reuse practices [21], (ii) the evaluation of tag recommendations in real-world folksonomy and Technology Enhanced Learning se ings [16,20], and (iii) hashtag recommendations in Twi er [22].
Apart from that, we provide an updated framework description (see Section 2) as well as a summary of use cases in the eld of recommender research that have been completed using TagRec (see  Section 3). Research areas encompass tag recommendations, resource recommendations, recommendation evaluation and hashtag recommendations. To date, TagRec has served the recommender development and/or evaluation processes in two large-scale European research projects, which have been published in 17 research papers. We conclude the paper with a discussion on future work and potential improvements of the framework (see Section 5).
We believe that our work contributes to the rich portfolio of technical frameworks in the area of recommender systems. Furthermore, this paper presents an overview of use cases which can be realized with TagRec, and should be of interest for both researchers and developers of tag-based recommender systems.

TAGREC
TagRec is a Java-based recommendation framework for tag-based information retrieval se ings. It is open-source so ware and freely available via our Github repository 1 . e Github page also contains a detailed technical description on the usage of the framework. Figure 1 illustrates TagRec's system architecture. e framework consists of (i) a data processing component, which processes data sources, (ii) a data model and analytics component, which enables access to the processed data, (iii) recommendation algorithms, 1 Figure 1: System architecture of TagRec. Here, the data processing component processes data sources in order to create a data model and data analytics.
en, this data model is used by recommendation algorithms to create recommendation results that are either forwarded to an evaluation engine or to a client application.
which calculate recommendations, (iv) an evaluation engine, which evaluates the algorithms, and (v) recommendation results, which can be passed to a client application. e mentioned components are described in more detail in the remainder of this section. Apart from that, we describe practical aspects of the framework that should be helpful when implementing and/or evaluating a recommendation algorithm. Finally, Table 1 provides an overview of the supported datasets, recommendation algorithms and evaluation metrics.

Data Processing.
e data processing component is responsible for parsing and processing external data sources. Currently supported datasets are listed in Table 1. ese datasets serve a wide range of application domains such as social bookmarking systems, learning environments, microblogging tools and music/movie sharing portals. e set of datasets can easily be extended by implementing custom data pre-processing strategies.
Data Model and Analytics. e data model is created based on described data processing steps and provides an object-oriented representation of the data in order to ease the implementation process of a novel recommendation algorithm. us, it enables easy access to the entities in the datasets via powerful query functionality (e.g., get the set of tags a user has used in the past). Furthermore, the data model of TagRec is connected to Apache Solr 2 and thus, enables fast access to content-based data of entities.
Another role of this component is the provision of basic data analytics functionality to get a be er understanding of the dataset characteristics. For example, dataset statistics, such as the total number of distinct tags or the average number of bookmarks per user, can be retrieved.
Recommendation Algorithms. TagRec contains a wide range of recommendation algorithms (see Table 1). As later described in 2   Section 3, algorithms for tag recommendations, resource recommendations and hashtag recommendations are provided. ese algorithms can be used as baseline approaches for a newly implemented algorithm.
e complete list of all variants of these algorithms is provided on the TagRec's Github page.
A key contributions of TagRec is that next to well-established approaches, such as Collaborative Filtering, MostPopular and Factorization machines, it also encompasses approaches based upon cognitive models of information retrieval, human memory theory and category learning. In [20], it has been shown that these cognitive-inspired approaches achieve high prediction accuracy estimates in comparison to classic recommendation algorithms. Besides, the algorithms have demonstrated their suitability for sparse datasets such and narrow folksonomies. Evaluation Engine. e evaluation engine quanti es the quality of implemented recommendation strategies by applying a rich set of evaluation metrics as listed in Table 1. One drawback of most recommendation evaluation frameworks is their focus on accuracy and ranking estimates, which restricts the evaluation to the performance of recommender systems [2].
To ll this gap, TagRec supports a variety of evaluation metrics to also o er indicators for diversity, novelty, runtime performance and memory consumption of algorithms. For evaluating an algorithm, TagRec has to be provided with three parameters, where the rst one speci es the algorithm, the second one speci es the dataset directory and the third one speci es the le name of the dataset sample. For example, java jar tagrec.jar cf bib bib sample runs Collaborative Filtering on a sample of the BibSonomy dataset. e calculated metrics are then either wri en to a "metrics" le or printed to the console.
Recommendation Results. As indicated in Figure 1, the algorithms' recommendation results can be either forwarded to the evaluation engine to retrieve evaluation metrics or to a client application for further processing (e.g., visualization). e KnowBrain tool [7] is an example of such a client application. It is an open source social bookmarking tool, which has been extended to cater the requirements of tag recommender evaluations in online se ings.
A screenshot of KnowBrain's graphical user interface is shown in Figure 2. It enables the bookmarking of Web links and their annotations by (i) selecting from a pre-de ned set of categories, and by (ii) assigning a variable number of tags. e user's tagging process is supported by a list of recommended tags that are selected based on

Tag recommendations
Research papers Model of human categorization [17,23,35] Activation processes in human memory [18,21,24,37] Informal learning se ings [5][6][7] Resource recommendations Research papers A ention-interpretation dynamics [15,34] Tag and time information [27,28] Recommendation evaluation Research papers Real-world folksonomies [20] Technology enhanced learning se ings [16] Hashtag recommendations Research papers Temporal e ects on hashtag reuse [22]  algorithms of the TagRec framework. e elicitation of categories allows for semantic context-based recommendation algorithms such as 3Layers [35]. Furthermore, the comparison of actually used tags with recommended tags gives insights into the online performance (i.e., user acceptance) of recommendation strategies.

USE CASES
In this section, we describe use cases that have been implemented using the TagRec framework. To date, TagRec supported the recommender development and/or evaluation processes in two large-scale European research projects. Results have been published in 17 research papers (see Table 2).

Tag Recommendations
Tag recommendation systems assist users in nding descriptive tags to annotate resources. In other words, given a speci c user and a speci c resource, a tag recommendation algorithm predicts a set of tags a user is likely to apply in annotating the resource [13]. Within this context, TagRec was used for the creation of (i) cognitive-inspired algorithms, and (ii) the evaluation of approaches suitable for formal and informal learning se ings.
Tag Recommendations Using a Model of Human Categorization. In [17,23,35], the authors introduced a tag recommendation algorithm based on the human categorization models ALCOVE [26] and MINERVA2 [12]. is algorithm is called 3Layers and simulates categorization processes in human memory. erefore, the categories assigned to a given resource, which a user is going to annotate, are matched against already annotated resources of this user. Based on this matchmaking process, a set of tags associated with semantically related resources is recommended.
Since TagRec enables to link a list of categories to a resource, it supported the development of 3Layers by providing functions for analyzing and deriving category information of resources (e.g., via LDA topic modeling [25]).
Utilizing Activation Processes in Human Memory. Activation processes in human memory describe the general and contextdependent usefulness of information. It was shown that these processes (especially usage frequency, recency and semantic context) greatly in uence the reuse probability of tags [21]. Based on this, a set of time-aware tag recommendation approaches (see [18,24,37]) was developed that utilize the activation equation of the cognitive architecture ACT-R [1]. erefore, TagRec was used to analyze the timestamps of tag assignments and to calculate tag co-occurrences for re ecting the semantic context of social tagging. Furthermore, TagRec enabled the hybrid combination of the components of the model (e.g., combining time-aware and context-aware recommendations).
Tag Recommendations in Informal Learning Settings. In the course of the European-funded project Learning Layers 3 , which aims at supporting informal learning at the workplace, tag recommendations were used to support the individual user in nding descriptive tags and the collective in consolidating a shared tag vocabulary. ese tag recommendations were used in two tools: (i) the Dropbox-like environment KnowBrain [7], and (ii) the Sensemaking interface Bits & Pieces [6].
To achieve this, TagRec was integrated as a tag recommendation library into the Social Semantic Server [5], which was used as the technical back-end for KnowBrain and Bits & Pieces. is shows that TagRec cannot only be used as a standalone tool but also as a programming library (or toolkit) to include recommendation functionality in existing so ware. A similar approach will be followed in another European-funded project called AFEL 4 , which engages in design and development of analytics for everyday learning.

Resource Recommendations
Resource recommender systems suggest potentially relevant web items (e.g., movies, books, learning resources, URLs, etc.) to users. Most of these recommender systems are based on Collaborative Filtering (CF) techniques, which aim to calculate similarities between users to suggest the most suitable web resources to them [33]. TagRec was applied to support and improve the development of CF approaches in tag-based online environments.
Mimicking Attention-Interpretation Dynamics. Seitlinger et al. [34] introduced the rst version of a CF-based recommendation approach that takes into consideration non-linear user-artifact dynamics, modeled by means of SUSTAIN. SUSTAIN (Supervised and Unsupervised STrati ed Adaptive Incremental Network) is a exible network model of human category learning that is thoroughly discussed in [29]. It assumes that learning is a dynamic process that takes place through the encounter of new examples (e.g., Web resources). roughout the learning trajectory, categories emerge and learners' a ention foci shi . In [15], an advanced, adapted version of the initial approach was presented and analyzed in detail. e resulting approach SU ST AI N + CF , rstly applies CF to calculate the most suitable resources for a user, and secondly reranks this list depending on a user's category learning model (i.e., SUSTAIN's user model). e algorithm has been implemented and developed within the TagRec framework. Features of the framework allowed for continuous evaluation and analysis of single factors of the model and with it, the associated change of recommendation performance. With this data, it was possible to gain deeper insight into the algorithmic approach and its parameters and thus, to further adapt the model to the requirements of our application area.
Resource Recommendations using Tag and Time Information. In [27], the Collaborative Item Ranking Using Tag and Time Information (CIRTT) approach was presented. CIRTT uses Collaborative Filtering to identify a set of candidate resources and re-ranks these candidate resources by incorporating tag and time information. is is achieved via the Base-Level-Learning (BLL) equation, which is one component of ACT-R's activation equation [1].
Since TagRec contains a full implementation of the activation equation, it could be easily adapted for the task of resource recommendations as well. Apart from that, TagRec was used to compare CIRTTT to other related resource recommendation methods (e.g., [40]). Another study of the recency e ects in Collaborative Filtering recommender systems was provided in [28].

Recommendation Evaluation
One of the most challenging tasks in the area of recommender systems, is the reproducible evaluation of recommendation results [11]. TagRec aims to support this process by providing standardized data processing methods, baseline algorithm implementations, evaluation protocols and metrics.
Evaluating Tag Recommendations in Real-World Folksonomies. Because of the sparse nature of social tagging systems, most tag recommendation evaluation studies were conducted using pcore pruned datasets. is means that all users, resources and tags, which do not appear at least p times in the dataset, are removed.
is clearly does not re ect a real-world folksonomy se ing as shown by [8].
To overcome this problem, TagRec was used in [20] to compare a rich set of tag recommendation algorithms using a wide range of evaluation metrics on six un ltered social tagging datasets (i.e., Flickr, CiteULike, BibSonomy, Delicious, LastFM and MovieLens). e results showed that the e cacy of a recommendation algorithm greatly depends on the given dataset characteristics, and that cognitive-inspired approaches provide the most robust results, even in sparse data folksonomy se ings.
Comparing Recommendation Algorithms in Technology Enhanced Learning Settings. Kopeinik et al. [16] is another example of using TagRec for the evaluation of a variety of algorithms on di erent o ine datasets. e paper focused on technologyenhanced formal and informal learning environments, where due to fast changing domains and characteristic group learning settings, data is typically sparse. e evaluation was divided in two se ings, the performance of (i) resource recommendation strategies, and (ii) tag recommendation strategies. In both cases, the authors compared the recommendation accuracy of a number of computationally-inexpensive recommendation algorithms on six ofine datasets retrieved from various educational se ings (i.e., social bookmarking systems, social learning environments and massive open online courses). Investigated approaches are either state-ofthe-art recommendation approaches, or strategies that have been explicitly suggested in the context of TEL systems.
To address the goals of this study, the TagRec framework already provided a wide range of required functionality such as the implemented data processing component, evaluation metrics and state-of-the-art algorithms. In the context of this research paper, it was further extended by a couple of algorithms that are considered particularly relevant to learning se ings and by additional statistics, which were needed to interpret evaluation results properly.

Hashtag Recommendations
Over the past years, hashtags have become very popular in systems such as Twi er, Instagram and Facebook. Similar to social tags, hashtags are freely-chosen keywords to categorize resources such as Twi er posts (i.e., tweets). One of the biggest advantages of hashtags is that they can be easily used by integrating them in the tweet text. Unsurprisingly, this has led to the development of hashtag recommendation algorithms that aim to support users in applying the most descriptive hashtags to their tweets [39].
Temporal E ects on Hashtag Reuse. In [22], a time-dependent and cognitive-inspired hashtag recommendation approach was proposed. In this paper, temporal e ects on hashtag reuse in Twi er have been analyzed with the help of TagRec in order to design a hashtag recommendation approach, which utilizes the BLL equation of the cognitive architecture ACT-R [1]. erefore, TagRec was extended with functions to access Apache Solr (see Section 2), which enables the content-based analysis of tweets using TF-IDF (see [22]).

RELATED WORK
In recent years, a multitude of recommendation engines have been created and made available either for commercial use or as opensource systems. In [14], a well-structured overview of such approaches is presented. e author di erentiates between So wareas-a-Service (i.e., SaaS), non-SaaS, open-source, academic and benchmarking recommender systems. We consider open-source and benchmarking frameworks that implement recommendation strategies ing academic purposes as most relevant to our work.
A considerable contribution to this area is LibRec 5 , a Java-based library that, so far, comprises around 70 resource recommendation algorithms and evaluation modules [10]. Another Java-based, opensource framework is RankSys 6 , which focuses on the evaluation of ranking problems and supports the investigation of novelty as well as diversity for academic research [3], which is re ected in its design (e.g., data input interfaces work with a triple of user, item and features).
Other examples of open-source recommender so ware are My-MediaLite 7 , an item recommender library that focuses on rating and ranking predictions in collaborative ltering approaches [9], CARSKit 8 , a recommendation library speci cally designed for contextaware recommendations, and Tag Recommender 9 , a so ware component that implements Tensor Factorization models for personalized tag recommendations in C++ [31].
However, to the best of our knowledge, an open-source recommender framework that implements a wide range of tag and resource recommendation algorithms (including a number of cognitiveinspired approaches) for the design and evaluation of personalized tag-based recommendation strategies was still missing.

CONCLUSION AND FUTURE WORK
In this paper, we presented the TagRec framework as a toolkit for the development and evaluation of tag-based recommender systems. TagRec is open-source so ware wri en in Java and can be freely downloaded from Github. e framework consists of ve components: (i) a data processing component, which processes data sources, (ii) a data model and analytics component, which enables access to the processed data, (iii) recommendation algorithms, which calculate recommendations, (iv) an evaluation engine, which evaluates the algorithms, and (v) recommendation results, which can be passed to client applications.
Apart from that, we summarized various use cases realized with TagRec from the elds of tag recommendations, resource recommendations, recommendation evaluation and hashtag recommendations. To date, TagRec supported the development and/or evaluation process described in 17 research papers. Speci cally, our framework was used for the realization of recommendation algorithms based on models of cognitive science. In these papers, it was shown that the cognitive-inspired approaches provided the most robust results, even in sparse data folksonomy se ings.
We believe that TagRec extends the already rich portfolio of recommender frameworks with a toolkit that is speci cally tailored to t tag-based se ings. Furthermore, the presentation of TagRec's use cases should be of interest for both researchers and developers of tag-based recommender systems.
Limitations & future work. Currently, one limitation of TagRec is that the data access is not standardized.
us, social tagging data is accessed from folksonomy les, whereas resource-related metadata (e.g., tweet content) is accessed from Apache Solr. us, our rst plan for future work is to implement a mechanism that integrates all data into Apache Solr. Apart from that, we want to further work on the stability and code quality of the framework. For example, we want to enhance the build and dependency management of the so ware using Apache Maven 10 .