Automatic allocation of papers to reviewers

Automatic allocation of papers to reviewers

At this time and age, conferences often have large number of participants and there are equally a high number of papers being submitted for review. Submissions at some conferences have increase from 150 in 1985 to over 400 in 2004 (Franklin et al, 2004). Assigning reviewers to submitted papers are often completed under a lot of pressure and limited time, with a very large number of submissions arriving close to the announced deadline. A manual allocation process is sometimes used to assign papers to reviewers. The problem with this is that it is a stressful, time-consuming task and only successful if the person assigning the papers knows the reviewers and their respective interests and expertise.

As the scope of conferences increase, the functionality for automatic assignment becomes an important part of every conference management system (Kalmukov, 2006). Assigning papers to reviewers involves some further considerations beyond matching papers and reviewers. A CMS will have to implement conflicts of interest (COI) detection and not assign any reviewers their own papers or those of close colleagues. Additionally, there is a need to balance the review assignments to ensure that no reviewer is overworked just because that person is an appropriate choice for many papers, and that each paper gets assigned a minimum number of reviewers (Dumais and Nielson, 1992). Papagelis et al (2005) also noted that the automatic assignment of papers is done with an algorithm that looks for matches between paper topics and reviewers interests, considers bids by reviewers on particular papers, implements COI detection and tries to balance workload. Sometimes however, according to Pesenhofer et al (2008), "most of the PC members neither bid nor choose their interests so that standard algorithms fail in computing a proper assignment. This is particularly due to the fact that a bidding process for 200 or more papers is a notoriously time consuming task".

One of such algorithm is the Latent Semantic Indexing. This was used by Dumais and Nielson (1992) and initially described by Deerwester et al (1990). Using the algorithm, Dumais and Nielson compared reviewers' abstract with submissions, ranking the submissions from most to least similar with the reviewer. The algorithms also need the reviewers to choose from a list of relevant topics where they are interested in and bid for special papers. If the PC members don't bid or choose their interests, the algorithms may fail in computing a proper assignment.

The algorithm designed by Kalmukov provides equal distribution of papers and is claimed to run faster than other similar algorithms. If a paper has one or more keywords that are related to a reviewer, then that paper will certainly be assigned. This is done so far as the numbers of papers a reviewer can assess are not beyond the limit allowed per reviewer. A CMS implementing this algorithm would have to accept only a finite number of keywords, ensured by using selection fields like checkboxes in the web forms. Kalmukov further explains that the algorithm uses a number-of-papers-per-reviewer calculation to ensure equal distribution of assignments. A complex similarity factor is now used to find the most suitable reviewer for all the papers. In doing this, the similarity factor relates the set of keywords describing the papers with the set of keywords chosen by the reviewers.

In a study done by Jin (2003), after the bidding process is over, an algorithm sorts all submitted papers into categories. According to them, all papers have to fall into one of the four categories; high, average, low and no interest categories represented as H, V, L and N respectively. Also no paper can be in more than one group simultaneously. The algorithm then sorts the groups in ascending order and in each group sorts the papers based on the number of selections by the reviewers.

Conflict of interest (COI) detection

One of the main concerns of the PC Chair is to identify members of the program committee that may have a conflict of interest in reviewing a specific paper. According to Papagelis et al (2005), scientists usually submit papers to a conference that they serve as reviewers. They also conclude that PC members are usually associated with authors of submitted papers, either because they are occupied in the same institute or project or because they have co-authored an article in the past. These have the potential to remove confidentiality from the review process thereby affecting the reputation of the conference. There are several approaches to detecting conflict of interests. In a study by Pesenhofer et al (2006), the potential conflict of interest detection was performed based on the occurrence of the last name of a program committee member in the author's line of a submission and the existence of parts from the PC members email domain in the submissions author field. If the PC members email and the submission come from the same domain, then the system will register a potential conflict of interest. Papagelis and his team designed mechanisms that recommend potential conflicts according to same institute appointment or previous co-authorship appointment techniques. The same institute appointment technique works by running string comparison on the email accounts of reviewers and that given by authors as they submit papers. A gradual string matching algorithm is applied that compares the different parts of the email accounts. This is better illustrated in the figure below

The previous co-authorship technique tries to identify pairs of PC members and authors of submitted papers that have co-authored one or more papers in the past. The data for this technique comes from a co-authors index as compiled by the DBLP Computer Science Bibliography. The DBLP server maintained at Universitt Trier provides information such as article title, author and co-author, if any, of major computer science journals and proceedings (Jin, 2003). The system then scans the set of paper authors and the set of reviewers' co-authors to identify matches that define potential conflicts. Matches are based on string comparison of their first and last names. Pesenhofer et al did not include this technique in their study but acknowledged that their system still missed some COI detection because of its non-inclusion. EDAS - a CMS, checks for conflicts of interest based on declarations of possible conflicts by the PC members while bidding for papers (Aleman-Meza et al 2006). They designed an algorithm for COI detection that works as follows. First, it finds all semantic associations between two entities, the reviewer and the author of a paper to be reviewed. Second, each of the semantic associations found is analyzed by looking at the weights of its individual relationships. Since each semantic association is analyzed independently of the others, all directions of the different relationships are eventually considered by the algorithm.

Session compilation

After the review process is over and the papers to be used have been chosen, the next step is to design the sessions for the conference. This is done by grouping the papers according to theme and each theme therefore becomes a session. The speakers in that session will include the authors of the selected papers. In the submission phase, the authors usually have to choose a certain number of research topics that are talked about in their papers. It may also be required, during the submission of the paper, to select keywords from a pre-defined list that highlight the topic focused on by the paper. Using the availability of this information, the technical committee can create thematic sessions. The sessions can either be in parallel especially when a huge amount of papers should be presented during the conference or in sequential order with no overlaps.

Some CMSs have added functionality that automates this process by using algorithms. Pesenhofer et al (2008) describe a process which makes use of clustering methods with a scheduling algorithm. They proposed a system to cluster the accepted papers with a hierarchical clustering algorithm to form a tree-like cluster known as a dendrogram. This approach creates a top down or bottom up ordering or the papers and this helps to create the sessions. According to them, papers down in the hierarchy are more equal to the linked papers than those up in the hierarchy.

Antonnia et al (2003) designed a system that performs information retrieval by content. They described other information retrieval by content methods including term-based and text clustering methods. Their prototype matching system assists the conference organizers to automatically establish semantic similarities among papers and allocate them into common themes. On the other hand, the system assists the attendees to retrieve the papers from the conference proceedings based on their content similarities. According to them, system aims to retrieve the documents that contain the same meaning from the entire document collection. The prototype-matching system analyzes a document collection structure and thus is domain adjusting. The system consists of three parts; document collection preprocessing and encoding, document processing and matching, and document retrieval.

Security and system integrity

Conference management systems are designed to store users' personal information and in almost all cases, financial details. They also store research papers which are the result of years of painstaking and expensive research. For systems like these, security and integrity of the system should be a top priority, if not the top priority. Many systems studied implement a decent level of security, by restricting the user accessibility level and protecting the database from direct access.

In a CMS described by Levacic et al, login protection is dependent on the server used by the system. If a user attempts to login to the system, the server searches the database for login information and uses the same to authenticate the user or otherwise decline the login attempt. Every login sets global variables and a role specific menu is generated based on the user role and these variables. In a situation where two or more users with different access levels have to perform the same action, the URL of the user with higher access rights is differentiated. This allows the higher role to access the action of the lower role but not vice-versa. The system ensures security of the registration and booking processes by implementing Secure Sockets Layer (SSL) to encrypt the financial details and other personal information.

De Troyer and Casteleyn implement security in their system by defining what they termed as a security class hierarchy which classifies visitors according to the security needs of the system. The classifications with the security requirements in brackets are; registered users (logging in), pre-registered users (logging in and confirming their registration) and not-registered users (register). According to their design, pre-registered users are a subclass of registered users because they need the extra functionality of confirming their registration. Also, a pre-registered user becomes a registered user by confirming their registration, while a not-registered user becomes a registered user by registering. This implementation is vague as they do not give the relationship between the pre-registered user and the not-registered user. The natural flow would seem to be a transition from being a not-registered user who registers to become a pre-registered user who confirms the registration to become a registered user who can then access more sections of the system after their login is authenticated.

Halvorsen et al (1998), in their discussion of the CMS known as ConfMan, noted that the system uses the HTTP login and password authentication mechanism of the host server to prevent database access. Protection is provided by using access control lists to determine which users or processes have access to certain objects as well as the operations that can be performed on those objects. ConfMan security is implemented in the database. They stated that for all login requests, a query is sent to the database to check the user login from the HTTP authentication against the list of trusted people. The user can proceed if the authentication query finds a match, otherwise an error message is displayed, and access is denied.

User role definition

In order to prevent users from having unauthorised access to certain sections of a CMS, user roles are defined for the system. Processes like online registration, viewing conference program and call for paper have to be accessible for everybody. On the other hand, review rating and comments, submitted papers must only be accessible to authorized users, as determined by the organizers. In some systems, users can be categorized as registered, pre-registered and not-registered users (De Troyer and Casteleyn, 2001). They defined pre-registered users as a sub-class of the registered users because pre-registered users need some extra functionality such as confirming their registration. Some other systems categorize users differently. Niomanee and Limpiyakorn (2009) assigned 5 user roles to their system. These are administrator, program committee (PC) chair, PC members or reviewers, authors and general users. In the CMS specified by Ciancarini et al (1999), PC members and reviewers are different roles. Also, they have not defined administrator as a role and even introduce the editor of proceedings role. Halvorsen et al (1998) have identified four user roles including organizers, PC members, authors and participants. According to them, organizers include general chairs, program chairs, publicity chairs, treasurer, and local arrangement group.

A typical CMS should have the following user roles; system administrator, PC chair, reviewers or PC members, authors, participants and visitors. The roles are defined as follows;

System administrator: manages conference data, does user administration, activates accounts (Levacic et al, 2005), configure initial context of system, edits and updates configuration, configures sending notification e-mails (Noimanee et al, 2009) and other system maintenance functions.

PC chair: opens registration and paper submission (call for paper), assigns reviewers to papers, disables paper submission after deadline, decides on acceptance of paper, sends notification e-mails to other users (Popovici et al, 2006), checks review status and general co-ordination and monitoring.

Reviewers: sets up topics of interests, responsible for reviewing and rating papers, download papers that allocated to him/her to review, and evaluate and debate on papers (Huang et al, 2008) and makes recommendation for final decision.

Authors: uploads paper, checks review status and views the final result, registers for a conference and uploads the final version of accepted paper and slides (Huang et al, 2008)

Participants: authors and other persons who are interested in this field of research. The role includes all persons registered for the conference (Pesenhofer et al, 2006).

Visitors: non-registered users just browsing the site for general information.

Please be aware that the free essay that you were just reading was not written by us. This essay, and all of the others available to view on the website, were provided to us by students in exchange for services that we offer. This relationship helps our students to get an even better deal while also contributing to the biggest free essay resource in the UK!