Web-based Conference Management System
The process of organizing a conference is time consuming if done manually. This project will automate the processes involved and produce a robust, efficient, secure and inexpensive system. A web-based system will infinitely enhance the management of activities that involve plenty of participants. This is an approach to implementing a CMS using web service technology and business logic to enhance the workflows of conference management tasks. The system should enable users to register, receive a user name and password, pay the registration fee, book accommodation and upload files of contributions. The system should also enable program committee members to send contributions to sets of reviewers for evaluation and ensure that reviewers get papers they are interested in. A literature review was conducted on the theme above and the results are given in the sections following.
The process of conducting the literature review consists of several activities: searching, obtaining, assessing, reading, critically evaluating and writing a critical review (Oates, 2006). The review method section is divided into some of these activities. It must be noted that in the actual process, these activities are not sequential but are very often recurrent.
II. Review method
The steps leading to performing the main literature review started with a meeting with the project supervisor even before the start of the research methods module. This meeting set the pace for the rest of the project work. The main purpose was to get a proper understanding and a more experienced opinion on the project theme. The meeting provided information on what the project theme is really about, where to search for and what types of literature would be best searched for. The supervisor suggested searching for at least 10 research papers on the topic. The first search was done using Google search engine. The very first string used was "web-based conference system". The results were links to several conference management systems but not any research papers on the theme. After searching without any successes, the situation was reported to the supervisor who provided 4 research papers based on the topic. The supervisor suggested that on modifying the search parameters being used. The initial search string was appended with the phrase "+ paper". This produced a much higher number of positive returns that the initial string. IEEE Xplore, ACM, Web of Science, CiteSeer, GoogleScholar, and ScienceDirect were all used to search for the string. IEEE Xplore produced particularly frustrating results because only the abstracts of the papers found could be accessed.
Many other search strings were used on these search engines. Other strings used include: 'distributed web-based conference system + paper', 'a novel approach forweb-based conferencemanagementsystem', 'virtual conference management system', 'web-based conference management system', 'conference management system'. Changing the search strings was increasing producing more results. An additional 5 relevant papers were obtained from this method. Another practice adopted for searching was using the suggested keywords from the papers as search strings. This helped to narrow down the search range and produced more specialized papers. Additionally, as advised by the supervisor, searches were performed for the references found in the obtained papers. As more papers were found, searches were performed on their references, iteratively. This process produced many of the very relevant papers used to perform the literature review. The citation index facility of the specialized search engines was also used to pick up other publications relevant to the found publications. Searches of keywords and book and paper titles were done on the Durham University library database but this yielded only one book.
Several approaches were taken to assess the credibility of the papers chosen. To ensure a certain level of credibility, the use of generic search engines to search for relevant publications was discontinued. For the papers obtained, their titles were cross searched in the other specialized engines. In some cases, there were results and in others there were no matches. Another approach used was to check if the papers were cited by others and the number of citations each paper had. This was done using the citation index facility of the engines. Other checks include using the generic search engines to check that the conferences some of the papers were delivered actually held. Also the number of years such conferences had been taking place was checked, as older conferences were more likely to be more credible. The title of a publication was also a factor that was considered. For documents in PDF format, some of the engines give a link at the side of the search results. In such cases, the choice was to go for those with the domain .edu (for example psu.edu, stevens.edu, umich.edu etc.). Some of the results also had links to Durham ConneXions, the Durham University library catalogue and subsequently the ScienceDirect website. The chosen publications were sent to the supervisor who gave approval.
C. Reading and Evaluating
The first step towards studying the publications was to quickly read the abstract. More information was obtained by quickly reading through the papers to pick out issues the writers paid a lot of time and space to. If a writer is to spend a lot of time on a topic, it most likely means that is important to the writer. Also, section headings were looked at to quickly pick out more details about the publication. Evaluating the texts meant determining which areas stood out to be the key issues or challenges peculiar to the field of study. A major yardstick for judging this is the repeated occurrence of a topic in several of the papers. After identifying the challenges, the methods or strategies employed by the writers to solve these were researched, from the same papers or by searching out papers more specific to those issues. In order to achieve an objective, unbiased evaluation, all papers were approached with the sole purpose of finding out the views of the writers on the issues identified. Having duly completed the assessment process, the papers that form the core references were chosen because of the extent of information given by the writers, how easy it was to understand the writers and the extent to which it cited other papers and vice-versa. There was a recommendation from the supervisor to look at web-based conference systems. The task was to evaluate these and understand their features. Systems looked at include MyReview1, Microsoft CMT2, CMS-Plus3, ConfMaster4, ConfTool5 and EasyChair6.
D. Writing the review
The review was written around concepts and not authors (Oates, 2006; Webster and Watson, 2002). The review identified the key concepts and challenges and talked about the approaches different authors have taken to address those issues. It is structured with headings and sections that outline and describes the work of others in these areas. In other words the literature was generated by discussing each concept as it relates to the study done by other authors.
A. Automatic Assignment of Papers to Reviewers
At this time and age, conferences often have large number of participants and there are equally a high number of papers being submitted for review. Submissions at some conferences have increased from 150 in 1985 to over 400 in 2004 (Franklin et al, 2004). Assigning reviewers to submitted papers are often completed under a lot of pressure and with limited time. It is usual to have a very large number of submissions arriving close to the announced deadline. Some systems provide for a manual allocation process to assign papers to reviewers. The problem with this is that it is a stressful, time-consuming task and only successful if the person assigning the papers knows the reviewers and their respective interests and expertise. As the scope of conferences increase, the functionality for automatic assignment becomes an important part of every conference management system (CMS) (Kalmukov, 2007).
When assigning papers to reviewers, there are some further considerations beyond matching papers and reviewers. Conflicts of interest (COI) detection has to be done so that any reviewers are not assigned their own papers or those of close colleagues. The number of papers per reviewer has to be almost equal to ensure that no reviewer has too much to do from being a good match for many papers, and that each paper gets assigned a minimum number of reviewers (Dumais and Nielson, 1992). Papagelis et al (2005) also noted that the automatic assignment of papers is done with an algorithm that looks for matches between paper topics and reviewers interests, considers bids by reviewers on particular papers, implements COI detection and tries to balance workload.
An algorithm known as Latent Semantic Indexing (LSI) was used by Dumais and Nielson (1992) though initially described by Deerwester et al (1990). Using the algorithm, Dumais and Nielson compared reviewers' abstract with submissions, ranking the submissions from most to least similar with the reviewer. The algorithm also needed the reviewers to choose from a list of relevant topics where they are interested in and bid for special papers. If the PC members do not make a bid or choose their interests, the algorithm may fail in computing a proper assignment. Sometimes "most of the PC members neither bid nor choose their interests so that standard algorithms fail in computing a proper assignment." (Pesenhofer et al 2008). The results of the experiment using the LSI algorithm were not particularly impressive. They tried another method, the n of 2n method. Using this method resulted in substantially higher mean ratings and performed better than the LSI method and human experts.
The algorithm designed by Kalmukov provides equal distribution of papers and is claimed to run faster than other automatic assignment algorithms. The author also gave a guarantee that if a paper has one or more keywords that are related to a reviewer, then that paper will certainly be assigned. This is done so far as the numbers of papers a reviewer can assess are not beyond the limit allowed per reviewer. A CMS implementing this algorithm would have to accept only a certain number of keywords, ensured by using selection fields like checkboxes in the web forms. Kalmukov further explains that the algorithm uses a number-of-papers-per-reviewer calculation to ensure equal distribution of assignments. A complex similarity factor is now used to find the most suitable reviewer for all the papers. In doing this, the similarity factor relates the set of keywords describing the papers with the set of keywords chosen by the reviewers.
In a study done by Jin (2003), in a CMS known as Confsys, an algorithm sorts all submitted papers into categories. This categorization is based on the result of a bidding process where reviewers show their interests in papers by bidding for them. All papers have to fall into one of the four categories; high, average, low and no interest categories represented as H, V, L and N respectively. Also no paper can be in more than one group simultaneously. The algorithm then sorts the groups in ascending order and in each group sorts the papers based on the number of selections by the reviewers. It also sorts the reviewers and tries to assign papers to reviewers based on the level of priority given to a paper. The order of assignment of each paper is from reviewers who bid high on it to those who bid low. A reviewer with less number of high priority bids is more likely to get assigned those papers that the bids were high. This algorithm also performs COI detection.
B. Conflict of Interest (COI) Detection
A major task for the organizers of a conference is to identify members of the program committee that may have a conflict of interest in reviewing a specific paper. Conflict of interest detection is also used by some of the algorithms that perform automatic paper assignment. According to Papagelis et al (2005), scientists usually submit papers to a conference that they serve as reviewers. They also conclude that PC members are usually associated with authors of submitted papers, either because they are occupied in the same institute or project or because they have co-authored an article in the past. These have the potential to affect the reputation of the conference. There are several approaches to detecting conflict of interests. In a study by Pesenhofer et al (2005), the potential conflict of interest detection was performed based on the occurrence of the last name of a program committee member in the author's line of a submission and the existence of parts from the PC members email domain in the submissions author field. If the PC members email and the submission come from the same domain, then the system registers a potential conflict of interest.
Papagelis et al designed mechanisms that recommend potential conflicts by applying same institute appointment or previous co-authorship appointment techniques. The same institute appointment technique works by running string comparison on the email accounts of reviewers and that given by authors as they submit papers. A gradual string matching algorithm is applied that compares the different parts of the email accounts. This is better illustrated in the figure 1.
The previous co-authorship technique tries to identify pairs of PC members and authors of submitted papers that have co-authored one or more papers in the past. The data for this technique comes from a co-authors index as compiled by the DBLP Computer Science Bibliography. The DBLP server maintained at Universitt Trier provides information such as article title, author and co-author, if any, of major computer science journals and proceedings (Jin, 2003). The mechanism scans the set of paper authors and the set of reviewers' co-authors to identify matches that define potential conflicts. Matches are based on string comparison of their first and last names as shown in figure 2, taken from their paper.
Pesenhofer et al did not include this technique in their study but acknowledged that their system still missed some COI detection because of its non-inclusion (p 5). Aleman-Meza et al (2006) noted that EDAS - a CMS, checks for conflicts of interest based on declarations of possible conflicts by the PC members while bidding for papers. They designed an algorithm that finds all semantic associations between two entities (the reviewer and paper author). Each of the semantic associations found is analyzed by looking at the weightings of its individual relationships. Every semantic association is analyzed independently of the others, all of the possible different relationships are eventually considered by the algorithm. Some other systems allow authors to indicate COI with reviewers and use only this to indentify COI. Sometimes, PC members or authors may not indicate where COI exist, so having an algorithm for automatic its detection appears to be a more efficient method.
C. Session compilation
After the review process is over and the papers to be used have been chosen, the next step is to design the sessions for the conference. This is done by grouping the papers according to theme and each theme therefore becomes a session. The speakers in that session will include the authors of the selected papers. In the submission phase, the authors usually have to choose a certain number of research topics that are talked about in their papers. It may also be required, during the submission of the paper, to select keywords from a pre-defined list that highlight the topic focused on by the paper. Using the availability of this information, the technical committee can create thematic sessions. The sessions can either be in parallel especially when a huge amount of papers should be presented during the conference or in sequential order with no overlaps.
Some CMSs have added functionality that automates this process by using algorithms. Pesenhofer et al (2008) describe a process which makes use of clustering methods with a scheduling algorithm. They proposed a system to cluster the accepted papers with a hierarchical clustering algorithm to form a tree-like cluster known as a dendrogram. This approach creates a top down or bottom up ordering or the papers and this helps to create the sessions. According to them, papers down in the hierarchy are more equal to the linked papers than those up in the hierarchy.
Antonnia et al (2003) designed a system that performs information retrieval by content. They described other information retrieval by content methods including term-based and text clustering methods. Their prototype matching system assists the conference organizers to automatically establish semantic similarities among papers and allocate them into common themes. On the other hand, the system assists the attendees to retrieve the papers from the conference proceedings based on their content similarities. According to them, system aims to retrieve the documents that contain the same meaning from the entire document collection. The prototype-matching system analyzes a document collection structure and thus is domain adjusting. The system consists of three functions; document collection preprocessing and encoding, document processing and matching, and document retrieval.
D. User Role Definition
In order to prevent users from having unauthorised access to certain sections of a CMS, user roles are defined for the system. Processes like online registration, viewing conference program and call for paper have to be accessible for everybody. On the other hand, review rating and comments, submitted papers must only be accessible to authorized users, as determined by the organizers. In some systems, users can be categorized as registered, pre-registered and not-registered users (De Troyer and Casteleyn, 2001). They defined pre-registered users as a sub-class of the registered users because pre-registered users need some extra functionality such as confirming their registration. Some other systems categorize users differently. Niomanee and Limpiyakorn (2009) assigned 5 user roles to their system. These are administrator, program committee (PC) chair, PC members or reviewers, authors and general users. In the CMS specified by Ciancarini et al (1999), PC members and reviewers are different roles. Also, they have not defined administrator as a role and even introduce the editor of proceedings role.
In their analysis of roles and tasks during different phases, Halvorsen et al (1998) have identified four roles and describe the relationship between these roles in figure 3
According to them, organizers include general chairs, program chairs, publicity chairs, treasurer, and local arrangement group. A typical CMS should have the following user roles; system administrator, PC chair, reviewers or PC members, authors, participants and visitors. The roles can be defined as follows;
System administrator: manages conference data, does user administration, activates accounts (Levacic et al, 2005), configure initial context of system, edits and updates configuration, configures sending notification e-mails (Noimanee and Limpiyakorn 2009) and other system maintenance functions.
PC chair: opens registration and paper submission, assigns reviewers to papers, disables paper submission after deadline, decides on acceptance of paper, sends notification e-mails to other users (Popovici and Brad, 2006), checks review status and general co-ordination and monitoring.
Reviewers: sets up topics of interests, responsible for reviewing and rating papers, download papers that allocated to him/her to review, and evaluate and debate on papers (Huang et al, 2008) and makes recommendation for final decision.
Authors: uploads paper, checks review status and views the final result, registers for a conference and uploads the final version of accepted paper and slides (Huang et al, 2008)
Participants: authors and other persons who are interested in this field of research. The role includes all persons registered for the conference (Pesenhofer et al, 2005).
Visitors: non-registered users just browsing the site for general information.
E. Security and System Integrity
Conference management systems are designed to store users' personal information and in almost all cases, financial details. They also store research papers which are the result of years of painstaking and expensive research. For systems like these, security and integrity of the system should be a top priority, if not the top priority. Many systems studied implement a decent level of security, by restricting the user accessibility level and protecting the database from direct access.
In some systems, login protection is dependent on the server used by the system. If a user attempts to login to the system, the server searches the database for login information and uses the same to authenticate the user or otherwise decline the login attempt. Every login sets global variables and a role specific menu is generated based on the user role and these variables. In a situation where two or more users with different access levels have to perform the same action, the URL of the user with higher access rights is different. This allows the higher role to access the action of the lower role but not vice-versa. Security of the registration and booking processes by implementing Secure Sockets Layer (SSL) to encrypt the financial details and other personal information (Levacic et al 2005).
De Troyer and Casteleyn (2001) implement security in their system by defining what they termed as a security class hierarchy which classifies visitors according to the security needs of the system. The classifications with the security requirements in brackets are; registered users (logging in), pre-registered users (logging in and confirming their registration) and not-registered users (register). According to their design, pre-registered users are a subclass of registered users because they need the extra functionality of confirming their registration. Also, a pre-registered user becomes a registered user by confirming their registration, while a not-registered user becomes a registered user by registering. This implementation is vague as they do not give the relationship between the pre-registered user and the not-registered user. The natural flow would seem to be a transition from being a not-registered user who registers to become a pre-registered user who confirms the registration to become a registered user who can then access more sections of the system after their login is authenticated.
Halvorsen et al (1998), in their discussion of the CMS known as ConfMan, noted that the system uses the HTTP login and password authentication mechanism of the host server to prevent database access. Protection is provided by using access control lists to determine which users or processes have access to certain objects as well as the operations that can be performed on those objects. ConfMan security is implemented in the database. They stated that for all login requests, a query is sent to the database to check the user login from the HTTP authentication against the list of trusted people. The user can proceed if the authentication query finds a match, otherwise an error message is displayed, and access is denied.
Security is can be handled using basic user authentication. Upon registration, a username and password is assigned to each user. Users are designated as reviewers with limited access and some users are designated as administrators, with unlimited access and the right to create new users. There is an htgroupaccess file defines www-adminfor administrators and www-usersfor normal users (Mathews and Jacobs, 1996). The forms that require administrator-only access have an appropriatehtaccessfile specifying that only users from thewww-admingroup have access to that form. In the database, an entry for the user name and password must be maintained for anhtpasswdandthe htgroupfiles for server authentication.
Aleman-Meza, B., Nagarajan, M., Ramakrishnan, C., Ding, L., Kolari, P., Sheth, A.P., Arpinar, I.B., Joshi, A. & Finin, T. (2006). "Semantic Analytics on Social Networks: Experiences in Addressing the Problem of Conflict of Interest Detection", in Proceedings of the 15th International World Wide Web Conference (WWW2006), Edinburgh, Scotland, 2006, ACM, pp 412-414.
Antonina, K., Barbro, B., Hannu, V., Jarmo, T., & Ari, V. (2003). "Prototype-Matching System for Allocating Conference Papers", in Proceedings of the 36th Annual Hawaii International Conference on System Sciences (HICSS'03), Hawaii, USA, 2003, IEEE Computer Society.
Deerwester, S., Dumais, S.T., Landauer, T.K., Furnas, G.W., & Harshman, R.A. (1990). "Indexing by latent semantic analysis", Journal of the Society for Information Science. 41(6), pp 391-407.
De Troyer, O., & Casteleyn, S. (2001). "The Conference Review System with WSDM", in Proceedings of the 1st International Workshop on Object Oriented Software Technology, Valencia, Spain, 2005.
Dumais, S.T. & Nielson, J. (1992). "Automating the assignment of submitted manuscripts to reviewers", in Proceedings of 15th annual international ACM SIGIR conference on Research and development in information retrieval, Copenhagen, Denmark, 1992, ACM, pp 233 - 244.
Franklin, M.J., Widom, J., Weikum, G., Bernstein, P.A., Halevy, A., DeWitt, D.J., Ailamaki, A. & Ives, Z.G. (2004). "Rethinking the Conference Reviewing Process," in Proceedings of the ACM SIGMOD International Conference on Management of Data(SIGMOD 2004), Paris, France, June 13 -18, ACM , pp 957.
Halvorsen, P., Lund, K., Goebel, V., Plagemann, T., Preuss, T. & Koenig, H. (1998). "Architecture, Implementation, and Evaluation of ConfMan: Integrated WWW and DBS Support for Conference Organization", Technical Report I- 1998.016-R, UniK.
Huang, M., Feng, Y., Desai, B.C. (2008). "ConfSys: a web-based academic conference management system", in Proceedings of the 2008 C3S2E conference, Quebec, Canada, May 12 -13, ACM, pp 141 - 143.
Jin, X. (2003). Confsys: the Cindi Conference Support System (pp 29-37). Unpublished MSc., Concordia University, Quebec, Canada.
Kalmukov, Y. (2007). "An analysis and experimental study of an algorithm for automatic assignment of reviewers to papers", in Proceedings of the 2007 international conference on Computer systems and technologies, Bulgaria, June 14 - 15, ACM, pp v5.4 - v5.5.
Levacic, V., Bogunovic, H., Loncaric, S., Kuzmic, B. & Blaona, B. (2005). "Chairman - A Conference Management System", in Proceedings of the 28th international conference on information and communication technology, electronics and microelectronics (MIPRO 2005), Opatija, Croatia, 2005.
Mathews, G.J., & Jacobs, B.E. (1996). "Electronic management of the peer review process", Computer Networks and ISDN Systems. 28(7-11), pp 1523-1538.
Noimanee, P. & Limpiyakorn, Y. (2009). "Towards a RESTful Process of Conference Management System", in Proceedings of the International MultiConference of Engineers and Computer Scientists 2009 (IMECS 2009), Hong Kong, March 18-20.
Oates, B. J. (2006). Researching Information Systems and Computing (pp 71 - 92). SAGE.
Papagelis, M., Plexousakis, D., & Nikolaou, P.N. (2005): "CONFIOUS: Managing the Electronic Submission and Reviewing Process of Scientific Conferences", in Proceedings of the 6th International Conference on Web Information Systems Engineering (WISE 2005, Industrial Track), New York, USA, 2005.
Pesenhofer, A., Mayer, R. & Rauber, A. (2005). "Improving Scientific Conferences by Enhancing Conference Management Systems with Information Mining Capabilities", in Proceedings of the 1st International Conference on Digital Information Management (ICDIM 2006), Bangalore, India, 2006.
Pesenhofer, A., Mayer, R. & Rauber, A. (2008). "Automating the Management of Scientific Conferences using Information Mining Techniques", Journal of Digital Information Management. 6(1), pp 3 - 11.
Popovici, Z.O. & Brad, R.O. Web based conference management system. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.64.7621&rep=rep1&type=pdf.
Webster, J & Watson, R.T. (2002). "Analyzing the past to prepare for the future: writing a literature review", MIS Quarterly Journal. 26(2), pp xvi-xviii.