Accounting studies that apply experimental research approaches and laboratory experiments offer an alternative and nontraditional perspective on a variety of accounting topics. Experimental accounting research can encompass a broad range of topics or sub-disciplines such as judgment and decision making in auditing, risk-taking behavior and analysis, managerial accounting and control systems, accounting information processing and systems, and international financial accounting, reporting and disclosure decisions. The focus of this paper is on judgment and decision making in auditing and the use of statistical and experiment designs in that research stream.
The basic aim of judgment and decision making research is to improve auditor judgments. In order to make improvements, it is necessary to understand how judgments are made and what the key factors are affecting auditor performance under different conditions. According to Trotman (2001), early research on auditor judgment had the purpose of evaluating auditor judgment quality and understanding how judgments are made while later audit judgment research has examined the determinants of judgment performance. The earlier research concentrated on ability and knowledge compared to the concentration of later research on performance as a function of ability, knowledge, environment and motivation. Most of the current audit judgment and decision making research has examined the interaction between the various factors affecting performance and understanding the impact of pre-decision factors on performance. Throughout the development of this research stream, the most common method of examining auditor judgments has been the use of experiments since, unlike many other areas of accounting research, databases to examine auditor judgments and the factors affecting them are not generally available. In addition, alternatives to experimentation, such as individual observation or survey methods, are also employed in audit judgment and decision making research; however, neither afford the researcher with the strength to infer a causal relationship between variables being studied.
This paper is organized as follows. The next section provides a brief overview of statistical and experimental design, followed by a discussion of the use of experiments in audit judgment research including an example of an experimental study. Finally, a critical analysis of statistical and experimental design in audit judgment and decision making is presented with future considerations for researchers employing experimental methods in this area.
OVERVIEW OF STATISTICAL AND EXPERIMENTAL DESIGN (Vinzi, 2010)
If an experiment can be defined as a research method in which a researcher manipulates or measures and controls one (or more) independent variables and observes the associated variation of the dependent variable(s), then the experimental design refers to how the independent variables are manipulated and controlled. The experimental design reflects the research strategy employed in order to obtain answers to the research question under consideration. The design should be developed to allow a researcher to answer his/her research question as reliably, objectively, and accurately as possible subject to any economic constraints. Once the research question has been established, the dependent variable(s) to be observed and the specific independent variable(s) to be manipulated or measured must be identified. The dependent variable chosen must be sensitive to changes in the independent variables in order to test any effect. The independent variables identified are generally manipulated or measured over different treatment groups which represents the presence (or absence) of the independent variable or different levels or amounts of the independent variable. An independent variable is measured, as opposed to manipulated, when it is not possible or perhaps impractical to be manipulated (for example, human characteristics such as gender). Choices have to be made about the number of treatments of the independent variable to be included in the design as this, as well as the number of independent variables included in the design, will affect the number of experimental units (observations or subjects) required for the study. Once the number of observations has been determined, a sampling scheme needs to be developed such that the results yielded from the experimental units, drawn statistically from the overall population of units, can then be generalized to the entire population. Nonprobability (reasoned) sampling or probability sampling can be utilized. Reasoned sampling involves choosing observations based on certain features so as to resemble the population they have been drawn from as a whole while probability sampling involves observations chosen at random and not by chance. Finally, extraneous (i.e. confounding) factors must be controlled so that variation in the dependent variable can be clearly attributed to the manipulation of the independent variable(s) and not influenced by other confounding variables. The main method for controlling for extraneous variables is through the randomization of experimental units between treatment groups which aims to ensure that confounding variables do not bias the results of the experiment and to ensure that different treatment groups can be considered equally.
In conjunction with experimental designs, the Analysis of Variance (ANOVA) is used to determine whether samples from two or more treatment groups come from populations with equal means. ANOVA tests are valid under the assumption that the dependent variable is normally distributed, the groups are independent in their response on the dependent variable and variances are equal for all treatment groups. The ANOVA is based on two independent estimates of the population variance obtained from sample data. A ratio is formed for the two estimates where one of the estimates is sensitive to treatment effect and error between groups and the other to error within groups. In ANOVA, the null hypothesis tested is the equality of dependent variable means and therefore the alternative hypothesis is that the treatment group means are different. The researcher wants to know if the difference in sample means is significant enough to conclude the real means do in fact differ between the groups. The answer depends on (1) the variability of group means, (2) the sample sizes in each group since larger sample sizes give more reliable information and (3) the variances of the dependent variable in each treatment group. For the same absolute difference in means, the difference is more significant if in each treatment group variance is small. Likewise, if the treatment group variances are widely dispersed, then the given difference of means is less significant. The key statistic in ANOVA is the F-test of difference of group means, testing if the means of the treatment groups formed by values of the independent variable(s) are different enough not to have occurred solely by chance. If the group means do not differ significantly then it is inferred that the independent variable(s) did not have an effect on the dependent variable. If the F test shows that overall the independent variable(s) are related to the dependent variable, then multiple comparison tests of significance are used to explore which values of the independent(s) have the most to do with the relationship. Again, the formulas for the F-test used in ANOVA reflect three things: the difference in means, group sample sizes, and the group variances. That is, the ANOVA F-test is a function of the variance of the set of group means, the overall mean of all observations, and the variances of the observations in each group weighted for group sample size. The next section will provide an overview of how experimental designs and ANOVA techniques are employed in the audit judgment and decision making research.
STATISTICAL AND EXPERIMENTAL DESIGN IN AUDIT RESEARCH
According to Trotman (1998), the use of experiments to examine the effect of particular factors on auditor judgments has major benefits. Audit judgment studies using controlled experimental settings allow researchers to remove many of the confounding factors that make audit judgment very complex. By controlling potentially influential variables, the researcher can infer causal effects. Experimental audit judgment studies utilizing the ANOVA technique were first introduced to the audit judgment literature in the early 1970s (Trotman, 1998). As noted previously, ANOVA measures the significance and amount of variance accounted for by the primary effects and interactions. A significant effect across treatments implies that an auditor's judgment varies systematically with changes in that treatment. A significant interaction between two treatments implies that an auditor is responding to particular patterns in the treatments. The significance of particular treatments can be determined which indicates the auditors' use of and patterns in the treatments. The major advantage of ANOVA is it allows clear interpretation of the significance of the treatments and interaction effects, therefore the design is useful in audit judgment studies where the objective is to assess the factors effecting an auditor's decision making while eliminating the effects of extraneous variables.
As much audit judgment and decision making research has this as its objective, experiments in various streams of audit judgment and decision making research are common, especially in the areas of policy capturing, group decision making, decision aids, knowledge and memory, and environmental or motivational issues. Each of these streams is briefly discussed in terms of their experimental approach. The main objective of policy capturing research is to develop representations of auditors' judgment policies. This is generally done using experimental settings where auditors are provided with a series of hypothetical cases, each case consisting of a number of treatments, and asked to make judgments in order to reveal a judgment strategy (Trotman, 1998). In terms of group decision making, the term audit groups refers to joint multi-person judgment and decision making. The experimental method employed is to have subjects perform judgments as individuals and then have some individuals formed into audit groups who reperform the judgments which are then compared to determine improvements in performance. The stream of research on decision aids addresses the question of whether or not a particular decision aid improves judgment performance or not. The experimental design for such studies is generally between subjects where subjects are randomly allocated to a treatment group with the decision aid or a control group without the decision aid. In experimental studies examining the relationship between ability, experience, knowledge and performance, one of the most common streams of JDM research in auditing, the researcher is concerned with how these knowledge differences and interactions with the other variables lead to performance differences (Trotman, 1998). Finally, experimental studies in environmental or motivational issues manipulate accountability, feedback or incentives in order to determine their effect on audit judgment performance. In the following section, an experimental study within this stream of research is reviewed, the experimental approach presented and the limitations of that approach considered.
STATISTICAL AND EXPERIMENTAL DESIGN IN AUDIT RESEARCH: AN EMPIRICAL EXAMPLE
The article, “The impact of commitment and moral reasoning on auditors' responses to social influence pressure”, was published in the journal Accounting, Organizations and Society, a highly reputable journal in the field of accounting research and one of the few which regularly accepts publications from authors employing alternative and nontraditional approaches such as the experimental approach is deemed in accounting research. The study empirically examines the effects of social influence pressures within accounting firms on auditor decisions (Lord and DeZoort, 2001). Specifically, the effects of obedience pressures from superiors and conformity pressures from peers on auditors' decisions sign-off on financial statements that are materially misstated are evaluated. Research in this area is motivated by a body of evidence suggesting that auditor judgment and decision making processes are impacted by pressures generated from within the organization on auditor attitudes, intentions and behavior (Lord and DeZoort, 2001).
In this study, the relationship between the dependent variable “auditor decision” and the independent variable social influence pressure is the main effect being studied. To do so, Lord and DeZoort (2001) identified a sample of 171 auditors from one international firm who participated in a between-subjects experiment conducted with obedience pressure from superiors and conformity pressure from peers as the social influence pressure treatments. Four separate experiments were conducted at training sessions in different cities in different geographic regions within the USA. The participants were presented a case scenario culminating with an audit judgment decision to sign-off on an account balance that was materially misstated and a brief explanation of that decision. The participants were assigned randomly to one of three pressure treatment groups which means that assignment occurred in such a way that every participant had an equal chance of being assigned to each treatment group. Participants in the control group received no information beyond the case itself. Participants in the conformity pressure group were informed as to the decision taken by one of their peers in the same case while participants in the obedience pressure group were informed as to the opinion of their supervisor on the decision to be taken in the case. Although, for purposes of this paper, the results of the study are not specifically under review, they indicate that obedience pressure significantly increased auditors' decision (willingness) to sign-off on an account balance that was materially misstated, although conformity pressure did not.
Several limitations are noted in the design of this experiment which may have affected the results. The first type of limitation to consider relates to the subjects in the experiment. In this experiment, the number of observations (subjects) within each treatment group was not balanced (i.e. of equal sample size) which implies that the power of the test is neither maximized nor as robust to violations of the hypotheses as if the experimental design had been balanced. Further, the subjects represented auditors within a certain age and range of experience which means the selection of the subjects participating in the survey was not random despite their assignment to treatment groups was apparently randomized. In addition, age and range of experience are factors which might be more affected by social influence pressures due to certain socialization issues that may change with maturity and tenure with a firm, which means there might be other extraneous variables affecting the result. The second type of limitation relates to the instrument used to conduct the experiment, in this instance a case scenario. The case scenario employed to conduct the experiment is a hypothetical scenario and therefore the results may not accurately reflect conformity and obedience pressures that may be much stronger in practice than in the experiment. One advantage of an experiment is that it allows the researcher to study phenomenon that might not be able to be studied in a natural setting (Peecher and Solomon, 2001); yet at the same time the artificiality of the setting (a training session) could also influence the outcome despite the researchers attempt to create as realistic setting as possible. These limitations reaffirm the importance of the experimental design phase in ensuring the success of an experimental study. Based on the overview of experiments and experimental design and their use in the audit judgment and decision making literature, the next section contains a critical analysis and considerations for the use of experimental methods in the audit literature.
CRITICAL ANALYSIS OF STATISTICAL AND EXPERIMENTAL DESIGN IN AUDIT RESEARCH
Three areas of critical analysis and considerations for the design and use of experimental methods in the audit literature are discussed within this section: issues of underlying theory, issues in the experimental setting, and issues with participants in the experiment. Most audit judgment research utilizing experimental methods falls into the explanatory research category where explicit theory explaining previously identified empirical generalizations is further developed, tested and reformulated (Peecher and Solomon, 2001). The design phase in this category of research is key. If adequate thought is not given to the pre-explanatory and design phase, numerous problems can result, the most severe of which are manipulating the wrong variables, setting manipulated variables at the wrong level, and failing to control extraneous variables. One of the main issues driving these problems is that the theory being tested, expressed as the hypotheses under study, is not sufficiently developed. This means that researcher may not have enough information to properly identify the dependent and independent variables to be included or excluded from the study and the levels at which the independent variables should be manipulated. The researcher also uses theory to identify participants (subjects) for the study, for example, to determine the number of participants necessary to be able to detect the hypothesized relationships.
In an experiment, the researcher effectively creates and places the participants in a setting that by design is a simplification of the real world in which the auditors operate and presents them with a simplified task (Peecher and Soloman, 2001). The issue inherent in this design relates to attempts to increase the generalizability of any cause-effect relationship by focusing on creating the most real-world like setting possible but perhaps at the expense of the relationships being studied. Further, experimenter bias issues need to be considered in designing the experimental setting as these can have unintentional biasing effects on the results. For example, if the experiment involves the coding of data when there are qualitative responses, there should be two coders whose work should be examined for consistency in coding in order to ensure the reliability and accuracy of the data. A final choice impacting the experimental setting is whether to conduct a controlled experiment or non controlled experiment. A controlled experiment is conducted in the presence of the researcher while the researcher is not present in a non-controlled experiment such as computerized research experiments. Given the difficulty of obtaining subjects, in this case audit staff, it is inevitable that non-controlled experiments are carried out which requires awareness of the threat of potential validity issues.
Finally, in terms of issues with participants in the experiment, many researchers assume that practicing auditors make the best participants for audit judgment and decision making experiments, and the more experienced the participant, the better (Peecher and Solomon, 2001). This can be linked to the previous point discussing the researchers attempt to make the study as real-world like as possible; however, the use of practitioners as subjects can result in several issues. First, because the participants who are the subject of audit judgment experiments are given the option of withdrawing from the experiment at any time, it may be difficult to follow a formal random sampling plan, and especially when dealing with variables such as specialized knowledge or task experience which are manipulated in audit studies. Second, and more commonly where the subjects in experiments are students, the decision has to be made on whether any monetary incentives need to be paid to subjects. While this practice is common in some areas of experimental accounting research, it is less prevalent in the audit judgment and decision making literature. Where some form of monetary incentive is provided, the challenge is to ensure that the incentive does not result in subjects using non-typical behaviors which invalidate the relationships and hypotheses being studied. One way to address this issue is to carry out a manipulation check by obtaining some kind of measure of the independent variable that shows that the subjects have consistently understood the experimental intent. Another way is to consider including a control group in addition to treatment groups where subjects in the control group are treated exactly the same as in the experimental group in all ways except they do not receive any treatment. The advantage to using a control group is that the researcher can conclude whether or not the treatment produced different results than the control group.
This paper provides an overview of statistical and experimental design as applied in the area of audit judgment and decision making research and addresses a range of statistical and experimental design questions relative to that stream of research. These design considerations and questions to be addressed are critical in developing a sound auditing judgment and decision making study using experimental methods.
Lord, A. T. and F. T. DeZoort (2001). "The impact of commitment and moral reasoning on auditors' responses to social influence pressure." Accounting, Organizations and Society 26: 215-235.
Peecher, M. E. and I. Soloman (2001). "Theory and Experimentation in Studies of Audit Judgments and Decisions: Avoiding Common Research Traps." International Journal of Auditing 5: 193-203.
Trotman, K. T. (1998). "Audit judgment research - Issues addressed, research methods and future directions." Accounting and Finance 38: 115-156.
Trotman, K. T. (2001). "Design Issues in Audit Judgment Experiments." International Journal of Auditing 5: 181-192.
Esposito Vinzi, V. (2009/10). “Statistical Design for Research: Sampling, Experimental Design & Analysis of Variance”. ESSEC Ph.D. Program: QRM1.