Building on previous researches on guided summarization practice with automated feedback, this research proposal describes the design and development of a feedback-based tutorial system for computer-aided language learning, one that assesses and improves the comprehension and expressive abilities of EFL students. The system simulates interaction with students via a specially-optimized speech recognition engine. It uses Latent Semantic Analysis (LSA) to measure the semantic similarity between a system-generated recording and the oral retelling of students, who subsequently receive detailed system feedback on their performance in comprehension, memorization and oral presentation. With the system, students repeatedly revise their speech in accordance with the detailed feedback and try to reformulate a semantically accurate/complete version, a process expected to facilitate their linguistic skills.
The unprecedented rise in the number of EFL (English as a Foreign Language) students coupled with pressing needs for international communication is often contrasted by the inadequate oral language teaching and training (Hu, 2004). The lack of qualified language teachers and proper methodology (Nunan, 2003) to meet the stringent requirements of language teaching has led to a compelling need for a means whereby students engage in independent studies with the help of a tutorial system. This research proposal explores the efficacy of an intelligent tutorial system that provides proper feedback and guidance based on oral retelling.
- Retelling has been found to provide substantial benefits to language learning, improving comprehension and recall of discourse (Lipson & Wixson, 1997). When students reformulate text, their language complexity is developed by internalizing text features (Brown & Cambourne, 1987), thus improving understanding and memorizing the ideas in texts (Anderson, 1994).
- Previous feedback systems (Steinhart, 2001) on English text summarization suggest that students that receive feedback in the form of semantics-based scores in summarization practices can significantly outperform students without the assistance of feedback. The gains in performance brought by feedback are proportional to the difficulty of the text.
- Research on cognitive studies has revealed the effectiveness of self-explanation as a metacognitive strategy in helping students improve understanding in learning (Aleven & Koedinger, 2002;). A number of intelligent tutoring systems that draws on self-explanation have been developed successfully, using simple techniques such as menus or templates (Aleven & Koedinger, 2002; Trafton & Trickett, 2001). This proposal endeavors to prove that the application of self-explanation may be even made more effective when implemented in a tutorial interactive system, where students are encouraged to elicit explanations or comprehension in their own words.
- Chinese EFL students are found to display distinctive pragmatic, lexical, syntactic and phonological characteristics in both written and spoken English (Xiao & Zuo, 2006), often attributable to influence from the mother tongue. Such characteristics make the development of EFL-oriented tutorial systems a different and more challenging task than that of those aimed at native speakers.
The theory of "linguistic spillover (Brown and Cambourne 1987)" describes the natural trend for language learners to internalize the linguistic patterns they are exposed to when later asked to compose their own stories (Smith, 1988). Linguistic patterns thus acquired can help English learners transcend limitations in their expressive abilities, e.g. vocabulary and syntax (Peregoy & Boyle, 1997).
At baseline, this research project will examine the effectiveness of a feedback-based tutorial system in assessing and improving EFL students' linguistic abilities such as comprehension, memory and oral skills, and explore the difficulties faced in developing systems for human-computer interaction.
I plan to develop the system to be used in various scenarios such as computer-aided language learning, where repeated oral retelling of listening materials is an essential task (Zhong, 2003). It can additionally serve as a valuable tool for assessing the oral linguistic ability of students without involving the cost- and time-consuming process of human grading. The results of this research may provide useful insights into machine understanding of human speech in dialogue systems.
The general design envisioned for an initial prototype of the proposed system is summarized below:
- Database Development. A database of semantically-related texts is established to compute latent semantic relationships.
- Speech Synthesis and Delivery. A speech synthesized from one of the texts is delivered to the student user who is required to retell the speech from memory.
- Speech Recognition. The student retelling is recognized by an ASR engine optimized for EFL and converted to text.
- Semantic Clustering and Comparison. The recognized text is then clustered into several semantic subsections using semantic clustering, each of which is compared with its semantic counterpart in the original computer-generated speech.
- Score and Feedback Generation. Scores for each semantic cluster and an overall score are presented to the student as feedback, along with comments indicating the general performance and specific areas that needs to be revised, such as semantic completeness and missing/unrelated sentences.
- Reformation of Retelling. The student, upon receiving the feedback, tries to improve retelling by better emulating the original speech. The system compares later attempts with previous ones, indicating the progress the student made.
Depending on the purpose, the scores can either be used to assess the student' linguistic capacity, or become part of the feedback rendered back to the speaker.
As can be seen from the above illustration, the proposed system integrates theories and findings from several interdisciplinary fields, e.g. Linguistics, Psychology and Computer Science. Thus a good coordination of the various components is essential.
- Semantic Comparison. To measure the textual similarity in terms of semantic closeness between the original text and students' paraphrased version, semantic algorithms such as Latent Semantic Analysis (LSA) can be introduced.
- Semantic Clustering. When comparing textual semantic similarity, the system uses a method called semantic clustering to partition the text into a set of semantic themes. Semantic techniques such as k-means clustering can be applied on the LSA vectors of the text. We then select a phrase or expression most representative of each cluster, which may assist the student in recalling and reproducing the original speech when necessary.
- EFL ASR and Parsing. The system builds on existing models of Automatic Speech Recognition (ASR), e.g. the open source CMU Sphinx. Semantics-based, the system is more lenient with recognition errors and occasional misrecognized words will not severely affect the final assessment.
- Evaluation. For the empirical research with EFL learners, around four to five classes of high school students will be chosen to assess the effectiveness of the system and to gain perspectives on its usability.
Latent Semantic Analysis (Landauer et al., 1998) is an unsupervised technique of deriving semantic relationships between documents and terms by representing them in semantic vector space deduced from a large corpus of related texts. The semantic similarity between two terms is measured by the cosine of the angle between their respective term vectors, calculated using the formula below:
Consequently, both terms and documents are represented as vectors and the semantic similarity can be calculated through comparisons between terms and documents.
However, prosodic and syntactic characteristics common in EFL students such as non-native pronunciation and syntactic errors have complicated the task of EFL speech and text processing. To tackle such problems, the ASR engine will be trained by EFL speakers on three areas: (1) Frequently-used vocabulary, as a means to improve vocabulary prediction. (2) Common syntactic structures, to be used in later syntactic parsing. (3) Typical regional accents, such as that of Cantonese speakers.
In addition, changes will be made in the system to better recognize pre-computed terms semantically similar to the original speech. For instance, the system will be geared to expressions like t-shirts, fashion, put on etc, when the topic is on clothes.
Three sets of evaluations will be conducted. (1) First, a test will be designed to examine the retelling accuracy and completeness between two groups of students, one practicing oral retelling with the help of the tutorial system, and the other without. (2) The second evaluation focuses on the feedback effect on revision. The semantic similarity scores awarded to no-feedback students and those with feedback are compared by computing their respective LSA vector cosines with the original recording. (3) Finally, to test the validity of the system as an independent assessor of students' capacity, the scores produced by the system and human raters will be compared and the human-computer correlation will be calculated.
Expected Findings and Significance
The proposed tutorial system differs from existing intelligent tutorial systems and, as such, makes a number of contributions to previous research.
- The tutorial system is the first to aim at improving the comprehension and oral skills of EFL students. EFL Students in China traditionally have overly concentrated on reading or writing, whereas their exposition to listening and speaking is appallingly inadequate (Yu, 2001; Nunan, 2003). The shortage of qualified teaching force means that their daily oral practice often falls short of proper guidance and feedback. The proposed tutorial system provides a feasible solution to such problems.
- With the introduction of Automatic Speech Recognition (ASR) technology optimized for EFL students, comprehension, memory and consequently interpreting skills of student users can be fostered far more effectively than text-based systems.
- The proposed innovations in tackling the textual and prosodic aspects in EFL speech processing have made the system an ideal tool for computer-aided language teaching/learning.
- The system incorporates a number of design and technical innovations into the system architecture and attacks the following problems from a technical point of view:
- Finer-grained algorithm for semantic similarity comparison
- Better metrics for evaluating the effectiveness of the system
- Exploration of effects of verbal and textual instructions
- Use of pictures for illustration purposes during the retelling process
- Investigation into the effects of misrecognition by ASR engines and ways to address them (e.g. using semantic context and refined prediction of vocabulary).
- The inherent nature of the proposed system has made its application more diverse than previous attempts, with perspective applications such as developing writing and oral interpreting skills.
- Aleven V., Popescu O., Ogan A. (2003), A formative classroom evaluation of a tutorial dialogue system that supports self-explanation, in: V. Aleven (Ed.), Proceedings of the 11th International Conference on Artificial Intelligence in Education, 2003, pp. 345-355.
- Anderson, R. (1994). Role of the reader's schema in comprehension, learning, and memory. In R. B. Ruddell, M. R. Ruddell, & H. Singer (Eds.), Theoretical models and processes of reading (pp. 469-482). Newark, DE: International Reading Association.
- Deerwester, S., et al, Improving Information Retrieval with Latent Semantic Indexing, Proceedings of the 51st Annual Meeting of the American Society for Information Science 25, 1988, pp. 36-40.
- Gambrell, L., Koskinen, P. S., & Kapinus, B. A. (1991). Retelling and the reading comprehension of proficient and less-proficient readers. Journal of Educational Research, 84, 356-362.
- Hu, Guangwei. (2004). English Education in China: Policies, Progress and Problems. Language Policy (2005) 4: 5-24
- Landauer, Thomas., Foltz P. W., & Laham D. (1998). Introduction to Latent Semantic Analysis. Discourse Processes 25: 259-284.
- Lipson, M., & Wixson, K. (1997). Assessment and instruction of reading and writing disability. An interactive approach. New York: Addison-WEFLey.
- Nunan, David (2003). The impact of English as a global language on educational policies and practices in the Asia-Pacific region. TESOL Quarterly, 37, 589-613.
- Steinhart, D.J. (2001), "Summary Street: An intelligent tutoring system for improving student writing through the use of latent semantic analysis", http://lsa.colorado.edu/papers/daveDissertation.pdf.
- Smith, F. (1988). Joining the literacy club. Portsmouth, NH: Heinemann. Peregoy, S.F. & Boyle, O.F. (1997). Reading, writing, & learning in EFL. A resource book for K-12 teachers. New York: Longman Publishers.
- Shuttleworth, M & Cowie.M, Dictionary of Translation Studies, Manchester: St Jerome
- Xiao, Jing and Zuo, Niannian. (2006). Chinglish in the oral work of non-English majors. CELEA Journal Vol. 29, No. 4
- Yu, Liming (2001). Communicative language teaching in China: Progress and resistance. TESOL Quarterly, 35, 194-198.
- Zhong, W, 2003, Memory Training in Interpreting, China Translators' Journal, Vol.7, 39-43