Justin Jernigan

Abstract

            The ability of English language learners (ELLs) to participate fully in the cultural life of an English-speaking community is largely dependent on their ability to use spoken language in socially acceptable ways. Assessment of the oral proficiency of ELLs needs to take into account the implicit social and pragmatic rules of English. Unfortunately, oral proficiency assessment is subject to limitations that partly stem from the lack of recognition of social and pragmatic considerations, which may be thought of as implicit rules governing the social and interactional use of language. 

            In this paper, I explore issues relating to the validity, authenticity, and reliability of oral proficiency assessments, the difficulty of assessment tasks, and the awareness of pragmatic and social contexts. I also offer several recommendations that teachers and others can follow when testing the oral proficiency of their students. The recommendations have value for improving the training of interviewers and raters who are responsible for evaluating speaking performance as well. The issues and recommendations all relate in some way to the need for a description of oral proficiency that includes, among other components, the implicit rules mentioned above. The central concern of the paper is the way in which incorporating techniques that target interactional and pragmatic aspects of language can provide for more accurate assessment of the oral proficiency of ELLs.

Introduction

            In order to participate more fully in the cultural life of an English-speaking community, English language learners (ELLs) need more than simply developing structurally accurate spoken language. They must demonstrate more than just understandable pronunciation and an awareness of English syntax. Oral proficiency assessment that focuses only on pronunciation and grammatical accuracy in speaking fails to take into account a critical aspect of language use—interactional and pragmatic competence, which may be defined as the ability of learners to use language in communicative contexts to convey their intended meanings or influence those around them. The common tendency among classroom ESL teachers to focus primarily on a learner’s phonology and syntax when evaluating oral proficiency, while understandable, must be guarded against, in that it does not take into account the full range of requirements of successful oral communication. The solution I propose in this paper is the addition of an orientation toward interactional-pragmatic competence in classroom instruction and assessment of ELLs. 

English speakers expect their interlocutors to operate according to the implicit interactional and pragmatic norms of that language during conversation and other forms of verbal communication. The interactional and pragmatic competence that learners need in order to be able to break implicit code native speakers understand is an acquired competence learners develop through exposure to and use of the language (Matsumura, 2003; Swain, 2005). ELLs are not generally expected to be able to talk about the norms or to discuss them but their ability to implement pragmatic and broader sociolinguistic aspects of English is integral to their success in speaking the language. Predictably, failure to account for implicit code in oral proficiency assessment may result in an incomplete view of language, thus presenting problems for the authenticity, validity, and reliability of such tests, as well as for the evaluation of the difficulty of assessment tasks. 

As a result, I argue that assessing the oral proficiency of ELLs in ways that take into account implicit code is an important goal for teachers, testing directors, and others concerned with the accurate assessment of ELL oral proficiency. ESL teachers typically spend a great deal of time developing the grammatical accuracy of their students, which may result in their being satisfied with syntactically based assessment of the ELLs in their classrooms. However, the lack of awareness of sociolinguistic and pragmatic aspects of language among second language teachers—at least at a level that allows for the implementation of such aspects in assessment or instruction—has been indicated in second language pragmatics research over the past several years (e.g., Bardovi-Harlig, Hartford, Mahan-Taylor, Morgan, & Reynolds, 1991; Davies, 2004) and is implied in calls for a greater emphasis on interactional and pragmatic elements in instruction and assessment by such second language researchers as Johnson (2001), Kasper and Roever (2005), and Swain (2005).

 An additional challenge is that instruments designed to target oral proficiency, such as the American Council on the Teaching of Foreign Languages (ACTFL) Oral Proficiency Interview (OPI), do not necessarily measure the constructs they are supposed to assess. In fact, speech occurring during such interviews has been found to be lacking in normal conversational features, such as turn-taking and topic management (Johnson, 2001; Salaberry, 2000). Such shortcomings are particularly problematic because tests and tools such as the OPI are used in the accreditation of foreign language teacher education programs. The National Council for Accreditation of Teacher Education (NCATE) and ACTFL (2002) national standards make clear that the assessment of ELLs’ interpersonal communication skills are to be assessed through oral interviews or similar tasks (p. 31).[1] Given the place of the OPI in teacher training with respect to assessing interpersonal communication and the limitations of OPI-type assessments, it is not surprising that we as language educators often neglect interactional (conversational) aspects of oral proficiency when evaluating our learners.

We need to distinguish at least two kinds of speaking performance: mechanical performance, by which I refer to the implementation of grammatical rules and accurate pronunciation; and interactional-pragmatic performance, which I use here to refer to how a learner puts the implicit social and pragmatic norms of the language into practice when speaking. These two kinds of performance occur simultaneously and are intertwined, but may be distinguished—provided awareness of both kinds of performance is sufficiently high. Despite their considerable importance, the social and pragmatic aspects are often underrated or ignored when assessing oral proficiency. 

When a teacher or assessor listens to a learner’s speech production and makes judgments about her overall oral proficiency, she may focus on the syntactic accuracy and pronunciation of the learner’s speech (i.e., the mechanical aspects of the learner’s speech) and may acknowledge the importance of sociolinguistic elements, such as politeness, but lack a framework within which to place those less mechanical elements (as suggested in Chalhoub-Deville & Deville, 2005). The teacher or tester, as a native or near-native speaker of English, is subconsciously aware of the interactional and pragmatic appropriateness of the ELL’s speech within and outside of her interview responses and of the degree of acceptability of the learner’s presentation (how the learner comes across in terms of interpersonal communication). However, this subconscious awareness is not easily incorporated into a holistic score reflecting oral proficiency.

We might consider the case of an ELL who gives oral interview responses that consistently engage the teacher or rater in interactive exchanges. One example may be the ELL’s use of pragmatic constructions such as modal verbs (e.g., would you like to…) in order to express politeness.  Likely the learner’s overall oral proficiency will be assessed as being higher than that of a learner who provides relatively straightforward responses lacking in expected pragmatic features. If the teacher or tester enters the interview with a focus exclusively on syntax and pronunciation, the evaluation may be made without the teacher’s even being aware that the learner’s social-pragmatic performance has been a major factor influencing the final result. 

As a result, we should expect that the task of determining oral proficiency more effectively should include filters referring specifically to relevant pragmatic constructs corresponding to language proficiency levels. Such definition is a primary goal of this essay, and achieving this goal is in many ways prerequisite for the other goals, which relate to testing and training. Only with a clear definition of the underlying construct can instruments be developed and training of teachers, interviewers, and raters be implemented such that assessment results more accurately reflect proficiency and competence in the actual speaking of English. After all, if the purpose of speaking a language is to communicate in the world outside of the classroom or testing situation and to participate in the language community, then authenticity, validity, and reliability in the assessment of oral proficiency must be a central goal.[2]    

I suggest that in practice, the lack of awareness of the importance of interactional and pragmatic competence may lead to ELLs being assessed inaccurately in at least two ways. First, learners who make effective use of nonverbal cues, status preserving strategies, interpersonal communication skills, and other aspects of sociolinguistic and pragmatic competence may be evaluated as having high language proficiency with respect to speaking, perhaps masking possible deficiencies in syntax, morphology, or vocabulary. On the other hand, ELLs who fail to implement interactional and pragmatic aspects of English may be assessed as having poor speaking proficiency despite a high level of grammatical competence and overall oral fluency. In both cases, the problem entails a failure to take interactional and pragmatic competence into account.

The potential for inaccurate evaluation may be most evident in oral interviews, such as the OPI, among the most common types of oral proficiency assessment instrument used. On such instruments, the interactional and pragmatic competence of learners is a significant element influencing how they are evaluated. This component of oral proficiency may take priority over grammatical competence, pronunciation, fluency, and other mechanical aspects in terms of the results of many types of assessments, yet teachers and testers may fail to recognize its importance. Davies (2004) urged teachers to conceive of language use as discourse rather than as sets of isolated speech acts (p. 210). In other words, we as language educators need to look at how language is used to develop relationships among people instead of viewing language as the sum of the grammatical structures and vocabulary used by our learners.

In this paper, I do not offer a general review and analysis of the numerous instruments utilized for the assessment of oral proficiency.[3] Rather, I explore some key issues and limitations related to oral proficiency testing of ELLs today and offer recommendations for testing and training. In the following pages, I consider research pointing to possibilities for the development of assessment instruments and for the training of classroom teachers as well as test interviewers and raters. Hopefully these instruments can aid in establishing measures of oral proficiency that more accurately reflect the actual and potential ability of ELLs to use English for oral communication in given contexts. I begin with definitions for several key components of oral proficiency. Then, after clarifying three important limitations of oral proficiency testing as it is currently practiced, I offer several recommendations, both for the testing process itself and for the training of raters, interviewers, and classroom practitioners who may engage in formal or informal assessment of ELLs.

Defining the Components of Oral Proficiency

            Oral proficiency refers to the ability of a speaker to use language appropriately in given contexts for spoken communication. Luecht (2003) discussed proficiency constructs as being viewed in one of three ways: 1) as the accurate use of structural and phonological components of language in application, 2) as holistic competency in all areas of communication, or 3) as contextualized language use for communicative purposes (p. 528). As a broad concept, oral proficiency may certainly be operationalized in various ways, depending upon the needs and goals of those who seek to examine or assess oral proficiency. In addition, because individuals acquire different aspects of language at varying instances, assessors will have difficulty in choosing elements cleanly indicating various levels of proficiency along a continuum. These elements are not neatly separable from one another—each component overlapping and interacting with one or more of the others.

Here, I suggest that the following four elements, referred to here as elements of mechanical performance, are traditionally given priority in oral assessments:

1)                  pronunciation, referring to the physical act of speaking, includes control over the formation of speech sounds (articulation) and word-level intonation, or what Fulcher (2003) calls “the outer manifestation of speech” (p. 25);

2)                  grammatical competence, referring to the ability to produce and understand language that is accurate and acceptable to native or near-native speakers in the areas of syntax and morphology (though such is not to be confused with metacognitive knowledge about grammar, which, as pointed out by Kasper & Rose, 2002, many ELLs possess);

3)                  vocabulary knowledge (or lexicon), which may be considered an additional component of grammatical competence or a distinct construct; and

4)                  fluency, closely related to both pronunciation and grammatical competence, including the ability to speak with rhythm and intonation at a rate that does not distract from the speech itself.

Furthermore, Fulcher (2003) usefully distinguished between accuracy and fluency, noting that errors in either area can indicate learner development (p. 30). For example, a learner whose fluency is disrupted by an overabundance of fillers (e.g., ‘uh,’ ‘um,’ etc.) may be demonstrating a developing underlying grammatical system with which her ability to speak has not yet caught up.

            I recommend here an additional consideration of interactional-pragmatic competence. The two constructs of the interactional and the pragmatic are linked, with interactional competence broader in scope than pragmatic competence. Looking at interactional competence first, we can define this aspect of oral proficiency as one’s ability to use discourse strategies in order to interact effectively with others in oral communication. Kasper and Rose (2002) described this component of oral proficiency as being rooted in the sociocultural notions of intersubjectivity and interactional practices (pp. 56-57). Kramsch (1998) defined it in terms of a learner’s ability to use the social aspects of language in co-constructing communication with another speaker. Regardless of the theoretical perspective one adopts in examining the construct of oral proficiency, the ability to use spoken language to interact with others in socially acceptable ways is an essential element of overall communicative competence and, thus, oral proficiency.

The range of interactional competence ELLs might exhibit in the classroom can be grasped in the following examples, adapted from actual exchanges: First, an ELL enters her classroom and is greeted by her teacher with, “Good morning!” The ELL replies by smiling and saying, “Yes,” rather than the expected response (“Good morning”). In this case, the ELL may lack knowledge of the speech acts that can be used in response to the teacher’s greeting; she is not aware that some greetings are simply returned in kind. In another case, a relatively advanced learner approaches a classmate with the intention of asking to borrow a pen. “Excuse me, might I borrow one pen?” he asks. Aside from the minor grammatical error—using one rather than the indefinite article a—or the more advanced use of the modal might, the ELL demonstrates insensitivity to register; he is using a formal level of English not usually appropriate for use with a peer. Such is in line with Bardovi-Harlig’s (1999) finding that even high levels of grammatical competence do not necessarily indicate that learners will have commensurate levels of pragmatic competence (p. 686).  Thus, likely this learner’s lack of interactional competence could generate more problems in communication than the small grammatical problems he may have.

            In addition to broader interactional competence, learners need to develop pragmatic competence in order to demonstrate oral proficiency. Pragmatic competence may be defined as the ability of a speaker to use language to convey her intended meaning during interaction with others (Kasper & Roever, 2005). As such, the ability to use speech acts correctly is central to pragmatic competence. Speech acts may be thought of as the language used to convey specific meanings or achieve some desired effect (Brown & Levinson, 1987). Examples of speech acts include apologies, compliments and compliment responses, request refusals, and invitations. In general terms, pragmatic competence relates to a learner’s ability to use language in ways consistent with the implicit code mentioned earlier. For those working with ELLs, the socially grounded nature of the pragmatic competence component of oral proficiency means that understanding the context in which an utterance occurs is a very important part of assessing the overall oral proficiency of the learner. For oral proficiency testing, this context dependency has implications for interviewer and rater training, which is important in light of Bardovi-Harlig’s (1999) finding that even high levels of grammatical competence do not necessarily indicate that learners will have commensurate levels of pragmatic competence (p. 686).

Limitations of Oral Proficiency Testing

One major limitation affecting oral proficiency assessment of ELLs is the interviewers’ and raters’ tendency to be largely unaware of the pragmatic and interactional dynamics of the oral assessment process affecting their judgments about learners’ proficiency. Fulcher (2003) has described pragmatic appropriacy as a construct that is outside the awareness of most speakers (p. 39). It is no surprise, then, that interviewers and raters conducting oral proficiency interview-type assessments tend to incorporate pragmatic and interactional performance into their evaluations of learner performance without realizing that they are doing so. This limitation can be viewed as an extension of the validity and reliability question. In his critique of the ACTFL oral proficiency interview (OPI), Salaberry (2000) pointed out that definitions of the construct of oral proficiency, including the notion of communicative language ability, may not recognize the very important role of strategic competence, which may contribute more to success on some oral proficiency assessments than linguistic ability (p. 295). Likewise, Jenkins and Parra (2003), studying the performances of international teaching assistants (ITAs) taking an oral proficiency test found that those who demonstrated non-verbal behavior and discourse strategies more closely resembling native speaker behaviors and strategies were rated higher by interviewers when overall language proficiency was neither clearly very high nor clearly very low (pp. 102-103). Finally, Chalhoub-Deville and Fulcher (2003) discussed the oral proficiency interview as constituting a unique genre of interaction (p. 503).

Another serious limitation of oral proficiency testing is that it tends to suffer from a lack of authenticity, validity, and reliability, often as a result of an incomplete view of the underlying construct of oral proficiency. With respect to authenticity, McNamara, Hill, and May (2002) identified oral proficiency interview assessment as inherently unnatural and contrived, regardless of how well or how poorly it reflects actual speaking proficiency (p. 223). Regarding validity, Salaberry (2000) cited research that suggested that the oral proficiency interview, which is a performance assessment, is not necessarily a direct measure of underlying proficiency (p. 292). Many ESL educators might agree that the performance of a learner on a single occasion might not be indicative of actual speaking proficiency, thus presenting a conundrum for the construct validity of oral proficiency assessment instruments. Likewise, reliability of oral proficiency tests may be questioned because such assessments are also often conducted in a single interview or testing session. As a result, there is a very real chance that a given learner may not be performing at an optimal level, particularly if he just recently arrived in the country, is unfamiliar with the type of assessment being administered, or is distracted by other factors related to being in a new linguistic and cultural environment.[4]   

A final important limitation facing oral proficiency testing is the problem of estimating the difficulty of specific speaking assessment tasks. Because the underlying concept of oral proficiency includes both the mechanical and the interactional-pragmatic components, those evaluating the validity and reliability of test instruments can find assessment tremendously challenging with respect to estimating the difficulty of specific tasks on a speaking assessment. Iwashita, McNamara, and Elder (2001), for example, studied the difficulty of tasks on a picture-cued narrative oral proficiency assessment from an information-processing perspective and found that performance on the test did not correlate with theorized task difficulty.

Recommendations for Testing

Here I offer recommendations for carrying out oral proficiency testing. These recommendations include balancing the mechanical and the interactional-pragmatic in terms of selecting appropriate assessment tasks, reliability enhancement by overtly working with, considering, and incorporating the opinions of ELLs’ teachers, shifting orientation of assessments from the specific nature of tasks toward the more global nature of assessment types, and finally confirming a library of interactional-pragmatic tasks aligned to specific proficiency level descriptions.

Naturally, I first recommend an inquiry into the test design itself. Bachman (2002) identified a need for balance between focusing on tasks and the competencies being evaluated, asserting that attention to assessment tasks, underlying abilities, and constructs is needed in order to obtain useful results from oral proficiency tests (p. 470). Similarly, Fulcher and Márquez Reiter (2003) manipulated pragmatic task features (e.g., directness, politeness) in exploring task difficulty on speaking tests. They found that ELLs and learners of Spanish whose L1 was English responded differently to tasks that varied in terms of social power, directness, imposition, and politeness, suggesting that the pragmatic content of tasks needs to be considered when designing assessments of oral proficiency (p. 339). Taken together, these findings point to the need for a balanced approach to test design that includes both the nature of the test tasks and the extent to which the assessment authentically reflects the underlying construct being assessed. Some loss in the reliability and validity of resulting instruments might be expected if this balanced approach is followed, but the qualitative nature of the instruments would offer the advantage of providing teachers and testers with useful evidence of how the learners tested are approaching and completing the tasks. 

            My second recommendation targets reliability and concerns the utilization of teachers of ELLs in the test design process. Accurate evaluation of the oral proficiency of ELLs is critical to their appropriate placement in ESL programs, which can in turn contribute to fuller participation in the English-speaking community. Teachers of ELLs can be instrumental in facilitating this participation. While test administrators, interviewers, and raters make efforts to produce more accurate and appropriate instruments, teachers are in regular contact with learners and are in many ways in the best position to contribute to our understanding of how learners are actually using spoken English and how their underlying proficiency might be most effectively targeted by assessment. Tapping into the insight of ESL teachers can enhance the credibility and reliability of oral proficiency assessments.

The third recommendation is to focus on the type of assessment instrument used. We may find focusing on assessment instrument type used more beneficial than concentrating solely on individual task characteristics so that learners and interviewers have more central roles in how tasks are approached, thus facilitating tasks that hopefully hone in on a test-taker’s actual level. Thus, this shift in focus from task to type should also allow interviewers and raters to guide the interaction with an eye on how the social and pragmatic contexts of the assessment may be affecting task difficulty for the learner.     

A final recommendation for testing relates to the type of assessment instruments used. Experience may convince some test designers that an assessment instrument is more or less effective at targeting the underlying competence by providing tasks at appropriate levels of difficulty. However, firm research into which means of assessment might best access different aspects of oral proficiency is needed. For example, Malabonga, Kenyon, and Carpenter (2005) tested computer-based assessments of oral proficiency and found that the majority of the test-takers in their study were able to use the computerized self-assessment to select tasks that were of appropriate difficulty levels (p. 74). Luecht (2003) has offered a test development model with both task development and test assembly specifications that should be useful in designing tasks and assessments that are appropriate for learners (pp. 529-530). Considering different modes of assessment could help to address the second limitation, related to the challenge of estimating the difficulty of tasks on oral assessments.

Recommendations for Training

            My recommendations for training of test interviewers, raters, and classroom teachers primarily address the first limitation discussed above, which relates to the need for personnel to be more aware of and responsive to the pragmatic and interactional aspects of oral proficiency tests. First, Kasper and Rose (2002) have posited, based on their review of several longitudinal studies in the area of developmental pragmatic ability, that learners in the early stages of second language acquisition tend to rely on unanalyzed formulae, gradually learning to make use of productive pragmatic elements in their speech (pp. 125-132). This finding suggests that raters or teachers attempting to assess oral proficiency need to take a learner’s developmental stage into account.  It may not be possible for low beginners to spontaneously adapt a known pragmatic formula (e.g., “no, thank you”) to an unfamiliar situation. For example, an ELL who has learned to call his dog using a formula such as “<whistle> Here, boy!” may find it impossible to adapt the formula to a very different situation, such as calling an older woman to walk toward him, which would require an invitation such as “Pardon me, ma’am, but would you mind stepping this way, please?”

By challenging teachers and testers to develop interactional-pragmatic awareness, I recommend the creation of a taxonomy of pragmatic constructions, possibly identified by developmental level, such that teachers, interviewers and raters could consult the rubric or chart when making decisions about an ELL’s performance on a speaking assessment. Such a rubric would have the added benefit of allowing raters to distinguish the mechanical and interactional-pragmatic aspects of oral proficiency more readily as well.

            Next, I recommend that oral proficiency test interviewers be consistently trained to account for pragmatic awareness and interactional dynamics within the conversations that take place in a speaking proficiency interviews. In a similar vein, consistency of interviewer behavior was the focus of Brown’s (2003) research, in which she found that interviewers conducting oral proficiency assessments varied in several areas of interactive behavior, including establishment of rapport, engagement in topical talk, and provision of feedback (p. 3). In addition, Young’s (2002) analysis of several studies comparing proficiency tests found that raters tend to focus primarily on grammatical competence and fluency when assessing lower-level students in interview-type assessments, while for higher-level students, the use of “stylistic devices” was viewed as a discriminating feature (p. 253). 

Adding further support to the call that raters and interviewers be trained to recognize the importance of social interaction and communicative contexts, Bardovi-Harlig and Hartford (1991) suggested that the maxim of congruence is at work in social interaction, and is particularly at issue in talk between native speakers and non-native speakers. Briefly, this maxim holds that one’s contribution to a conversation should be congruent with one’s status relative to the interlocutor in a given situation or context. A corollary to the maxim was added as well: if congruence is not possible, a speaker should use status-preserving strategies to reduce the negative effects of non-congruence (p. 42). In related research, McNamara, Hill, and May (2002) asserted that, with respect to oral proficiency tests, any definition of the construct of speaking must include recognition of the social and interactional contexts in which given speech events occur (p. 224).  

Finally, I recommend that classroom teachers of ELLs be guided to stress the central place of social context in interpersonal communication. Hassall (2004) reported many Indonesian EFL students she had met were regularly taught that it was acceptable to state one’s position or feelings directly in English with little concern for politeness (p. 999). Clearly this type of message is harmful to learners of English in that it fosters a false view of the language and culture, underemphasizing social constraints and politeness strategies. Teachers who have regular interaction with ELLs need to be trained and equipped to help students recognize the importance and breadth of the interactional-pragmatic elements so that learners are not perceived as being inconsiderate or rude.

ELLs must acquire the norms of communication through interaction and authentic language use for the most part. There is evidence that many aspects of interactional and pragmatic competence can be improved by classroom instruction (Kasper & Roever, 2005). In addition, research suggests that exposure to real language used in context is the most important factor in the acquisition of interactional and pragmatic competence (Matsumura, 2003). Thus, classroom teachers of ELLs must expose their students to authentic language and provide them with contextualized instruction, as well as assessment, in order to foster the development of the interactional-pragmatic competence. Pairing ELLs with native and near-native speaking conversation partners, watching and analyzing video footage of actual high school or university classes conducted in English, and assigning learners to interview native or near-native speakers are just a few of the activities teachers could incorporate into their instructional and ongoing assessment repertoires to achieve the goal. 

Conclusion

            The central concern of this paper has been to motivate educators and testers to examine and incorporate interactional and pragmatic elements in their efforts to encourage and evaluate the development of oral proficiency in their students. Without a proper focus on all of the relevant components of overall communicative ability, test results will not accurately reflect learners’ abilities. I hope other professionals in the field will take up the challenge to further our understanding of the important construct of oral proficiency and contribute to more accurate assessment that takes into account the various components of oral proficiency more fully. In particular, classroom-based research into the acquisition and measurement of interactional-pragmatic competence must be a priority if advances in oral proficiency assessment of ELLs are to continue.

References

Fulcher, G. (2003). Testing second language speaking.  New York: Pearson Longman. 

Kramsch, C. (1998). Language and culture. Oxford: Oxford University Press.

Author Bio

Justin Jernigan is currently an Assistant Professor of English as a Second Language at Georgia Gwinnett College and has over ten years of ESL/EFL teaching experience.