Neural machine translation tools in the language learning classroom : Students ’ use , perceptions , and analyses

Hannam University nrbteach@gmail.com Web-based machine translation (wbmt) tools have long been used by language learners, but until recently, their value as learning tools was limited by the limited accuracy of their outputs. In the past few years, however, the advent of neural machine translation has drastically improved the accuracy of wbmt, considerably increasing their attractiveness to language learners. Accordingly, the present exploratory study seeks to delve into students’ attitudes and beliefs regarding the use of wbmt tools for English language learning. Surveys are used to collect data from 80 upperyear Korean-speaking university students regarding their use of and attitudes toward using wbmt tools. The results indicate that the majority of students use them to support their language studies both at home and at school, and for a range of purposes. Most students reported having limited trust in the accuracy of the outputs, but in general, the results reveal disparities among students in terms of their dependency upon and perceived value of such tools. Furthermore, the students evaluated the output of two popular wbmt tools, revealing evidence of struggle in terms of their ability to critically analyze their outputs. The pedagogical implications associated with this issue are discussed.

Web-based machine translation (wbmt) tools have long been used by language learners, but until recently, their value as learning tools was limited by the limited accuracy of their outputs.In the past few years, however, the advent of neural machine translation has drastically improved the accuracy of wbmt, considerably increasing their attractiveness to language learners.Accordingly, the present exploratory study seeks to delve into students' attitudes and beliefs regarding the use of wbmt tools for English language learning.
Surveys are used to collect data from 80 upperyear Korean-speaking university students regarding their use of and attitudes toward using wbmt tools.The results indicate that the majority of students use them to support their language studies both at home and at school, and for a range of purposes.Most students reported having limited trust in the accuracy of the outputs, but in general, the results reveal disparities among students in terms of their dependency upon and perceived value of such tools.Furthermore, the students evaluated the output of two popular wbmt tools, revealing evidence of struggle in terms of their ability to critically analyze their outputs.The pedagogical implications associated with this issue are discussed.

Keywords:
Web-based machine translation; English as a foreign language; Neural machine translation; Korean students; analysis of translator output introduction Sociocultural theory posits that language learning is mediated by the support of the teacher, collaboration with peers, and various tools (Vygotsky, 1978).With respect to tools, dictionaries have long been a coveted tool for fostering self-mediated language learning.In the Korean university context where smartphones have become ubiquitous, students can easily access electronic dictionaries (eds) as well as wbmt tools.Until recently, however, wbmt output was typical of poor quality and easily detected by teachers; seriously limiting their value as language learning tools.In 2016, however, Google introduced to the world neural machine translation (nmt; Lee, Lee, Kim, & Lee, 2015), marking a significant shift in the digital language learning landscape by markedly improving the accuracy of wbmt outputs.In terms of English language learning, those outputs can provide students with advanced English ability levels a point of comparison from which they can analyze their own written output.For students with limited proficiency levels who are struggling to keep pace with their peers, especially in competition-driven learning environments such as Korea, reliance upon wbmt outputs may simply be regarded as a necessity.
In order for language learners to use wbmt tools effectively, it is essential that they be able and willing to critically analyze the outputs which they produce (Fountain & Fountain, 2009).The obvious and highly concerning downside to using wbmt for the purpose of language learning is that it can enable students to shift from cognitively engaged language learning processes to simply typing l1 sentences into the computer and mindlessly scribing the results.For such reasons, teachers are often inclined to regard their students' use of wbmt as laziness (Van Praag & Sanchez, 2015) and can experience frustrations when expected to invest their time providing corrective feedback for the output of a machine (Steding, 2009).On the other hand, however, prohibiting the use of wbmt based solely on the assumption that students are incapable of critically analyzing their outputs could be depriving them of a potentially valuable language learning tool.As we move deeper into the digital age, and wbmt continuously improves in accuracy, language educators cannot afford to overlook the implications that these powerful tools have in relation to the language learning classroom.

Purpose of the study
In recognition of the reality that wbmt tools are readily available to students, and in many cases, are capable of producing outputs of similar, if not higher quality than that of many students (Garcia & Pena, 2011), it is an issue which requires further investigation.Accordingly, the goal of this paper is to provide language educators with information pertaining to the students' use of, evaluations of, and attitudes towards wbmt tools.Specifically, the extent to which students rely upon, trust, and value wbmt for the purpose of language learning are evaluated.In an effort to initiate explorations into Korean university students' critical analytic skills, comparisons between the students' assessments of the outputs of popular wbmt tools and that of several expert raters are conducted.It is hoped that the results of the study will prompt further in-depth studies regarding the use of wbmt in the classroom and ultimately help to develop pedagogical guidelines for teachers to support their students in using wbmt critically and effectively.
2. What are the students' general perceptions towards the use of wbmt tools in relation to language learning? 3. How adept are students at analyzing wbmt outputs and recognizing translation errors?

literature review
Neural machine translation nmt is a relatively new method of translation which was first proposed by Google and Montreal University in 2014 (Lee et al., 2015).In the short time since its conception, the accuracy of nmt has improved rapidly to the point of overtaking the phrase-based statistical machine translation (pbsmt) technique which had been in wide use previous to 2016 ( Lewis-Kraus, 2016).In simple terms, the key difference between nmt and pbsmt is that nmt performs translations based on the content of the entire sentence as opposed to parts of the sentence which pbsmt relies upon.In machine translation, improvements are measured by bleu scores; a metric which is determined by comparing computer translations to human translations (Papineni, Roukos, Ward, & Zhu, 2002).As a benchmark, a gain of one bleu score point is regarded as being very good, whereas gains of two points are considered outstanding.When Google first launched nmt, comparisons of English-French translations were found to reveal an astonishing seven-point gain, forever changing the field of wbmt (Lewis-Kraus, 2016).
Despite the substantial improvements in accuracy, nmt remains far from perfect.Mike Schuster of Google is quoted as saying, "It's much better than it was before, but it's not as good as humans" (Lewis-Kraus, 2016, "Epilogue: Machines Without Ghosts," para.3).He adds that it is not among Google's goals to replace humans in translation, rather it is to help them.In particular, pragmatics remain a challenge for computers.Peter Lepahin, the general director of a major Hungarian translation company, explains that "The more underlying meaning and connotation is present in the text, the more elaborate and sophisticated it is, the clumsier mt becomes" (Keszthelyi, 2017, p. 23).nmt still struggles with rare or unknown vocabulary and has a tendency to over-or under translate some sentences (Tu, Lu, Liu, Liu, & Li, 2016).Since conversational language is characteristically quite context-dependent, as is the Korean language in general (Song, 2005), it is imperative that language learners remain aware that the accuracy of wbmt continues to be bound by such limitations.

Web-based machine translation as language learning tools
Teachers often hold a negative view of wbmt use in the classroom and tend to regard its use as a shortcut that links them directly to their l1 and leads to cognitive disengagement from the language learning process (Van Praag & Sanchez, 2015).Thus, it comes as little surprise that past studies have commonly focused on preventing the inappropriate use of wbmt (Fountain & Fountain, 2009;Steding, 2009).However, it is important that teachers avoid jumping to the conclusion that students are using the tools uncritically.Moreover, as Garcia and Pena (2011) reveal in their study of beginner language learners, it is important to recognize that the accuracy of the outputs can actually surpass that of what the students are capable of producing on their own.It is perhaps for this reason that Niño (2009) found students of weaker language abilities to be more likely to use wbmt tools than stronger students among learners of Spanish.
Simply prohibiting the use of wbmt in the classroom has been found to be largely ineffective, as students continue to use them regardless of such regulations (Cook, 2010;Fountain & Fountain, 2009;Steding, 2009;White & Henrich, 2013).White and Heinrich (2013) argue that language teachers' efforts are better spent working with students to help them learn to use wbmt tools effectively.They also note that students tend to focus more on producing linguistically accurate sentences than creating linguistically complex sentences or expressing personal voice in l2; two of the key aspects which are commonly lost in wbmt.It is worth noting that the debate over the utility of wbmt is, in many respects, analogous to debates which emerged about two decades ago regarding electronic dictionaries (eds).Tang (1997), for example, dismissed concerns that students necessarily use eds indiscriminately to translate unfamiliar words.It was found that eds actually help students to "bridge the gap between their prior knowledge and new knowledge" (Tang, 1997, p. 55-56).Similarly, Myers (2000) investigated the use of handheld translators among Chinese learners of English, reporting that they practiced saying the unfamiliar words which they searched for, kept notes about the words and phrases they discovered, improved their spelling, and quickly developed a preference for searching words in English rather than in Chinese.In short, there are undoubtedly some positive effects associated with translation in language learning.
This study is particularly concerned with the students' use of wbmt for the purpose of facilitating written and oral communication.Swain (1993) writes that "producing language forces learners to recognize what they do not know or know only partially" (p.159).In the absence of mediating artifacts, however, the cognitive demands of language production created by such gaps can leave students, and particularly those of limited English proficiency levels, feeling overwhelmed and at risk of withdrawing from classroom participation (Payne & Ross, 2005).While it would be difficult to argue that indiscriminate use of wbmt is an ideal solution, it is essential that teachers recognize that translation itself is a natural part of language learning.As Rogers (1996) points out, "Second language learners frequently find themselves attempting to translate from their mother tongue into their new second or foreign language, particularly to fill lexical gaps or to locate l2 words which are only dimly recalled" (p.69).Similarly, Cook (2010) argues that translation "develops both language awareness and use, that it is pedagogically effective and educationally desirable, and that it answers student needs in the contemporary globalized and multicultural world" (p.155).

English language production in the Korean university context
Bloom's Revised Taxonomy (Krathwohl, 2002) represents a hierarchy of thinking skills ranging from lower order to higher order.The categories include 1) remembering, 2) understanding, 3) applying, 4) analyzing, 5) evaluating, and 6) creating, with the latter two skills representing higher order skills.It is well-documented that the English education in the Korean public-school system is focused heavily on the tests such as the College Scholastic Aptitude Test (csat) that are heavily focused on lower order thinking skills (Choi, 2012;Jeon, 2010;Kim, Hutchinson, & Hall, 2015).Even at the post-secondary level, studies reveal that higher level cognitive dimensions of evaluating (checking and making judgments) and creating (planning, generating, and producing) are much less commonly emphasized (Lee, Lee, Makara, Fishman, & Hong, 2014;Baek, 2016).Writing courses, for example, have been criticized for being excessively product focused and lacking in the feedback which is essential to supporting individuals in the development of process-based writing skills (Kim & Kim, 2005).Similarly, active student participation in university speaking courses is commonly hampered by the students' high school English learning experiences which tend to be heavily focused on test-taking and reception skills rather than the development of practical communication skills (Choi, 2012;Jeon, 2010;Kim, Hutchinson, & Hall, 2015).
It is also of critical importance that teachers recognize the reality in which student language learning histories are often far from uniform in Korea.Varying levels of access to a thriving private English education industry which are largely determined by the financial resources of a students' family can result in substantial disparities among the students' ability levels (Bae, 2010).Although students with privileged learning backgrounds often thrive in communicative language learning environments, attempts to engage students of limited proficiency levels often fall flat (DeWaelsche, 2015).Even though students of limited English proficiency may recognize that using wbmt to produce their English output is not ideal, it may be regarded by many of those students as the only way to keep pace with their more able peers in competition-fueled learning environments.

Participants
This study analyzes data collected from 80 Korean university students enrolled in their second, third, and fourth years of study in an elective, communicative English language course at a private university in South Korea.The students ranged in age from 20 to 29 (mean = 22.8) and were divided among four classes, each of which was taught by the researcher.Thirty-three participants were male and 47 were female.A total of 85 students were enrolled in the classes, but three were omitted from the study due to being absent on the day of data collection, and two surveys were discarded for being incomplete.Twentyseven (30.0%) of the participants were of business-related majors, 16 (20.0%)social sciences, 15 (18.8%) humanities, 11 (13.8%)arts, 10 (12.5%) engineering, and two (2.5%) from each information/technology and natural sciences, respectively.The course was subjected to a relative grading system, fueling competition among students for grades.
On two occasions, students participated in group-oriented activities in which students collaborated to translate Korean conversations to English.This was designed to familiarize students with the challenges and inherent subjectivities associated with translation.They were also required to write answers in English to various interview questions on a weekly basis in preparation for the various conversational activities which followed.The teacher warned against the limitations of wbmt tools and provided students with a demonstration of their limitations at the beginning of the course.As an alternative for coping with gaps in knowledge, students were encouraged to use mobile ed applications which included sample sentences and corresponding translations.However, there were no strict rules prohibiting the use of wbmt tools.

Data elicitation
The surveys were distributed at the beginning of the class in the 15th week of study and students were provided with about 20 minutes to complete them.Students were informed that participation in the survey was voluntary and that identities would remain anonymous.Surveys were written in the students' l1 (Korean) and consisted of three sections: (1) self-reported English and translation abilities as well as information regarding their use of wbmt tools; (2) nine Likert-style items designed to measure use of and attitudes towards wbmt tools; (3) assessment of sample English translation outputs.
To provide a point of comparison, the students were asked to self-report abilities and experiences in relation to other Korean university students.Whereas the students' reports of general English ability were conducted with a five-point scale which included 'average' as an option, response options to Korean-English translation ability level pressed students to choose between 'good' and 'poor' by eliminating the neutral option.Similarly, in order to gain insights into the students' experiences with written production, they were also asked to report their level of experience with respect to written English.The Likert-style questions in the second part of the survey required students to respond to the items by choosing from the following options: 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree (neutral), 4 = agree, and 5 = strongly agree.
In the third section of the survey instrument, students were asked to evaluate the English translation of a sample Korean text (Lee & Clyde, 2011; see Appendix A).The text, consisting of a simple conversation pertaining to the 2018 Winter Olympics in Korea, was chosen for its contextual relevance.The genre of conversation was selected over more academic texts because the course was primarily focused on non-academic, conversational and communicative English.Alternate versions of the survey were randomly distributed to the students so that forty students evaluated the output of Google Translate, and forty students evaluated the output of Naver Translate.Details of the two wbmt tools are detailed below.
Google Translate.Google has been utilizing Neural Machine Translation (gnmt) since 2016 and boasts service for over 100 languages.In the current study, translations were made using Google V.5.9.0.This application provides electronic translations and also includes a text-to-speech feature which allows students to listen to the output in either English or Korean.Also featured is a back-translate feature which allows students to check the how accurately the English translation output translates back into the original l1.
Naver Translate.Naver Translate is operated by Naver, a major computer corporation in South Korean.The version used in this study is V.2.2.0.Naver's wbmt (known as n2mt) has direct links to its electronic dictionary, allowing students to investigate the meaning of individual vocabulary words in either English or Korean with a simple click.From there, they can also view sample sentences, effectively accessing information related to collocations and contextualized use of the vocabulary words.Like Google, Naver Translate also includes a back-translate feature.
Assessment of Translator Outputs.The task of evaluating the quality of a translation is marred by a substantial degree of subjectivity.As Papineni et al. (2002) explain, Typically, there are many "perfect" translations of a given source sentence.These translations may vary in word choice or in word order even when they use the same words.And yet humans can clearly distinguish a good translation from a bad one.(p.312) For the purposes of this study, the students were tasked simply to rate the sentences as 'acceptable', 'unacceptable', or 'uncertain'.In this way, judgment of the translation output is designed to focus on simply determining whether or not the reader feels that the original meaning is fully and accurately conveyed to the reader as opposed to a detailed linguistic analysis.Since the ratings of the translation outputs are inherently subjective, six expert raters were recruited to provide a point at which the students' assessments could be compared.The raters consist of three Korean raters (kr), each with advanced degrees in English related fields, and three native English-speaking raters (nesr), each of which were proficient readers of Korean and experienced with Korean-English translation.The raters are described below in Table 1.

Survey data
The data from the first two parts of the survey were culminated and summarized in the results section.Since the responses to items regarding wbmt use were open-ended, codes were assigned to the responses and then grouped into categories.The students' responses to the Likert-style items were converted to percentages, and the mean overall scores and standard deviations were calculated.Finally, using spss version 21, a correlation analysis between student responses to the Likert-style questions and their self-reported ability levels and experiences were calculated.

Analysis of web-based machine translation outputs
Scores of one (+1) were assigned to translations assessed as 'acceptable', scores of zero (0) were assigned to assessments of 'uncertainty', and scores of minus one (−1) were assigned to translations deemed by the participants to be 'unacceptable'.By calculating the mean score for each sentence, the overall student assessment of the translations' accuracy was calculated.In this way, ratings for each sentence received an overall score ranging from −1 (unanimously unacceptable) to +1 (unanimously acceptable).As a point of comparison, the students' evaluations of the translations were compared to the assessments of the six expert raters.The correlation coefficients between students' and expert raters' assessments were similar for both the Google and Naver translation outputs (r = 0.770 and r = 0.734, respectively).Table 3 below reveals the self-reported English language abilities and their perceived abilities to translate Korean to English.The results show that more students perceive their English ability to be below average (37.5%)than above average (12.5%).The remaining half (50%) of students reported the belief that their ability is average.In terms of translation ability, more students rated their ability to translate from Korean to English as either poor or very poor (66.3%) than good or very good (33.8%).

Student use of web-based machine translation
Sixty-eight (85.0%) of the students reported using wbmt either inside or outside of the classroom.Fifty-three (66.3%) reported using wbmt tools in class, and 44 (55%) reported using them outside of class.Thirty-six of the students (55.4%) who reported using wbmt tools reported using English-Korean and Korean-English translations roughly equally, 19 (29.2%) reported using them mostly for Korean-English translations, and 13 (19.1%)used them mostly for English-Korean translations.There were two main translation tools that students used; Naver (n = 45) and Google (n = 30).Papago, a recently developed translation application developed by Naver, was also mentioned by six students.15 students (18.8%) reported using more than one wbmt tool.
Table 4 below provides a summary of the purposes reported by the students for using wbmt tools both outside of and inside of the school.The results reveal that, in both contexts, searching for the meaning of vocabulary was the most common purpose.Outside of school, students reported using wbmt tools to complete school assignments and to support their efforts to write in English as the second and third most common reasons.Inside of school, the second and third most commonly reported reasons were to facilitate oral communication and to write in English, respectively.The reported frequency of wbmt use is summarized below in Table 5. Outside of school, occasional use was most frequently reported, whereas students most frequently reported using them a few times a week in the school setting.Both inside and outside of school daily use of wbmt tools was the second most common response.

Student attitudes toward web-based machine translation
Student trust.The results of the Likert-style items are reported below in Table 6.It is revealed that a large number of students (41.3%) do not trust the outputs to produce their writing assignments.However, it is striking that a substantial portion of students (23.8%) reported that they do trust the outputs of wbmt for such assignments.This may, in part, be explained by some of the students' lack of confidence in their own writing ability, as is indicated by only 13.8% agreement and 50% disagreement to Item 9 (I trust the accuracy of my own Korean to English translations more than electronic translations).The results of Item 11 indicate that slightly over one-third (33.8%) of students have put their trust in wbmt outputs to complete English assignments in the past.The results of Item 15 are indicative of the students' confidence in their analytic abilities, with 62.5% reporting the belief that they would be able to detect mistakes in wbmt outputs, and 12.6% indicating that they would not.Value of wbmt tools for language learning.In Table 7 below, Item 10 reveals that nearly half (48.8%) of students reported the belief that wbmt has value as a language learning tool, whereas 21.3% disagreed with this notion.Item 12 shows that few students (12.6%) believe that wbmt tools are a detriment to language learning.Similarly, Item 16 reveals that only 13.8% of students surveyed are opposed to the idea of using wbmt tools in the classroom, whereas over half (51.3%) believe that their use should be permitted.

Student dependency on electronic translators. Table 8 reveals the students' tendencies to
rely or depend upon wbmt in their language learning processes.Item 13 reveals that only two students (2.5%) agreed that wbmt tools negate the need to engage in the process of learning to write in English, whereas the vast majority of students (91.3%) expressed disagreement with this idea.On the other hand, a considerable number of students (48.8%) reported having previously relied upon wbmt tools to interpret English words, phrases, or sentences which are beyond their level of comprehension.
Correlation analysis.Table 9 below reveals the statistically significant coefficients that resulted from a correlation analysis of the students' abilities and experiences and their attitudes toward the respective survey items.Weak but statistically significant correlation coefficients were revealed in relation to Item 9 (I trust the accuracy of my own Korean to English translations more than electronic translations) and each English writing experience, self-reported English ability, and self-reported Korean-English translation ability.Similar coefficients were also revealed in relation to Item 15 (If the electronic translator makes an error, I am likely to notice it) and the respective measures of experience and ability.
In other words, the students' confidence in their own writing abilities and abilities to notice errors produced by wbmt tools tends to be slightly higher among students who report having relatively greater English ability and writing experiences.Further details pertaining to the correlation relationships among the respective survey items are available in Appendix B. The results reveal that line A2 (Is PyeongChang's third bid for Olympic bid?) to be the most unacceptable translation from the perspective of the students with a score of −0.03.Only six of the fourteen students (42.9%) who rated this sentence as unacceptable offered an alternative translation, however, most demonstrated difficulties in providing translations that offered substantial improvements upon the original output (for full details, refer to Appendix D).The students seemed to remain heavily influenced by the original Google Translate output, typically doing little more than omitting or changing a small number of words.An example of this can be observed in the student suggestion 'Is PyeongChang's third for Olympic bid?' (Female, 21, Natural Sciences) in which the repeated word 'bid' is simply deleted from the original translation output.In comparison to the students, the experts scored Line A3 (This time, I won the competition cities by a big margin) as the most erroneous translation.The use of the pronoun 'I' in this interpretation can certainly be regarded as incorrect, as 'Pyeongchang' is a place rather than a person, and accordingly cannot be referred to in the first person.Three of the four students who attempted to provide alternative translations for Line A3 accounted for this issue by changing the proverb.For example, 'This time, it won the competition cities by a big gap' (Male, 23, Business).However, as can be observed in this example, each of the students' suggestions for alternative translations failed to identify the need for a more appropriate verb choice.
Naver Translate.Table 11 below reveals the ratings for the Naver Translate output.On the whole, the students rated the output only slightly lower than Google Translate's output with a score of +0.50.In comparison, the expert raters viewed Naver Translate's output much more negatively, scoring it −0.11 and rating five of the nine translation outputs negatively.Line C4 (Wow, Pyeongchang is finally happy to finally realize its dream) was regarded as least acceptable output by the students.However, of the 22 students who rated this sentence as unacceptable, only four (18.1%) offered alternative translations.The removal of the repeated word 'finally' in the students' alternative translations suggests that this lexical issue is what influenced students to rate the translation negatively.The expert raters generally agreed that the quality of the translation was problematic, scoring it -0.50.In this case, most of the students' translations seem to offer improvements over the original output (see Appendix D).In comparison to the students' evaluations, the expert raters scored Line C1 (Pyeongchang finally arrived!) as the most unacceptable and unanimously ranked it as such.The verb choice 'arrived' in this sentence is problematic because mobility is not a characteristic of the city of Pyeongchang.The only way that this interpretation may be considered acceptable is if one were to interpret the meaning of 'arrived' metaphorically as 'arriving on the world stage'.The results suggest that many of the students were not as sensitive to this subtle issue as the expert raters were, as 17 students (42.5%) reported it as acceptable.It is also worth noting that substantial disparities in scoring were reported for the title, line N (Pyeongchang, Finally, Makes Dreams), and Line D2 (Yes, it was the third challenge.Two times failed.),with students scoring the translations much more positively than the expert raters in both cases.

Research question 1: Students' reported use of WBMT tools
In brief, the results of this study reveal that wbmt tools are being used frequently and extensively among Korean university students, both inside and outside of the classroom.While the participants reported using such tools for a range of purposes, simple vocabulary translation was by far the most frequently reported purpose.This suggests that the lines between wbmt tools and eds are becoming blurred and that it may be the ability of wbmt tools to produce and interpret longer phrases or sentences that is making them a more convenient choice for students.This may be especially true with respect to the most frequently reported wbmt tool, Naver Translate, as it provides direct links to Naver Dictionary.This can allow students to further analyze the accuracy of the translated outputs with minimal effort.Although Naver Translate was the most commonly reported translation tool used by the students, a large portion of the students also reported using Google Translate.

Research question 2: Trust, perceived value, and dependency
The survey results provide insights into the students' general attitudes towards wbmt tools.To begin with, it was revealed that only a small portion of the participants in this study reported trusting the output of wbmt tools for their English assignments, a finding which suggests that even in cases in which students are using translation tools, they are not necessarily doing so uncritically.However, the fact that only 13.8% of the students expressed the belief that their own written English production is superior to that of wbmt tools may be a cause for concern.This result may be attributed to the public school English education system in Korea which tends to place considerably more focus on developing the students' reception skills rather than production skills (Jeon, 2010;Kim, Hutchinson, & Hall, 2015).As a consequence, the students who perceive their written production skills to be lacking may be at a heightened risk of becoming dependent upon wbmt outputs rather than engaging cognitively in the process of language production (Nio, 2009).This point is further evidenced by the correlation analysis in this study which revealed a weak but statistically significant, positive relationship between the strength of the students' selfreported English abilities and their trust in wbmt outputs.
Similar to the findings of past studies (Garcia & Pena, 2011;Fountain & Fountain, 2009;Nio, 2009;Steding, 2009;White & Henrich, 2013), most of the students reported the belief wbmt tools are not a detriment to language learning and that their use should be permitted in class.In general, it seems the students hold a positive attitude as to the utility of wbmt tools for supporting the language learning process.This point is further affirmed by the fact that only a small minority of participants agreed that the existence of wbmt tools negates the need to learn how to write in English, a result which suggests that the vast majority recognize the value of engaging cognitively in the process of language learning.The positive attitudes of the students toward wbmt tools may also be indicative of the students' awareness of the fact that significant improvements in the accuracy of wbmt tools have been made in recent years (Lewis-Kraus, 2016).Whereas past studies have noted that students of lower proficiency levels tend to have greater reliance upon wbmt tools (Garcia & Pena, 2011;Niño, 2009), the improved accuracy of their outputs may tempt even advanced students to use them to support their language learning efforts.

Research question 3: Identification of WBMT output errors
The results of the participant's analysis of outputs of the most commonly used wbmt tools (Google Translate and Naver Translate) show reason for both promise and concern with regard to using such tools for language learning purposes.On the positive side, the students' assessments were reasonably close to that of the expert raters.However, in many cases, the high degree of subjectivity associated with the evaluation of such translations (Song, 2005) makes it difficult to draw concrete conclusions regarding the quality of the students' assessments.On a more negative note, there were a number of cases in which the wbmt outputs contained obvious errors and still received a high number of positive scores by the students.This alone is a matter of concern, but perhaps of even greater concern is the infrequency at which the students demonstrated the ability to provide marked improvements upon the original wbmt outputs.Instead of restructuring sentences, the students tended to simply omit or rearrange a limited number of words.These finds should be concerning to teachers because it is indicative of a lack in terms of the critical analytic skills which are required to use wbmt tools effectively as language learning tools.While this is undoubtedly a concern in any context, it may be particularly troubling in contexts such as Korea where such skills are seldom emphasized in the public English education system (Lee, Lee, Makara, Fishman, & Hong, 2014;Baek, 2016).

Pedagogical implications
There are a number of pedagogical implications associated with the use of wbmt that are worthy of discussion.To begin with, it is important to heed DeWaelsche's (2015) warning that communicative activity can fail to develop among students of lower proficiency.Under ideal conditions, students are assigned to classes with peers of similar learning histories and abilities so that they can be taught ability-appropriate content (Vygotsky, 1978).In reality, however, this ideal is often not achievable in many university contexts.As such, teacher assumptions that wbmt-mediated learning is closely associated with laziness and cognitive disengagement (Van Praag & Sanchez, 2015) may be shortsighted, at least in classes which are designed to focus on oral communication.Rather than struggling and potentially disengaging into a reticent state, wbmt tools can enable struggling students to quickly transfer their focus to oral production, providing them with a beginning 'script' which can afford them the opportunity to participate actively and potentially recognize existing errors if or when they interfere with communication.Since the cognitive demands associated with English production can be overwhelming to students of lower proficiency levels (Payne & Ross, 2005), wbmt tools may actually help students who would otherwise feel hopeless to participate actively in such communicatively oriented language learning environments (Garcia & Pena, 2011).Accordingly, rather than engaging in futile attempts to prohibit the use of wbmt in the classroom (Fountain & Fountain, 2009;Steding, 2009;White & Henrich, 2013), greater efforts need to be placed on helping students to use these powerful tools critically and analytically.

Limitations
This study has a number of limitations and implications for further study.To begin with, as a survey-based, exploratory study, the students' views regarding translator use were limited to the survey items.In-depth, qualitative studies may reveal a greater depth of knowledge as to the reasons that particular groups of students tend to rely on wbmt outputs.In the case of Naver Translate, for example, some ambiguity may have existed with respect to differentiating between the translation function and the dictionary function, as the former contains direct links to the latter.Information regarding the extent to which the students used the accessory features of the respective wbmt tools remains beyond the scope of this paper and represents an area which is in need of further investigation.It also has to be acknowledged that accurate written production was not the focus of the course in which the students were enrolled.Accordingly, it is worth noting that the students may have been willing to settle for 'good enough' when it comes to the accuracy of the outputs because they had access to teacher feedback while preparing their writing primarily as a precursor to participating in communicative activities with their peers.
Considerable work still needs to be done in terms of breaking existing taboos with regard to using wbmt as well as with helping students to use the tools effectively.Further studies aimed at obtaining students' attitudes and opinions through qualitative means (interviews, focus groups, etc.) are needed to help deepen our understanding of the potential benefits associated with wbmt tools.There is also a need for further studies to investigate whether attitudes of wbmt dependency result from the students' general lack of investment in English language learning or if, perhaps, students tend to view them as tools which are necessary for keeping pace with more proficient peers within a competition-fueled education system.Furthermore, investigating the use of wbmt in academic writing courses as opposed to conversational English courses may also yield different student opinions and outcomes in relation to their assessments of wbmt outputs.

conclusion
This study provides an initial glimpse into student attitudes toward the use of wbmt tools which, influenced by the advent of nmt, have recently improved considerably in terms of accuracy.Its primary purpose is to urge teachers to fully acknowledge that, in many cases, the accuracy of outputs may surpass that which students are able to produce on their own.Accordingly, regardless of teacher opinion, wbmt may be regarded as a necessary tool for students, especially when placed under pressure to keep pace with historically privileged and more-able peers in the language learning classroom.Patterns of student use of wbmt tools as well as their assessments of wbmt outputs indicate a need for increased pedagogical emphasis on helping students to develop their productive and analytic skills in English.
In the absence of such measures, student dependency and indiscriminate use of wbmt tools is bound only to increase.Rather than pretending that this powerful tool does not exist or banning its use entirely, it is important for teachers to consider that we may be able to better serve our students by developing lessons geared towards helping them use these tools effectively and appropriately.Lastly, it will be important to monitor how student attitudes towards wbmt tools continue to change in the future as the accuracy of their outputs continues to improve.

Table 1 .
Characteristics expert translation raters

Table 2 .
Table2reveals the students' reports of their past experiences with writing in English.The results show that nearly a third (32.5%) reported having very little or no English writing experience and only three students (3.8%) reported having had extensive experience.Extent of students' past English writing experiences

Table 3 .
Self-reported English and translation abilities *reported in relation to other Korean university students

Table 4 .
Student reasons for using WBMT tools

Table 5 .
Frequency of students' WBMT usage

Table 6 .
Students' level of trust in WBMT outputs

Table 7 .
Student perceptions of WBMT tools to assist in language learning

Table 8 .
Electronic translator dependency

Table 9 .
Correlation coefficients of self-reports and survey items Table10below reveals the students' and experts' assessments of the sample Google Translate output.Overall the students assigned this translation an average score of +0.52, whereas the expert raters gave it a slightly lower score of +0.35 (For details pertaining to the experts' assessments, see Appendix C).
Google Translate.

Table 10 .
Student and expert assessments of Google Translate output