holistic

of Writing

Holistic Scoring:

Often referred to as "impressionistic" scoring
Involves the assignment of a single score to a piece of writing on the basis of an overall impression of it.
Individual features of a text, such as grammar, spelling, and organization, should not be considered as separate entities.
Has the advantage of being very rapid (Hughes 1989: 86).
Holistic Scoring Scale used by Educational Testing Service for evaluating the Advanced Placement Examination in foreign languages:
- works well and can be altered to fit the level of the students and the focus of instruction.
- numerical scale that ranks performance at levels described as "superior," "competent," and "incompetent."
- for each level, the descriptions can be changed to reflect the kind of performance that teachers expect at a given level of language ability.
- reliability of this scoring method is considered good when the raters are trained to establish common standards based on practice with the kinds of writing samples that they will be evaluating (Cooper 1977).

Demonstrates Superiority	9	Strong control of the language; proficiency and variety in grammatical usage with few significant errors; broad command of vocabulary and of idiomatic language
Demonstrates Competence	8 7	Good general control of grammatical structures despite some errors and/or some awkwardness of style. Good use of idioms and vocabulary. Reads smoothly overall.
Suggests Competence	6 5	Fair ability to express ideas in target language; correct use of simple grammatical structures or use of more complex structures without numerous serious errors. Some apt vocabulary and idioms. Occasional signs of fluency and sense of style.
Suggests Incompetence	4 3	Weak use of language with little control of grammatical structures. Limited vocabulary. Frequent use of anglicisms, which force interpretations on the part of the reader. Occasional redeeming features.
Demonstrates Incompetence	2 1	Clearly unacceptable from most points of view. Almost total lack of vocabulary resources, little or no sense of idiom and/or style. Essentially translated from English.
Floating point		A one-point bonus should be awarded for a coherent and well-organized essay or for a particularly inventive one.

Taken from Johnson's Grading the Advanced Placement Examination
in French Language, Princeton, NJ: Advanced Placed Program
of the College Board, 1983.

Analytic Scoring:

A method of scoring that requires a separate score for each of a number of aspects of a task, such as grammatical accuracy, vocabulary, idiomatic expression, organization, relevance, coherence.
Disposes of the problem of uneven development of subskills in individuals.
Scorers are compelled to consider aspects of performance which they might otherwise ignore.
The very fact that the scorer has to give a number of scores will tend to make the scoring more reliable.
In some schemes, each of the components is given equal weight. In other schemes, the relative importance of the different aspects, as perceived by the tester (with or without statistical support), is reflected in weightings attached to the various components.
Disadvantages:
- It takes more time than holistic scoring.
- Concentration on the different aspects may divert attention from the overall effect of the piece of writing. Inasmuch as the whole is often greater than the sum of its parts, a composite score may be very reliable but not valid (Hughes 1989: 93-94)
  "Students should receive a copy of the analytic scoring criteria in advance. They will then know what is expected of them and how to interpret the evaluation of their written work. They can readily see where their strengths and weaknesses lie and can, over time, visualize their progress with subsequent evaluated samples of their writing" (Terry 1992: 247).

Evaluation Criteria for Analytic Scoring

Grammar and Structures	A = excellent control, very few errors B = comprehensible, some errors C = substantial and significant errors D = one or more blocks to communication F = unintelligible
Vocabulary Choice	A = skilled use of vocabulary relevant to context; widely varied B = clear, appropriate, and sophisticated choice of vocabulary C = errors, but evidence of attempts at sophistication and appropriateness D = errors and/or inappropriate choice of vocabulary F = totally inappropriate choice of vocabulary masking meaning
Fluency and Creativity	A = highly creative and original; extremely smooth text; amount of text commensurate with topic and content; good organization B = good degree of creativity; relatively smooth text; acceptable organization; sufficient amount of text C = some attempts at creativity; loose organization; text is rather monotonous due to lack of variety of structures D = hard to follow; organization undermines intelligibility; very little originality and creativity F = no evidence of creativity; no attempt at organization; no apparent structure
Content and Relevance	A = significant, interesting, appropriate; well thought-out; appropriate to assignment B = generally good work, but facts may be unsupported; repetitions or clichés may be apparent C = careless development of data relevant to content D = no effort to make content significant to the composition F = incoherent and wildly inappropriate content

Works Cited

Cooper, Charles R. and Lee Odell, (Eds.). 1977. "Holistic Evaluation of Writing." In C.R. Cooper and L. Odell (Eds.),
Evaluating Writing. Urbana, IL: National Council of Teachers of English, 3-31.

Hughes, Arthur. 1989. Testing for Language Teachers. Cambridge: Cambridge University Press.

Johnson, Leonard W. 1983. "Grading the Advanced Placement Examination in French Language." Princeton, NJ:
Advanced Placement Program of the College Board.

Scott, Virginia Mitchell. 1996. Rethinking Foreign Language Writing. Boston: Heinle & Heinle Publishers.

Terry, Robert M. 1989. "Teaching and Evaluating Writing as a Communicative Skill." Foreign Language Annals.
22 (1): 43-54.

_____. 1992. "Improving Inter-rater Reliability in Scoring Tests in Multisection Courses." AAUSC Issues in Language
Program Direction: Development and Supervision of Teaching Assistants in Foreign Languages. Boston:
Heinle & Heinle Publishers.