Judul Jurnal : Why normalized gain should continue to be used in analyzing pre-instruction and post-instruction scores on concept inventories
Abstrak :
Penelitian Nissen dkk. pada jurnal terbaru
berargumen tentang penggunaan Cohen's d, sebagai pengganti normalized gain
yang lebih umum digunakan, dalam analisis skor pra-instruksi dan
pasca-instruksi pada concept inventories yang digunakan untuk mengukur
keefektifan pembelajaran. Mereka berpendapat bahwa peningkatan hasil
normalisasi bersifat “prescore biased”. Peneliti memberikan lima contoh,
termasuk satu contoh yang dikutip oleh Nissen, yang menunjukkan tidak adanya prescore
biased ketika data dianalisis dengan cermat, yang menunjukkan bahwa masalah
dalam analisis mereka adalah bias variabel yang dihapus. Kami menunjukkan bahwa
Cohen's d kurang informatif dibandingkan dengan normalized gain ketika digunakan sebagai ukuran parameter
tunggal efektivitas pengajaran, meskipun, seperti yang ditunjukkan oleh
Nissen, d lebih banyak digunakan di bidang lain. Kami percaya bahwa para peneliti
pendidikan fisika harus terus menggunakan normalized gain untuk menilai efektivitas pedagogi pendidikan. Namun,
karena populasi siswa yang berbeda dapat memiliki respon yang berbeda secara
signifikan terhadap pedagogi yang sama, dalam interpretasi normalized gain,
penting untuk mempertimbangkan pengukuran tingkat kemampuan siswa. Dalam
menganalisis perolehan yang dinormalisasi untuk Force Concept Inventory
(FCI), skor rata-rata pada Tes Kemampuan Penalaran Ilmiah Lawson atau SAT harus
dipertimbangkan, karena skor ini sangat berkorelasi dengan normalized gain,
yang mengindikasikan kemampuan siswa mungkin berdampak lebih besar pada hasil
yang dicapai di kelas dibandingkan dengan pedagogi tertentu yang digunakan.
I.
INTRODUCTION
·
In 1997 Hake introduced normalized gain as
a measure of change when the same concept test is used to gauge student
understanding at the beginning and again at the end of a physics course.
·
Normalized gain is the change in the class
average score divided by the maximum possible gain. … normalized gain is the
fraction of concepts learned by a class that were not known at the beginning of
the course.
·
Nissen et al. claim that normalized gains
are “prescore biased,” and that Cohen’s d should be used in place of normalized
gain.
II. NORMALIZED
GAIN IS NOT PRESCORE BIASED; OMITTED VARIABLES BIAS
A. Examples 1
and 2: Harvard and LMU
At Harvard, individual students’ normalized gain is not
correlated with FCI prescore.
In our initial study of 65 LMU students, we indeed found a much stronger
correlation between Lawson prescore and normalized gain.
The correlation between g and FCI prescore for these students was
only a consequence of the correlation between FCI prescore and Lawson score (r =
0.50, p = 0.00001). Students with higher Lawson scores tended to have higher
FCI prescores. This makes sense because it is reasonable that students with
greater scientific reasoning ability would likely learn more in their
high school classes.
B. Example 3:
Finland
Again FCI prescores were significantly correlated with Lawson
prescores (r=0.43, p<10−4), and this again seems to account
for the misleading appearance of gind depending on prescore. Savinainnen’s
results show no prescore bias in using normalized gain gind.
C. Example 4:
Edward Little High School
… Steinert’s classes, which varied dramatically in reasoning
abilities because some were honors classes and some were not, had dramatically
different normalized FCI gains, even though he used the same methods to teach
all classes(VI).
This shows once again no prescore bias for g.
D. Example 5:
Arizona School for the Arts
This is yet another example showing no prescore bias in normalized
gain g.
III. NISSEN’S
DATA; MISSING DATA
·
Nissen et al. reported their own data for
89 courses, showing correlation between gain and prescore (r = 0.43).
Their study did not include analysis of either SAT or Lawson data.
·
Another problem with their analysis is the
large quantity of missing data.
·
Even if there was no systematic error
introduced by the large amount of missing data in Nissen’s analysis, that
analysis suffers from missing variable bias, and therefore one should not
accept their claim that normalized gain is prescore biased.
IV. NORMALIZED
CHANGE
· Marx and Cummings advocated for replacing individual normalized gain gind by individual normalized change c, defined in terms of an individual’s pretest and post-test scores:
One argument they give for this new measure is that the calculation of gind in some extreme cases can lead to strange results.
·
… it is possible for a student to have
learned nothing from a course, justifying a 0, it is hard to imagine a course
(at least an IE course) in which someone actually knew less at the end of the
course than at the beginning.
·
A student with a perfect prescore has no
possibility of demonstrating learning by achieving a higher post-test score,
and so it seems reasonable to delete that data point.
·
Nissen et al. argue in most of their paper
that normalized gain is positively prescore biased, but in their consideration
of the work of Marx and Cummings, they claim normalized gain is negatively prescored biased. In fact, normalized gain is neither positively nor negatively
prescore biased, as we have shown in Sec. II.
V. NORMALIZED
GAIN VS COHEN’S d
·
Nissen et al. argue for the use of Cohen’s
d instead of normalized gain, where d is defined as the difference between two
means divided by the pooled standard deviation s. Nissen et al. proposed to
use it to measure the effect size of the change in average scores on the same
test, pre-instruction to post-instruction:
We have already shown that the claim of prescore bias for normalized gain is not valid. We shall now show just how misleading use of Cohen’s d can be.
· Normalized gain has been used by many for
the last 20 years to provide that valuable information, which is often used to
guide instructors toward use of more effective interactive engagement (IE)
methods, as it did one of us. That kind of revealing information is lost if
one considers only Cohen’s d.
VI. DISCUSSION
… in order
to determine what the value of the gain might imply about the effectiveness of
the pedagogy for the conceptual learning achieved by the students, it is
necessary to consider the class’s scores on either the Lawson test, SAT, or
ACT, because the reasoning abilities of the class, as reflected in these
scores, may well be a stronger determinant of conceptual learning than the
pedagogy that is used in the course.
.png)
