Merangkum Jurnal KUY (1)

 Judul Jurnal : Why normalized gain should continue to be used in analyzing pre-instruction and post-instruction scores on concept inventories

DOI

Abstrak :

Penelitian Nissen dkk. pada jurnal terbaru berargumen tentang penggunaan Cohen's d, sebagai pengganti normalized gain yang lebih umum digunakan, dalam analisis skor pra-instruksi dan pasca-instruksi pada concept inventories yang digunakan untuk mengukur keefektifan pembelajaran. Mereka berpendapat bahwa peningkatan hasil normalisasi bersifat “prescore biased”. Peneliti memberikan lima contoh, termasuk satu contoh yang dikutip oleh Nissen, yang menunjukkan tidak adanya prescore biased ketika data dianalisis dengan cermat, yang menunjukkan bahwa masalah dalam analisis mereka adalah bias variabel yang dihapus. Kami menunjukkan bahwa Cohen's d kurang informatif dibandingkan dengan normalized gain  ketika digunakan sebagai ukuran parameter tunggal efektivitas pengajaran, meskipun, seperti yang ditunjukkan oleh Nissen, d lebih banyak digunakan di bidang lain. Kami percaya bahwa para peneliti pendidikan fisika harus terus menggunakan normalized gain untuk menilai  efektivitas pedagogi pendidikan. Namun, karena populasi siswa yang berbeda dapat memiliki respon yang berbeda secara signifikan terhadap pedagogi yang sama, dalam interpretasi normalized gain, penting untuk mempertimbangkan pengukuran tingkat kemampuan siswa. Dalam menganalisis perolehan yang dinormalisasi untuk Force Concept Inventory (FCI), skor rata-rata pada Tes Kemampuan Penalaran Ilmiah Lawson atau SAT harus dipertimbangkan, karena skor ini sangat berkorelasi dengan normalized gain, yang mengindikasikan kemampuan siswa mungkin berdampak lebih besar pada hasil yang dicapai di kelas dibandingkan dengan pedagogi tertentu yang digunakan.

 

I.        INTRODUCTION

·       In 1997 Hake introduced normalized gain as a measure of change when the same concept test is used to gauge student understanding at the beginning and again at the end of a physics course.

·       Normalized gain is the change in the class average score divided by the maximum possible gain. … normalized gain is the fraction of concepts learned by a class that were not known at the beginning of the course.

·       Nissen et al. claim that normalized gains are “prescore biased,” and that Cohen’s d should be used in place of normalized gain.

 

II.      NORMALIZED GAIN IS NOT PRESCORE BIASED; OMITTED VARIABLES BIAS

A.    Examples 1 and 2: Harvard and LMU

At Harvard, individual students’ normalized gain is not correlated with FCI prescore.

In our initial study of 65 LMU students, we indeed found a much stronger correlation between Lawson prescore and normalized gain.

The correlation between g and FCI prescore for these students was only a consequence of the correlation between FCI prescore and Lawson score (r = 0.50, p = 0.00001). Students with higher Lawson scores tended to have higher FCI prescores. This makes sense because it is reasonable that students with greater scientific reasoning ability would likely learn more in their high school classes.

B.    Example 3: Finland

Again FCI prescores were significantly correlated with Lawson prescores (r=0.43, p<10⁻⁴), and this again seems to account for the misleading appearance of g_ind depending on prescore. Savinainnen’s results show no prescore bias in using normalized gain g_ind.

C.    Example 4: Edward Little High School

… Steinert’s classes, which varied dramatically in reasoning abilities because some were honors classes and some were not, had dramatically different normalized FCI gains, even though he used the same methods to teach all classes(VI).

This shows once again no prescore bias for g.

D.    Example 5: Arizona School for the Arts

This is yet another example showing no prescore bias in normalized gain g.

 

III.    NISSEN’S DATA; MISSING DATA

·       Nissen et al. reported their own data for 89 courses, showing correlation between gain and prescore (r = 0.43). Their study did not include analysis of either SAT or Lawson data.

·       Another problem with their analysis is the large quantity of missing data.

·       Even if there was no systematic error introduced by the large amount of missing data in Nissen’s analysis, that analysis suffers from missing variable bias, and therefore one should not accept their claim that normalized gain is prescore biased.

 

IV.    NORMALIZED CHANGE

·       Marx and Cummings advocated for replacing individual normalized gain g_ind by individual normalized change c, defined in terms of an individual’s pretest and post-test scores:

One argument they give for this new measure is that the calculation of g_ind in some extreme cases can lead to strange results.

·       … it is possible for a student to have learned nothing from a course, justifying a 0, it is hard to imagine a course (at least an IE course) in which someone actually knew less at the end of the course than at the beginning.

·       A student with a perfect prescore has no possibility of demonstrating learning by achieving a higher post-test score, and so it seems reasonable to delete that data point.

·       Nissen et al. argue in most of their paper that normalized gain is positively prescore biased, but in their consideration of the work of Marx and Cummings, they claim normalized gain is negatively prescored biased. In fact, normalized gain is neither positively nor negatively prescore biased, as we have shown in Sec. II.

 

V.      NORMALIZED GAIN VS COHEN’S d

·       Nissen et al. argue for the use of Cohen’s d instead of normalized gain, where d is defined as the difference between two means divided by the pooled standard deviation s. Nissen et al. proposed to use it to measure the effect size of the change in average scores on the same test, pre-instruction to post-instruction:

We have already shown that the claim of prescore bias for normalized gain is not valid. We shall now show just how misleading use of Cohen’s d can be.

·   Normalized gain has been used by many for the last 20 years to provide that valuable information, which is often used to guide instructors toward use of more effective interactive engagement (IE) methods, as it did one of us. That kind of revealing information is lost if one considers only Cohen’s d.

 

VI.    DISCUSSION

… in order to determine what the value of the gain might imply about the effectiveness of the pedagogy for the conceptual learning achieved by the students, it is necessary to consider the class’s scores on either the Lawson test, SAT, or ACT, because the reasoning abilities of the class, as reflected in these scores, may well be a stronger determinant of conceptual learning than the pedagogy that is used in the course.