Classroom Assessment Techniques CAT

Sharyl Eve Toscano PhD, MS, BS, RN-CPN ·
Associate Professor · School of Nursing · University of Alaska


Classroom Assessment Techniques CAT Poster

Context of the Inquiry

Pharmacology is a knowledge based course taken in preparation for application in clinical practice. Group breakouts allow for application of key concepts but often fail to require individual accountability. Traditional testing methods require that students pass a multiple choice exam without much attention focused on what is being taught in class prior to that terminal assessment.

Course Artifacts

Focus of the Inquiry

Classroom Assessment Techniques (Angelo & Cross, 1993)will guide my classroom inquiry. Each classroom assessment technique requires planning, implementation, and responding. By closing the loop prior to subsequent lectures and or activities results from the assessment will direct classroom teaching. Student will complete individual assessments through the course. Individual assessments require individual accountability for course content compared to group break out activities. Group break out activities don't provide feedback on individual students who might be struggling with a concept. Individual questions in class fail to emphasize the number of students in the class failing to grasp a particular topic.


Does individual accountability via CAT assessment placed before, during, and post synchronous learning improve students' perceptions of  learning in three categories: prior knowledge,recall, and understanding; analysis and critical thinking; and synthesis and critical thinking?

 

Course Artifacts

Course Design and Implementation

CATs aimed at assessing skill in synthesis and creative thinking Expository poem (from focused listing/as assigned)

CATs aimed at assessing skill in analysis and critical thinking Defining Features Matrix (as assigned)CATs aimed at assessing prior knowledge, recall, and understanding Muddiest Point (weekly) Focused listing (as assigned)CATs are chosen based on goal inventory. Based on the Teaching goals inventory, self -scoring worksheet higher-order thinking skills and discipline-specific knowledge and skills scored the highest. These results direct me to CATs aimed at assessing prior knowledge, recall, and understanding, assessing skill in analysis and critical thinking, and assessing skill in synthesis and creative thinking. muddiest point, defining features matrix, and focused listing with expository poem.

CATs aimed at assessing prior knowledge, recall, and understanding

 Muddiest Point (weekly)

 Focused listing (as assigned)

CATs aimed at assessing skill in analysis and critical thinking

 Defining Features Matrix (as assigned)

CATs aimed at assessing skill in synthesis and creative thinking
 Expository poem (from focused listing/as assigned)

Evaluation of CAT exercises using GIFT Group Instructional Feedback

1. What works?

2. What doesn’t?

3. What can be done to improve it?

This is done in anonymous small groups turned in to faculty a minimum of 3 times over the semester. CAT choices may alter based on GIFT results.





Course Artifacts

Findings

FINDINGS

Thirty-eight students consented to participate in the study. The GIFT assessment was completed a mid-term and again at the completion of the course. 

Researcher Reflections Muddy Points (8 completed over the semester)
Muddy points were a good opportunity to clarify topics and concepts in the course. Unexpectedly, muddy points also provided a venue to reinforce the scope of nursing practice and the scope of the course in relation to pharmacology. This is evident in the following example, “Many of the muddy points are outside the scope of this course and/or your license so I’ve left those for you to investigate on your own and look forward to what you find. We assess symptoms and protect from adverse events or reactions. We do not proscribe. Many of the questions were related to prescribing medications which isn't consistent with the objectives of this course.”  

5/17/2013
Common themes included clarification request for the process of: enzyme induction, enzyme inhibition, and the effect of protein binding.

Response to muddy points was recorded and posted on BB for review. At the start of the following class I called out on individual students in a Socratic format. Each individual was able to recall key facts identified by the muddiest point review.

ITEM ATTEMPT STATS On Corresponding exam items to muddy point 1

PTN binding 81%, 83% and 83% correct (3 items)
Enzyme induction 92% correct  
Enzyme inhibition 97% correct 


Midway Muddy Point
Many items in the muddy point were not related to the content covered. Students strayed to readings and or exam policy. A reminder of the purpose of the muddy point was reviewed which provided better muddy points to follow. Students may need reinforcement on the purpose of the assignments to improve engagement in the classroom assessments.

6/14/13
I've done the respiratory lecture in pharmacology multiple times and it has been a problem area. My objective has been to convey a strong understanding of rescue, versus maintenance and preventative medications. This time I used the respiratory CAT Matrix prior to the respiratory session where in students struggled a bit prior to class to create categories of rescue, maintenance, and preventative medications. I gave them the grid outline. There were 2 to 3 emails requesting additional specificity on the assignment which would require I do the assignment. I followed up with an email stating, “focus on covering drugs from each class rather than covering all the drugs you can find (prototypes work but where there isn't one listed in the book or in the notes pick one). Just be sure to cover all of the chemical classes. Or, you might focus on functional class and chemical classes within functions which works well for this unit (anti-inflammation; bronchodilator . . . . . )  

Hope this helps. These classroom assessments are meant to be "figured out" as part of the process so the vague assignment instruction is intentional. “The result was positive and for the most part all of the students were able to complete the assignment and had a better understanding of the concepts as introduced in the class. As an instructor the class went much smoother than previous classes taught. One or two students commented after class that the matrix was hard but helped them understand the material. One student submitted notes and 2-3 used different headings than rescue, maintenance, and /or preventive. Instead they used the supplied headings from the book. The headings in which I've grouped them for better understanding is not found in the standard pharmacology books.  

Therefore, it appears a few students were unable to apply unique concepts or terms and perhaps wanted there answer to be “correct” and verifiable in the text. I will see how this theory test on the GIFT 2 which occurs next week.

6/17/13
Consistent feedback on the GIFT 1 CATS ASSESSED DURING THIS GIFT ASSESSMENT PERIOD

Focused listing, expository Poem, Respiratory matrix, and muddy points


Amenable to changes for second portion of the course

Action items shared with students. Summary items will result in brief survey after GIFT 2 where questions come from the narrative data in GIFT 1 and 2

1. Returning matrix to student prior to testing (not sure if information is correct)

      a. The class that follows the matrix is meant to clarify this area however specific feedback seems to be important. There is a problem regarding timing between the assignments and testing.  

      b. Plan to make a second copy of the matrix assignment such that a few minutes at the end of class could be used to partner up and work on some peer feedback on the assignment where the instructor is available for questions. Ideally most of the questions will be covered in the lecture.  

      c. Alternate possibility is to post Master matrix however b has more peer interaction and allows for added learning potential for both the learning and the peer teacher (evidence suggest peer educators learn better if given opportunities where they teach the material) . . . . I will attempt the option in B and will see how that works for the class using assessment on the GIFT 2 

2. More instruction on the matrix  

      a. This is tricky for me because part of the assignment is to figure out the matrix categories and struggle a bit with the information before it is introduced in class. The results have been tremendous so this might be a case where I make a subtle change.  

      b. Plan: will describe what is meant by the terms in the categories and we will view the grid template in class prior to the assignment. I will reassess using the GIFT 2 assessment.


3. Other items relate more to planning of future classes related to grading structure: will consider combining these comments with those from gift two and surveying the students as a group such that each student can weight in so I can get a clear picture of agreement. (follow up after GIFT 2)

Summary findings on GIFT 2CATS ASSESSED DURING THIS GIFT ASSESSMENT PERIOD Cardiovascular matrix, Diuretic matrix, Muddy points                 

Main Themes

What works?

MATRIX  

 

  • Helped organize  
  • Provides quick reference of vast material
  • Helped to clarify readings
  • Studying before (by creating the matrix) is helpful for class time
  • That it is usual (similar assignment for each section)
  • Diuretic matrix because it was in the book
  • info on one piece of paper easier to study because we have to organize it 

What doesn't?

Can't find this information in the book (cardiovascular matrix) (common theme in each group) 

Muddy, sometimes feel rushed to fill them out

What can be done to improve it?

MATRIX should be pass fail and student should be able to do the matrix again if they do it wrong

Mirror info in class better

Suggested resources for CV matrix  

CV matrix better if broken down into what conditions they treat


Muddy point discussion board

Muddy Points
Muddy points were a good opportunity to clarify topics and concepts in the course. 

Unexpectedly, muddy points also provided a venue to reinforce the scope of nursing practice and the scope of the course in relation to pharmacology. This is evident in the following example, “Many of the muddy points are outside the scope of this course and/or your license so I've left those for you to investigate on your own and look forward to what you find. We assess symptoms and protect from adverse events or reactions. We do not proscribe. Many of the questions were related to prescribing medications which isn't consistent with the objectives of this course.”  

Follow up items:
1.b Few students actually picked up the copy of their copied matrix despite the request for copies in GIFT ONE.

1.C A master cardiology matrix was posted which reinforced course content. Students recognized that there were multiple ways to organize the information based on their individual matrices created prior to posting the master

2.b Clarification of matrix headings was helpful for students in getting started on their matrix. A select few students still wanted more direction in these assignments. IMPORTANT FINDING is that the matrix assignments did require students to work independently on course content prior to reviewing that content in class. Feedback supports that this was a high level student task that required effort and attention.  

Other findings: Only three students completed the IDEA survey. Results were not favorable compared to previous offers where ratings are in the top two levels with positive qualitative comments. Previous idea surveys have good participation compared to this offering where only 3 students responded. Conducting the GIFT assessment at the end of the semester might have affected student participation in the IDEA survey. OF interest is that fact that the course was still rated high quality despite the other ratings to the contrary.  

 


Course Artifacts

Reflections

Does individual accountability via CAT assessment placed before, during, and post synchronous learning improve students' perceptions of learning in three categories: prior knowledge, recall, and understanding; analysis and critical thinking; and synthesis and creative thinking? YES

Live course revision might seem radical but when grounded in CAT it was very meaningful for both me and the students. Changes were grounded in the CAT which acted to document achievement and/or lack of achievement of difficulty concepts. Instead of waiting for a future offering to address lack of achievement, I was able to address it immediately and prior to testing.

I was equally surprised by both the result of mutual sharing in the classroom and the importance of explaining the “why” behind the CAT to the students. Mere encouragement and support of student effort was not enough. I had to constantly reinforce the objective behind CAT. Students struggled to create matrix where “the answer was not in the book”. Despite their accomplishment of creating a matrix that was accurate, students still had a desire to see the “right answer”.  This was an opportunity to post a faculty generated matrix which reinforced course content but students were also able to recognize that there were multiple ways to organize information based on their individual matrix they had developed.  Students voiced a desire to continue to work with their CAT.  A look back at the assignment after having had the content in class might add to the students’ sense of accomplishment.  

This type of exercise required a great deal from students in the form of assignments but also in the form of GIFT assessments. Very few students filled out a course evaluation for this offering compared to previous offerings.  

The GIFT supports that:

  • Individuals engaged independently and worked independently on course content prior to reviewing that content in class.
  • This was a high level student task that required effort and attention. As a faculty member I noted that classroom discussion and questioning during the class period following a classroom assessment activity were meaningful and engaged.
  • Muddy points acted to address gaps in learning objective prior to testing which resulted in higher test scores on key concepts related to the muddy points reviewed.

Unexpected Reflection on CATs in the spring semester of 2014

I received my spring IDEA surveys a few weeks into the third offering of pharmacology. I had been disappointed by the summer IDEA surveys which consisted of minimal participation (8% or 3 students) and ratings were much lower than my usual responses.  Only one open ended response was provided and it focused on the need for more class time and discontent with computer testing. I had worked so hard on CATs that summer and the GIFT feedback was promising. I had expected amazing IDEA results or at a minimum I had expected students to mention them in the evaluations. I had already seen evidence of success as part of my inquiry process.

In reviewing my fall IDEA surveys I had 61% participation with very high ratings. Twenty one students provided comments. Unexpectedly, the responses from the second offering of the course supported that: sharing knowledge about pedagogy (in these case CATs) fostered student understanding, engagement, and enthusiasm for the course and perhaps the instructor. In summary students recognized that I had developed and communicated a system that was “effective”, and “evidence based”. Students named specific CATs like the muddy point the “pre class assignments” as helpful in making a “complex” subject “understandable.  What was so insightful was the student’s characterization of the “system” as “unusual” and their recognition that a student needed to “trust her and you will succeed”. The methods I have used over the years are often presented in an open format without strict rubrics or lengthy instruction. By sharing pedagogy, grounding my teaching practices in evidence, sharing my experience AND EMPHSIZING PRACTICE; I appear to have improved their experience. This is my most surprising finding from this MLV project. It makes perfect sense to me. If the student perceives what you’re doing as madness, a faculty member who explains the method behind the madness might transform the perception of madness to a perception of “brilliance”.

Course Artifacts

Faculty Contact

Sharyl Toscano

Associate Professor
School of Nursing
University of Alaska Anchorage
3211 Providence Drive, HSB 352
Anchorage, AK  99508

907.786.6377
setocano@uaa.alaska.edu