Musings from my experiences of PaCT implementation
I have had the privilege of being the Evaluation Associates project lead for Progress and Consistency Tool (PaCT) implementation support across the country in 2017. We have worked as the Consortium for Professional Learning alongside our partners, the University of Auckland, to provide support to individual schools and Kāhui Ako. We have also used the expertise of the mentor schools, who are early adopters of PaCT, to provide a school-based lens on how to get the best out of the tool.
This blog is intended to support those who are using PaCT or those who have some familiarity with it. However, for the uninitiated, PaCT (the Progress and Consistency Tool) is part of the recently developed progress tools, which include PaCT and the Learning Progressions Framework (LPF). They are, in my opinion, one of the most rich and informative curriculum resources available to schools. I think the illustrations within each of the frameworks exemplify the curriculum and what good learning experiences look like better than anything we have had before. Therefore, helping teachers learn these tools has been meaningful and rewarding for me.
When working with schools who are interested in using PaCT, I highlight four fundamental principles. I believe addressing these is helpful, if not vital, for successful implementation. They are:
- Level of sophistication
- Kind of way
- Best describes
I would like to briefly describe these principles and why I see them as fundamental to successful use of the tools.
Level of sophistication
This principle describes the basis on which the progressions within each aspect of the frameworks are developed. Each set of illustrations along the progression describes an increased level of sophistication displayed by students as they respond to text, complete tasks or solve problems. This provides a clear idea for teachers of the type of behaviours, skills and strategies students need to develop and show as they move through the primary years of schooling. A positive consequence I have observed is that it also inspires teachers to think about the opportunities and experiences they are providing their students. Many teachers have commented they would likely start using similar type activities to those shown in the illustrations because they are rich, authentic, challenging and give the students a great chance to show their levels of sophistication. This should not be mistaken as using the activities as assessment tasks!
For me, there are two main implications from this principle.
- If a teacher needs to consider the level of sophistication of a student’s reading behaviour, text response or problem-solving strategies then the teacher needs to know his/her students’ learning behaviours very well. For most teachers, this will not be a problem as they will be observing, conversing with and listening to students on a daily basis through instructional activities.
- Teachers will not be able to consider responses, texts and answers to problems unless they have an understanding of the processes used by the student. For example, two students may both be able to get the same type of addition questions correct but it is possible for one student to be employing considerably more sophisticated strategies to answer the problem than the other student. Therefore, the number of answers correct in a test only tells some of the story. The teacher would need to consider the strategies employed by each student before placing them on the additive thinking continuum.
Kind of way
This term has stuck with me ever since I was introduced to the developing tool in November 2013, and for me it is integral to being able to use the tool effectively and efficiently. ‘Kind of way’ simply means that, when trying to place students against a set of illustrations, teachers need to consider if this is the ‘kind of way’ that the student would:
- write ideas, experiences or information (Writing)
- solve these problems (Maths)
- respond to a text (Reading)
Again, I see two major implications from using the ‘kind of way’ principle.
- Teachers need to know their students and how they write, their reading behaviours and how they answer mathematical problems.
- In relation to the PaCT, it is not an assessment tool in that there is absolutely no intention for teachers to take activities from within the illustrations and ‘test’ students against them. Yes, teachers assess students’ abilities, skills, behaviours and strategies in relation to each of the aspects but using their knowledge of their students. This means PaCT is remaining true to the idea of an Overall Teacher Judgment, with emphasis on the word ‘teacher’. Using activities as tests would indicate to me that teachers do not trust their professional knowledge of their students.
The National Standards were developed using a standards-referenced approach which was quite a move away from a criterion based approach which most schools were employing at the time. In hindsight, I believe more time spent on highlighting and fully explaining this distinction would have made the transition to National Standards smoother. Teachers were used to the criterion-referenced approach where they looked at discrete objectives, skills or learning goals and judged whether students had achieved them or not. The standards brought a much broader, less exacting perspective on how to assess a student’s progress. The Standards/criterion/norm-referenced assessment document from Assessment Online describes the differences well:
“Standards-referenced assessment utilises broad descriptors of expected achievement as reference points in relation to which professional judgments about progress and achievement are made.”
“A criterion-based assessment is designed to judge the quality of a student’s work against benchmarks of expected performance in relation to a specific competency or body of knowledge appropriate to the student, without the need to compare students. That is, an individual student either has the skills, knowledge and understandings being assessed or not.”
In relation to PaCT some teachers are still taking a criterion-referenced approach to making judgments which has led to set descriptors being used as bullet-pointed checklists. This is not the way forward as it is affecting the authenticity of the tool, the power of the illustrations and makes the process very slow, laborious and unmanageable.
We instead need to think about using the principle of ‘best describes’. This principle uses a standards-referenced approach to making decisions. By looking at each set the teacher can identify which set of illustrations best describes what the student is capable of in relation to the aspect of Reading, Writing or Mathematics. ‘Best describes’ is in essence the same concept as ‘best fit’ but I personally prefer it as it seems like a richer term that places more emphasis on the student. The teacher judges which set ‘best describes’ the student’s level of sophistication when responding to texts, answering problems or writing ideas, experiences or information. So it is not ‘mastery level’, ‘working level’ or the best or worst example this student has shown. It is where the student is generally best described.
- Use an ‘on balance’ approach to this ‘best described’ process. e.g. “On balance the student is best described for this aspect at Set 4.”
- Don’t mess with the tool by creating rubrics, bulleted descriptors or checklists. That is not how PaCT is intended to be used.
I am a great believer that if something is made too hard to use, it won’t be used. So as a facilitator with schools I always discuss logistics, efficiencies and practicalities.
When supporting schools, I first and foremost want PaCT to fully replace the school’s previous OTJ process. It needs to replace what has gone before, not be an add on! If it is an add-on, it will increase workload and also possibly decrease the dependability of judgments across students.
Secondly, when beginning the implementation process, planning for using all three frameworks should be considered. This should not be regarded as a long-term project as it is possible to implement the use of all three frameworks in a reasonably short period. It can make sense to start with one framework (Reading, Writing or Mathematics) but I would strongly recommend a quick move to implementing the remaining frameworks.
Additionally, I would recommend for teachers of students in Year 4 to 8 that the most efficient method of making judgments is by aspect, rather than individually. In other words, rather than going through student by student a teacher can look at an aspect and place all students in relation to that aspect at once. This means a teacher can think like this:
“Well if I have placed Emma at Set Three then, from what I have seen and heard, Sio, Emily, Thomas, Jack and Anaru would be at Set Three too.”
The PaCT Good Practice Guide supports this idea by showing this diagram:
The PaCT online tool also supports this process by having the group mode for adding judgments. This mode makes it a lot quicker for entering data aspect by aspect for a class.
There are also efficiencies for those teachers of After 1, 2, 3 year students. While they will generally be needing to make judgments individual by individual, they can consider these three things:
- As junior teachers making interim and anniversary judgments throughout the year they will become very familiar with the illustrations and sets. This fact alone should reduce the time spent making judgments.
- Most students in the After 1, 2, 3 years categories will be best described by one of the first three or four sets. This means high levels of familiarity will be needed only with the lower sets of illustrations.
- The PaCT online tool makes it easy to enter individual data using the ‘All aspects’ mode. This mode allows you to enter all aspects at once for a child which is especially useful if you have made your judgments offline.
The modes for entering data can be found at the top right of the screen and look like this:
- There is always a bit of work needed at the start of any process but teachers can definitely get faster and more efficient when using PaCT.
- Dealing with manageability could be the making or breaking of successful implementation. If the tool is used how it is intended, replaces previous OTJs processes and is not dragged into a criterion-referencing process it can be efficient and extremely worthwhile. However, in my experience if people are unaware of efficient tips, tricks and processes then something new can be easily dismissed as being too tough. It is appropriate that school leaders check in periodically with teachers to manage and overcome any difficulties, misconceptions or inefficiencies that may arise.
So there you have it – my four principles for PaCT implementation. By addressing these principles PaCT can provide a rich source of information in relation to the demands of our New Zealand Curriculum and provide an insight into the progress students make in reading, writing and maths.
To access PaCT/LPF implementation support please email firstname.lastname@example.org with your request and we will be in touch.
The Progress and Consistency Tool website - https://pactinfo.education.govt.nz/
The Learning Progression Frameworks - https://lpf.education.govt.nz/
The PaCT Sandpit - https://pact-sandpit.education.govt.nz/saml/temp_signup/
Assessment Online Standards/criterion/norm-referenced assessment- http://assessment.tki.org.nz/Media/Files/Standards-criterion-norm-referenced-assessment