Thursday, December 10, 2009

Auditor General fun

The Ontario AG's report was released Monday, and contained two sections of importance to public education in Ontario. One has received some reporting (thanks to Moira Macdonald) whereas the other may have been mentioned but received little to no coverage.
The auditor general examined, as part of a cyclical review of all areas of government, the Education Quality and Accountability Office, as well as the Ministry of Education's Literacy and Numeracy Secretariat. The former definitely received a more positive review than the latter. I was personally contacted a few times by the EQAO both in advance of and after the release of the AG's report Monday-- the opportunities to report on it in print for me are few, but this is an appropriate venue to do so. The AG's report is overall favourable to the EQAO, noting it does its job -- testing students against the established Ontario curriculum and reporting those results in a fair and efficient manner. It's also done so while reducing costs by 20% while providing the same level of service. The office has four main recommendations for the EQAO, which I've copied and pasted below without the EQAO responses. I realize this lengthens this post considerably, but it's important to have them here verbatim.
Recommendation 1: To improve the Education Quality and Accountability Office’s (EQAO’s) test development and administration process and to ensure that student assessments continue to be reliable and objective and that all students are given the opportunity to demonstrate their competence, the EQAO should:
• highlight to principals and teachers any significant changes in the compliance requirements outlined in the guides to administer EQAO testing;
• improve the process for selecting the schools visited by quality assurance monitors to ensure that all school boards and large private schools are periodically monitored;
• assess the equity of including exempt students in the overall assessment results as having not met the provincial standard; and
• identify schools and school boards where the number of exempt students appears to be relatively high and follow up to ensure that exemptions are justified.
Recommendation 2: To improve the assessment marking process to ensure that results continue to be valid, consistent, and reliable, the Education Quality and Accountability Office should:
• consider adopting on-line training for assessment markers;
• examine different methods to increase the number of validity reads for each marker, especially early in the marking process; and
• consider implementing supervisory backreading to help improve marker accuracy.
Recommendation 3: To ensure that assessment results continue to be reliable, consistent, and valid, the Education Quality and Accountability Office (EQAO) should enhance its quality assurance procedures by:
• implementing a formal complaints process to help determine if there are any trends and to identify potential actions that could prevent non-compliance with assessment guidelines or student cheating;
• considering more complete disclosure when test results at a particular school are withheld as a deterrent against non-compliance with assessment guidelines;
• outlining in its administration guides potential penalties for violating EQAO policy;
• tailoring its quality assurance processes to address unique risks associated with different assessments;
• reviewing Grade 9 applied mathematics results to assess whether incorporating EQAO results into the student’s final markis effective in motivating students and, if so, suggest a more consistent approach; and
• investigating any abnormally large variations in school assessment results from year to year and ensuring that they are justified.
Recommendation 4: To further improve its policies and processes and the procedures designed to produce accurate and reliable reports that can be used to improve student performance, the Education Quality and Accountability Office (EQAO) should:
• consider formalizing its pilot initiative to provide more open-ended questions for principals, teachers, and students to obtain better feedback on any concerns with the assessment process and ways to improve it;
• develop a more formal outreach strategy to give all schools and school boards an opportunity to gain further insight into the value of EQAO data and how it can be used to improve student learning; and
• increase the understanding of parents and the general public of how the assessment process enhances student learning.
The EQAO's response to most recommendations demonstrates either that it is making changes to meet the recommendations for pending assessments or that it will take the matter under advisement. Under the third set of recommendations I would love to see those come into play-- as an example, today one has to go to the so-called blue pages at the Ontario College of Teachers to see where some have flaunted test procedures.
Compare that to the LNS portion of the audit, where five recommendations (which I won't post here or this post would start inducing somnolence) point to a need for greater control and accountability. The AG notes both dollars going out the door that aren't adequately tracked or accounted for, as well as recommending the LNS does a better job of tying its efforts into EQ and report card assessments to ensure the impacts of its programs are effectively measured and reported. I particularly liked the recommendation stating all board and school improvement plans should be a matter of public record. These are no doubt full of eduspeak and other terms, however a competent translator (such as a parents' groups, journalists, etc.) could ensure those with interest understood what each says.
Overall, sadly, both these audits got overshadowed by the sexier sections of the report on social program spending and accountability. I lamented this with an EQAO staffer on Monday, noting they got out-sexed in the report and that few media would bother with a 'good news' story on the office's audit when there was waste and scandal to report.


Ret Dir said...

I particularly like these two:

. assess the equity of including exempt students in the overall assessment results as having not met the provincial standard; and
• identify schools and school boards where the number of exempt students appears to be relatively high and follow up to ensure that exemptions are justified.

The additional element required here is to ensure that students are only identified as exceptional (and thus eligible for exemption) using the same criteria between boards. Public boards have often raised concerns in the past that the identification rates in many Catholic Boards seem suspiciously high - which is a broader question than exemptions, since these students if they do write have the right to accommodations in completing the test items, such as having items read to them, or their answers scribed.
I'm a big fan of the EQAO, bu the AG has identified one of its possible weaknesses. It's currently an uncontrollable variable in the EQAO assessments, and thus makes the results from board to board and school to school truely incomparable.

Education Reporter said...


Didn't mention this in the post, but the AG noted the numbers of exempt students have dropped significantly. What remains a concern, as you've pointed out, is following up with the boards who have the higher numbers.
The validity of the assessment is weakened when it doesn't apply to as many students as possible, as the AG points out.


Anonymous said...

Way to stay completely clear of the wasted that LNS is ER, RetDir. - wonderful about EQAO but seriously, that we're not getting value for our tax dollar with the LNS should scream loudly to communities.

I've heard from some classroom teachers that the LNS folks are more of an intrusion and interference than anything else.

Education Reporter said...

The LNS as a concept actually can work very well. Having done a six-week sit in at a 'turnaround' school (now an Ontario Focused Intervention Partnership 1 school), the LNS officer can bring best practices from other boards and schools to the schools they assist. Now, the year I was there some of the staff didn't like the actual person they were assigned that year, but they did like the PD and other tools brought forth in the LNS-funded professional learning community times.

It's the other side-- tracking the dollars and the time, correlating the time spent to achievement gains, that's lacking according to the AG. The school I was at always had money left over to spend on books, but eventually they would spend and account for it.

Plus (and I might regret saying this), at $340 million, it isn't that much in an $18-billion ministry (or is is $19B? I can't remember). Given the number of schools across the province being helped (or not, depending on perspective) that isn't bad. What's bad is they're not tracking the spending or the results.

As to the teachers, if they're at LNS school and they aren't the sort of teacher who plays well with others, they're going to see the experience as intrusive. The good teachers however will accept, welcome, challenge, debate and ultimately improve from having others involved in their teaching.


Anonymous said...

I don't think that the LNS represents good value for the money spent.

Here's a unique idea. How about doing something really out there and adopting better reading programs that would see more kids reading better in the first place?

Getting rid of the Nelson math series would also be a bonus.

Education Reporter said...

Anon 12 Dec. 10:50

Really? Really?
So having most elementary students in the province reading at a B level isn't an admirable result? You think it would be different if different strategies were tried? I think a small bump might occur, but that's it.

Having 75% of eight-year-olds (or 11-year-olds, for that matter) reading at a B level is proving to be more difficult than possible. The LNS is one of the resources that help schools get their F-level or D-level readers become D- and C-level readers (which are both still below the provincial EQ L3 standard).

Critics can and no doubt will use the AG report to slam the ministry and what it's trying to do. Go right ahead. I'll look at the same report and see little criticism of what the LNS is actually doing. Its weakness is in accounting for what's been done.

As an aside, I'll note the AG, in the EQAO section, also noted what a piss-poor job we all do in explaining what exactly a L2 or L3 or L4 actually means. People might have a more realistic understanding if they knew.


Anonymous said...

What would be proof of the effectiveness of the LNS is to identify those successes, those individual schools which have literally reversed their measurable data re: literacy.

Wouldn't the measure of success put and end to whether it was good money for value?

I too have heard that some teachers have issues with the LNS.

What's the truth?


Education Reporter said...


Agreed. I've told the ministry on a number of occasions that it does a poor job of explaining what happens in the LNS and what the secretariat does in schools.

Effectiveness? Improvement, as you've stated. It's out there, but you have to be aware of what schools are OFIP schools when EQ results are published and some boards don't release their lists out of fear the schools will be labelled.

As to the teachers with issues— well, some teachers are OK with the concept. Others never will be, likely because some of them prefer to close the door to their classrooms and teach independently in their own little world. Increasingly, schools and classrooms don't run like that anymore. During the EWA small school conference the "closed-door / open-door" teaching philosophy was brought up again and again.