Motivating with Data

Here is a chart that shows how Michelle's students are progressing on their math facts.  The data points represent the total number of items answered correctly by the class on that day's timed multiplication test.  One thing to note is that each student's test is not the same--they're working on different levels of multiplication, and once they pass one level, they move on to the next.  So improving the total number of facts correct is actually quite challenging, and may get harder as the kids get better.  Lee Jenkins argues that this type of a run chart works to motivate 70% to 90% of students, while only 50% of students are motivated by grades.  I'm curious to see how it works out!  Stay tuned for more examples of data in action!


Evan

Fwd: Methodology Momentum

Happy weekend, everyone.

I'm forwarding an interesting exchange I had recently with Lee Jenkins, another Deming/Baldrige advocate.  He has some intriguing ideas about motivation, goals, and grades.  If you have a few moments this weekend, I invite you to read over our email messages (below), and take a look at the images he attached.  Here are the questions I'm mulling over after reading his work:

1) what is the most motivating way of using data?

2) what's wrong with using traditional grades to motivate students?

3) should students be asked to set their own goals?

I welcome your own thoughts on these matters, and I look forward to continuing the conversation about best practice in pursuit of Baldrige-based Quality.

:)

Evan

Begin forwarded message:

From: "Lee Jenkins" <Lee@LtoJConsulting.com>
Date: August 22, 2011 6:41:42 AM HST
To: "'Evan Anderson'" <evananderson2@gmail.com>, "'Amy Vorderbruegge'" <amyvord@gmail.com>
Cc: "'Margaret Byrnes'" <margbyrnes@aol.com>, "'Sue Deuber'" <sdeubie@aol.com>
Subject: RE: Methodology Momentum

Evan,

1.        Of course you can call me Lee.

2.       I do believe you understand the writing on goals.  Pulling numbers out of the air (the normal way of setting goals) is a huge problem.  The only real number is how we did in the past.  Attached is a picture of a school run chart in their second year of what I call LtoJ.  The goal is to outperform last year.  Note they did this on 6 occasions. Every time you see a star, this is an all-time-best, meaning the students answered at least one more question correctly than any prior week in the year.  Kids get this goal, even at a young age.  Students, do not need to set their own goals.  They have their own run chart and have a desire to improve by having their own personal all-time-best.  Another picture is attached which shows a collection of school starts.  Every time an individual student, a class, or a grade level has an ATB, it is recorded on a star, announced over the PA and posted on the wall.  The teachers say this is one of the very best things they have ever done for special education as these students rarely meet goals plucked out of the air, but often can and do improve, and then improve again.

3.       I am not sure you will want to measure everything.  Since the book was published more work was done, however, with dichotomous rubrics which are more precise than the traditional ones.  I've attached one for you to see.  We've created 4, 6, 8 and even 10 point dichotomous rubrics.

4.       In the book Wad-Ja-Get, the authors quote research stating that grades motivate 50% of the students.  This matches information gathered in my seminars.  Further, teachers tell me that the LtoJ quizzes that are not graded motivate 70-95% of the students.  If they are ever graded, motivation goes down.  (We have figured out a way to fairly grade spelling tests so that teachers are not giving two tests per week.  Write if interested)

Best to you,

Lee Jenkins

Lee@LtoJConsulting.com

480-221-7603 (cell)

From: Evan Anderson [mailto:evananderson2@gmail.com]
Sent: Sunday, August 21, 2011 7:05 PM
To: Amy Vorderbruegge
Cc: Margaret Byrnes; lee@ltojconsulting.com; Sue Deuber
Subject: Methodology Momentum

Aloha Amy, Margaret, and Lee-

I've spent my weekend as I spent most days this summer--reading up on TQL/BBQ, to get focused on what really makes a difference for students. Thank you, Margaret, for the opportunity to preview the new edition of your book.  I've also found Lee Jenkins's book Improving Student Learning incredibly instructive, and I'd like to start a conversation with him and Margaret about the topic of goals.  In Jenkins's work, he states that focusing on the goals too much can be a distraction, since what we're really always going for as our aim is getting better, all the time (increase the positives and decrease the negatives).  Target-type goals, then, tend to fall short because they are very likely to overshoot, leading to disappointment, or undershoot, leading to results that are lower than what could have been.  

If the focus is not on goals, then what? Well, Jenkins presents the three cornerstone charts of continuous improvement:  the class line chart, the student line chart, and the scatter chart.  All of these tools are meant to measure improvement. If the tools are used consistently throughout the year to measure class and individual progress toward the mastery of meaningful content, then quality can be built into the process. And celebrations of success can occur every time that line chart moves up, rather than just when (and IF) the class meets its class goal.  I don't believe he's saying "don't have class goals", but I do believe he's saying "don't make class goals the be-all, end-all of your quality processes".  Intriguing, especially considering my spotted history of setting and striving for appropriate class goals in the K/1 classroom.  I also have a hard time holding on simultaneously to the two Baldrige decrees: know exactly what your students should know and be able to do at the end of the year and give students the power to set their own goals.  Again, it may be my K/1 bias creeping in here, but it always seemed insincere to ask the students to set their own class goal when I already know what it is they need to achieve.

A few useful guidelines from Jenkins (may I call you Lee?):

For a year-long course of study (Singapore Math, for example), give a preview test of all content to be mastered this year.  A sample of questions from the end-of-year test is fine--he suggests using a number of items equal to the square root of the total items to be mastered.  For Singapore Math Level 4, for example, the test has 38 total questions (A+B), so the sample would be about six questions.

Then, quiz the class weekly on a randomly selected set of six of these items.  Plot the total number of items correct on the class run chart, and have individual students plot their own run chart in their data folder. Scatter charts can show the distribution of scores and may help in adjusting instruction.

Finally--here's the kicker--DON'T MAKE THE QUIZZES PART OF THE GRADE.  This way, the message is clear: we are all response-able for the success of the class, and we are using the data as guiding feedback for instruction group improvement.  Improvement is grounds for celebration, but lower or higher performance is neither punished nor rewarded. He presents some alternatives for giving grades, though we at Voyager don't necessarily need grades, now, do we? :)  Our narrative can include specifics about areas of strong understanding and opportunities for development, all of which the student will have a firm grasp on, anyway.  Hmmm . . .maybe the student should write the narrative?

One question I'd like to bring up in this esteemed company concerns the "21st Century Skills".  It is easier, it seems, to track progress as a class in an area like high-frequency word reading, or math facts.  However, these things are not necessarily what we think are most important.  The "4 Cs" of Collaboration, Communication, Creativity and Critical Thinking are more slippery, and require more qualitative data.  Jenkins gives an example of an art rubric score run chart that reflects some of these skills, but such rubrics are more subjective, I would guess.  The big question for me about TQL/BBQ education is "how can we use it to measure what really matters?"

If we can answer that question, I think we will be on our way from "Good to Great" in a hurry.

I look forward to continuing our conversation together, as we move Voyager forward. 

Onward Always, 

Evan Anderson

Training and Methodology Coordinator

Voyager Public Charter School

Honolulu, Hawaii

808-389-0270

Twitter: @UnclePapa

All School Agreements

Who ever thought that reviewing All School Agreements would be such a difficult process! The delta responses mirrored our feelings exactly. Thanks for hanging in there with us. We will search for a more efficient process and bring this to the table again. Clarity in all school systems is an essential part of TQL and I value your input. Our move to everyone on one campus has provided us the opportunity to exhibit All One Family and Unity in Diversity! Let's hear it for being pushed out of the comfort zone!

-Amy

HSA Results 2006-2010

I talked with the guys at the accountability office earlier this week, and they said that the 2011 HSA score report is due out on Friday.  Which is great, because I've been waiting on it since April, when Deb already had the data and made nice charts for us to wring our hands over.

One of many things I don't like about the way the data are usually reported to us is the way they report by grade level, then compare last year's fourth graders to this year's fourth graders, as if that tells us much about our fourth grade instruction.  At a small school such as Voyager, the classes vary so much from one year to the next, that it's really comparing apples to oranges.  

What would be better, I think, is to track each class of Voyager students longitudinally until graduation.  Assuming the tests are equally challenging each year, and generally aligned with state standards (don't know if I can make this assumption), we should be able to compare this year's fourth graders with their performance as third graders last year, and see how well our fourth-grade instruction served them.  Capice?

So . . . I borrowed the last four years of data from the principal's report (thanks, Deb! Awesome organization!), and lifted the math scores by strand (more useful for planning instruction than an overall "math" score).  Then, I followed each class of students as they took the test, year by year.  The result is attached.  Once we get the scores from 2010, I can add the last data point in each of the charts, and then we can take a gander and look for pukas in math understanding.  It's just a little piece of the assessment puzzle, and it's fraught with problems, but more information is usually better, and I figure we should use what we have.  

One caveat about the data reported by "strands":  the "percent proficient" numbers are low because the test has a large margin of error, probably because of the low number of problems in each strand.  So there's a high percentage of kids whose score in a given strand is "too close to call".  Only the students whose score is safely above the standard deviation for that strand is deemed "proficient", or above.  So don't look at the scores that are like 8% and grieve--there were 80% of the students in some cases who MIGHT be proficient, the test just isn't good enough to tell us who is and who isn't. 

But the scores can be useful, I believe, in revealing our relative areas of strength and weakness in math, and which classes may need the most help this year to be successful (seventh graders).

The question of margin of error and statistical significance might also be why we were allowed to slip by with a 61% this year for AYP, even thought the goal was 64%.  At a school as small as ours, a 61% may just as well be a 64%, statistically speaking.  

I'd like to do the same type of charting with our schoolwide DRA and SRI scores, so that we can track progress over multiple years.  Still searching for the Google App that catalogs and analyzes assessment data for us. :)

It was a very good day to be a teacher,

Evan

Methodology Moment

Best Quantum quote of the day, from Elijah in Miss Michelle's class:

Elijah persisted in asking a question to which Michelle had no answer.  "I don't know," she honestly replied.

"That's an ILLEGAL RESPONSE," said Elijah.  "You can ask for more time, or phone a friend, or . . . "

"I guess I need more time," decided Miss Michelle.

Of course, Michelle, if you want to "phone a friend" over email, we may be able to help you out with the answer to Elijah's question!

Speaking of Michelle, attached is a graph of her class's multiplication timed fact test. It can also be viewed on the ILT page of our PLC blog here.  I believe the goal is forty correct in one minute?  This "run chart" shows the total number of problems the whole class got right. We can't wait to see how many they get right next week!  If it's a new class best, then I'd like to join in an impromptu celebration :).  

Thanks for being a fantabulous faculty.  I am truly blessed to have found my teaching home among such talented and committed educators.  And Inez says Voyager still "sparkles".

Here is my Purpose, Vision, and Mission, for those who are interested in seeing it in writing.  Upon further reflection, it looks like my purpose was actually an "aim", so I've revised it a bit.  

AIM: Joy in Learning about Learning

Purpose: To learn (about professional and organizational learning)

Vision: I am the chiropractor of quality.

Mission: I empower myself and my Voyager colleagues to be masterful, creative, collaborative, joyful educators with purpose by gently stretching and aligning our teaching practice, freeing our creative capacity to flow through us, enlivening our classrooms and inspiring our students.

I'll report back on my Quality Factors tomorrow. Thank you for putting so much thought into drafting them today. I thought the process Amy chose for drafting the factors was efficient, and produced useful results. Next step will be to operationally define what the factors mean.

Speaking of "useful", here's an inventive way of turning our Singapore Math baseline assessment "lemons" into lemonade. It's from a Quality consultant named Lee Jenkins, whose books I'm really enjoying:

1) Give a "preview" assessment of the skills to be mastered by the year's end (Most of us have done this already--painful though it may have been).

2) Give short weekly quizzes on a small, randomly chosen subset of the items on the assessment (take the square root of the total assessment items, and that can be the number of questions on the quiz: Levels 3A and 3B, for example, have 31 questions in all. Square root of 31 is about 5, so select five questions at random from a hat each week).

3) Count up the total number of items answered correctly by the whole class, and post it on a class "run chart". Each student can keep their own run chart as well.

4) CELEBRATE every time the class hits an all-time high number of correct responses.

5) DO NOT attach a grade to any of the scores. The goal here is to build learning motivation and keep it high. Grades add anxiety, comparison, and judgment to the learning process, says Jenkins, and research does not support the use of grades in motivating students to learn (whoah . . . that's deep). Of course, DO analyze the type of items students are generally getting correct or incorrect whenever it's helpful to adjust instruction. We need to get as much mileage out of every assessment task as possible.

6) At the end of the year, give the full assessment again, and gape in awe at the amazing results.

Fascinating stuff. If anyone is willing to try out this model of Baldrige-Based Quality, I would be happy to compile the data and get your run chart up and . . . running. :) We could even do a whole grade-level or whole-team run chart!

Thank you for reading all the way to the end.

Evan

Voyager Staff Satisfaction Survey Results

Thanks to the 23 staff members who replied to the survey. Here are the results of the survey. The chart is interactive, so as you "mouse over" the bars on the chart, they will reveal their meaning (you can also use the colors and the legend on the right--I just don't know how to widen the legend box. Your help welcome!).

Looks like our greatest opportunities for improved satisfaction are higher pay and walking the Voyager talk. Where shall we begin?

Staff Interest Survey Results

Dear Voyagers-

I am happy to report a 100% return rate by teachers on the staff interest inventory.  I now know more than I have ever known about the rich background, varied talents, and diverse tastes of our teaching staff.  I will be using this information to tailor our professional development experiences to our unique characteristics as a staff (like guqin music at our next faculty meeting).

Attached are some visual representations of what I've gleaned from the survey thus far:  Why We Teach, How We Learn, Our Music, and Methodology Mastery.  They can also be viewed at voyagerpcs.blogspot.com, for those of you who have gained access.  :o)  I hope you enjoy perusing the data as much as I did. 

My next survey will be a Teacher Satisfaction Survey, which the Baldrige folks recommend we give every 6-8 weeks to gauge morale and adjust our processes when there is a need. I will report the results from this survey within a week of receiving your responses. Thanks for your kokua!

Enjoying the extended weekend,

Evan

p.s.  Sue would like permission to share our individual answers to the question "why do you teach?" for a "Meet the Teachers" bulletin board.  If you are willing to have your answer to that question shared with the larger Voyager community, please reply to this email with the word "yes". 

All About Us

Our Music:


How We Learn:


Our Understanding of Voyager's Three Methodologies:

Why We Teach

Thanks to everyone who has responded to my Voyager Staff Interest Inventory so far! I made a word cloud of our collective answers to the question "Why do you teach?" I think it's beautiful (Click to enlarge):

Welcome to our Voyager Blog!

I have set up this blog as an extension of our Professional Learning Community.  Its purpose is to facilitate ("make easier") sharing the brilliance and challenges of our teaching practice, and to give us access to all the content we want to share.  Here are some of the most notable features:

Video! Or photo slideshows, or powerpoints, or whatever.  Any visual media can be put up here, with the proviso that it enhance our professional learning, and our ability to serve our keiki.  Or that it's just darn cute and heartwarming (see above).  Based on four years of parent surveys, the photos and video have been by far the most popular aspect of my classroom blog, so I want to keep a good thing going!

Google Calendar link (see sidebar):  By using google calendar for our school calendar this year, we can integrate the calendar seamlessly with online calendars we are already using, like iCal and Google Calendar.

Agendas, minutes, and document sharing link (also on sidebar):  Our communication task force's idea last year was to put all the agendas, minutes, and shared documents in one place, so we have a permanent archive of everything that goes down here at Voyager.  These documents can all now live "in the cloud", and we can share them via Google Docs.

Wikis:  Leslie Witten (PLT teacher last year) has created a school wiki for us to share and compile resources as well.  The link is on the sidebar to the right (password is the same as for our wireless network).

Twitter feed:  I am working to include a scrolling, real-time feed of some twitter streams that are relevant to our work at Voyager.  If there's one that you follow that you would like to add, please let me know!

Email posting:  EVERYONE can post to this blog.  You can share videos, photos, websites, stories--whatever you like--by simply emailing the post to evananderson2.joyinlearning@blogger.com.  Try it!

Please give me feedback and tell me what's missing.  I want this to be a fun and functional space for collaboration, so the more input and participation we get, the better.  Below are some slides from the new Voyager teachers orientation on Friday.

Mahalo,

Evan