Note: This is the second of a four-part series on organizational goal-setting.

January is a popular month to set new goals, so I decided to kick-off this year with a four-part series on this very topic. In Part I of the series, I proposed four conditions that organizations should understand prior to setting a goal.

1. Organizations should understand the capability of the system or process under study.

2. Organizations should understand variation within the system or process under study.

3. Organizations should understand if the system or process under study is stable.

4. Organizations should have a logical answer to the question, “By what method?”

Absent an understanding of these conditions, goals are too often “arbitrary and capricious.” Black’s Law Dictionary defines arbitrary and capricious as “a willful and unreasonable action without consideration or in disregard of facts or law.” As I am using the concept as applied to educational organizations and accountability systems, this is akin to “a willful and unreasonable goal without consideration or in disregard of system capability, variation, or stability.” I’ve developed 10 Key Lessons for Data Analysis to serve as the antidote to the rampant “arbitrary and capricious” goals seen throughout the education sector. Analyzing data using process behavior charts (PBCs) is central to these lessons.

In Part II and III of the series I’ll outline the 10 Key Lessons and then tie up the series in Part IV with an applied example from United Schools Network to bring the lessons to life.

Origins of the 10 Key Lessons

The lessons are derived from three primary sources, Donald Wheeler’s Understanding Variation and Making Sense of Data as well as Mark Graban’s Measures of Success.[1] All three sources built on the foundational work of W. Edwards Deming. The list of lessons from both authors are very similar. This is not surprising given that Graban learned about these methods after finding Understanding Variation on his own father’s bookshelf and subsequently learning much from Dr. Wheeler. I’ve taken what I’ve found most useful from all three books and made several modifications that I think will be helpful for users to employ the process behavior chart methodology in educational contexts.

The rest of this post will focus on Lessons 1-5, and then in Part III I’ll outline Lessons 6-10. It is worth noting again that the lessons center process behavior charts like the one included as Figure 1 in my first post of the series. I believe deeply in this methodology because those who fully grasp these methods:

• have the ability to understand messages contained in their data;

• have the ability to differentiate between common cause and special cause variation; and 

• know the difference between reacting to noise and understanding signals.

Lesson 1: Data have no meaning apart from their context.

Anyone looking at the data should be able to answer some basic questions such as: Who collected the data? How were the data collected? When were the data collected? Where were the data collected? What do these values represent? What is the operational definition of the concept being measured? How were the values of any computed data derived from the raw inputs? Have there been any changes made over time that impact the data set (i.e., change in the operational definition of the concept being measured, change in the formula being used to compute the data, etc.)?

Lesson 2: We don’t manage or control the data; the data is the Voice of the Process.

However, we do manage the system and processes from which the data come. A key conception of the systems view of organizations is that improvement efforts require someone from the outside that has Profound Knowledge collaborating with the people working in the system (i.e., students) and the managers that have the authority to work on the system (i.e., teachers, school leaders). If we wish to make breakthrough improvements in our schools and school systems (like the transformation in Japan after World War II that Deming supported), we must make time to work on the system of learning and to continually improve it with the help of our students. The System of Profound Knowledge provides the theoretical foundation for the transformation of conventional classrooms to those guided by quality learning principles.

Lesson 3: Plot the dots for any data that occurs in time order.

The hashtag plot the dots (#plotthedots) was developed by the improvement team at the National Health Service in England.[2] The primary point of ‘plot the dots’ is that plotting data over time helps us understand variation and leads us to take more appropriate action. As Dr. Donald Berwick put it:

Plotting measurements over time turns out, in my view, to be one of the most powerful things we have for systemic learning.[3]

Plotting data on a run chart and connecting consecutive data points with a line makes analysis far more intuitive than data in a table. The run chart serves as the foundation for plotting the dots on a process behavior chart once you have enough data to do so (see Lessons 4 & 5). The bottom line is that both run charts and process behavior charts will always tell us more than a list or table of numbers.

Lesson 4: Two or three data points are not a trend.[4]

The plotting of dots should occur as soon as you have decided to collect some set of data which occurs over time. This encompasses almost all data we are interested in improving in schools. Many of us will have relied on comparing two points from consecutive weeks, months, or years. However, the problem with looking at just two or three points of data is that it tells you nothing about trends nor anything regarding how much the data varies naturally.

Lesson 5: Show enough data in your baseline to illustrate the previous level of variation.

A run chart can be converted into a process behavior chart once we have enough data to construct the natural process limits. In practice, limits based upon an average moving range will begin to solidify when 17 or more values are used in the computation of those limits (when using the median moving range solidification begins when 23 or more values are used in the computation).[5] But we often must work with fewer data points in real life, and useful limits can be computed with as few as five or six values.[6]

The Power of Process Behavior Charts

Continual Improvement uses as its primary tool the process behavior chart, and the methods include visualizing process behavior, interpreting PBCs, using PBCs effectively, knowing how to create useful charts, and understanding the underlying logic of PBCs. There are other tools in the improvement tool kit, but the process behavior chart is arguably the most important. With these charts in hand, we can avoid the “arbitrary and capricious” goals that pervade our sector.

***

John A. Dues is the Chief Learning Officer for United Schools Network, a nonprofit charter management organization that supports four public charter schools in Columbus, Ohio. Send feedback to jdues@unitedschoolsnetwork.org. 

Notes:

[1] Mark Graban’s Summary of Key Points from Measures of Success can be accessed at www.measuresofsuccessbook.com/wp-content/uploads/2019/05/Summary-of-Key-Points-Measures-of-Success.pdf.

[2] England, National Health Service. Making Data Count, 23 December 2019, www.england.nhs.uk/publication/making-data-count/.

[3] Donald M. Berwick, Escape Fire: Designs for the Future of Health Care. (San Francisco: Jossey-Bass, 2004).

[4] You can compute limits for as few as four data points based on the work of: Donald J. Wheeler, Making Sense of Data: SPC for the Service Sector. (Knoxville, TN: SPC Press, 2003), 104. However, when only four values are used to compute the limits, those four values will always fall within the limits.

[5] Ibid., 164.

[6] Ibid., 129.