Research — MidSchoolMath (2024)

Abstract

For more than two decades, the US has shown the world’s greatest decline in the middle grades on international tests in mathematics.

In part, this is because US curriculum has typically followed a pedagogical format that emphasizes an ‘instructional methodology’ (often referred to as ’stand and deliver’). This approach tends to lead to lower achievement gains. Statistically, a teacher providing information in front of the classroom has one of the very lowest learning effects with students. The Effect Size (ES), or number of standard deviations of improvement is a .22, less than a year of growth. Even in the world of modern technologies, curriculum is still focused on this approach, with a high percentage of learning time devoted to instructional videos.

Attempts to enhance curricular approaches have had only moderately better results. Across all studies in educational research, the ES for most new curriculum trials is a .4, or four tenths of a standard deviation of improvement, and lower on gold standard randomized controlled trials.

Over the past decade, MidSchoolMath has looked to the leading research in the field to find pedagogical methods which would lead to the very highest learning effects. MidSchoolMath only includes approaches to learning and teaching that showed ES greater than a .7 in randomized controlled trials.

Core Curriculum by MidSchoolMath featuring The Math Simulator™ combines the approaches with the highest learning effects into a single math product. The result in the classroom was an unexpected ES of 1.24, among the very highest in educational research on a gold standard National Science Foundation trial. The study was then replicated with similar results, outperforming the nation’s leading publishers in both achievement and engagement.

The Problem

Understanding the problem is the most important aspect of coming to a solution. While it is widely thought that US students are performing poorly in math, the actual problem is a decline in scores called the “mid school math cliff.”

Fourth grade students actually perform above average on international tests in math, and recently scored statistically equivalent to top scoring Finland on TIMSS (Trends in International Math & Science Study). However, by the time US students reach 10th grade, they typically have fallen three tiers, the greatest decline among all OECD countries worldwide. Understanding the problem as a “cliff” helps better define the issues of US students and typical US classrooms, and the merits of proposed solutions.

In TIMSS research, it was found that US students tend to utilize the strategy of “memorization” more frequently than peers in higher performing countries, a strategy which seems to hold as sufficient until the later elementary grades, where the strategy begins to fail. International testing of mathematics ability at the 10th grade level focuses upon application of math coherently, especially extension of math facts to contextualized situations. It is in this testing format, where relevance is considered essential, that US students perform most poorly.

In part, US curriculum is implicated in the “mid school math cliff” by lead authors of the Common Core, and likely related to US textbook publishers. World-wide data show an inverse relationship between the amount of content and student achievement. US textbooks have been among the world’s largest since 1997. The primary conclusion of the Common Core authors is that a more focused, and even smaller curriculum, would likely lead to increased achievement.

Not only does curriculum need to be more focused, but additionally, research indicates that typical textbook models of teaching tend to be didactic, teaching of specific approaches to solve problems as opposed to creating situational contexts that require students to develop strategies to apply to novel situations. Procedural knowledge and “math facts” are not contrarian to strategic methods. It is simply that the pendulum is too heavily bent toward providing procedural knowledge, rather than allowing students opportunity to develop the procedures. According to lead CCSS author Jason Zimba most current curriculum resides in this manner without sufficient rigor. In our perspective, curriculum is better created through a high degree of focus on presenting questions that are perplexing, rich in context, and delivered through approaches that provide proper feedback mechanisms for learning.

Curriculum Design Based on Gold Standard Research

The magnitude of gains seen in most middle school math programs is notably similar to innovations in other areas such as literacy or science. Most new implementations garner an effect size of .4 (ES = .4), which essentially signifies improvement of four tenths of a standard deviation. Translated, this is equivalent to approximately one year of growth. Effect sizes over .5 typically indicate improvement greater than that expected by standard interventions.

In 2010 MidSchoolMath set out to build Core Curriculum and intended to take advantage of cutting-edge research in the field. Professor Emeritus Dr. John Cooney, who had spent three decades in the field to determine what tools and approaches to content would lead to the highest learning effects, led this effort. MidSchoolMath specifically looked for approaches to teaching and learning which showed the highest levels of effect sizes, above .7. Five different approaches were determined to be most essential for effective teaching and learning: Piagetian methods (ES = 1.42), distributed practice (ES = .71), adaptive technologies (ES = .7), immediate feedback (ES = .73), and the testing effect (ES = .7). Properly utilized, all have shown effect sizes that translate to greater than 1.8 years of growth as compared to a year of ‘business as usual’ approach.

Every aspect of the curriculum was developed to utilize these five pedagogical approaches, often combined to achieve greater effects than would occur alone. For example, during development we noticed a large number of programs utilizing machine learning to adapt to students ability level (e.g. a Piagetian approach). The program would first test student proficiency, and then provide students with an instructional video at each student’s ability level. While the practice may appear sound, by utilizing an instruction approach with such a low effect size (e.g. talking head), the gains were significantly lower than should be expected. MidSchoolMath combined the Testing Effect and Adaptive Learning to create Test Trainer Pro, which has a theoretical effect size of a full standard deviation (1.8 years of growth), and anecdotally is considered one of the most powerful curricular tools we have developed to date.

Finally, in addition to utilizing research with proven effects, MidSchoolMath determined that counteracting the “mid school math cliff” would require a completely different level of curricular approach. While traditional curriculum could be enhanced or furthered through background research, the need for a central product - one that could truly transform learning- was determined essential. With the support of the National Science Foundation, MidSchoolMath developed The Math Simulator™, which has now been shown to have extraordinary learning effects on randomized controlled trials.

The Math Simulator™ is truly a significant breakthrough in online math curriculum and resources in three distinct ways:

The simulations provide situations where student input is directly linked to functional output. It is this central and unique mechanism in The Math SimulatorTM that truly drives student achievement: students directly experience how the math actually works while having the opportunity to fail in a safe environment.
The simulations provide a rich context where the math is meaningful, and embedded in a memorable, relatable story-based concept. This advances how a math standard can be genuinely conceptualized.
The simulations helped teachers break the traditional, yet not effective, “stand and deliver” classroom protocol, by removing information (data) and allowing students to determine, on their own, what variables are necessary to solve the problem. This shift in pedagogy is essential to supporting mathematical thinking. By removing the propensity of giving too much information, and encouraging the teacher to pose a natural question, students can focus on the understanding of (1) how the math functions and (2) the concept of the math standard.

Thus, with Core Curriculum by MidSchoolMath featuring The Math Simulator™ and the math classroom experience has been transformed. It supports students and teachers alike with an entirely new structure for classroom activities that draws upon evidence-based protocols to support learning, rather than relying on traditional lectures.

What is Gold Standard Research?

Gold standard research is defined by level of proof and assurance from the Randomized Controlled Trial (RCT) compared to other ways research can be conducted.

Core Curriculum Components & Research Indicators

The primary goal of each curriculum component is to elicit significant achievement gains and increase student engagement, while providing ease of use for the teacher. Each component was included only if supported by research-based evidence. Core Curriculum by MidSchoolMath features The Math Simulator™, which has demonstrated among the highest effect sizes across educational interventions on randomized-controlled trials.

Efficacy Research: Indications of Extraordinary Results

Four studies have been completed to date on Core Curriculum by MidSchoolMath, including a correlational study conducted by the Legislative Education Study Committee state of New Mexico, and three randomized controlled trials conducted under National Science Foundation research. Each study is described below with a discussion section that encompasses all studies.

New Mexico Legislative Education Study Committee Correlational Study

This study is an analysis of schools to determine the grade-level impact of the curriculum in New Mexico schools using results from the 2016 Partnership for Academic Readiness for College and Careers (PARCC) math exam in New Mexico schools. The data is public and can be found on the New Mexico Public Education Department (PED) website.

Study 1 Summary

In 2017, ten schools in New Mexico used Core Curriculum components school wide. As a 3rd party, the New Mexico Legislative Education Study Committee audited math performance of each of the schools to determine if the schools were performing at the statewide average. Results indicated that scores in schools utilizing MidSchoolMath were higher in all grades, with statistical significance in Grade 6, 7 & 8. Tim Bedeaux, who conducted the correlational study, wrote: “When interpreting these results, it’s important that we don’t conflate correlation and causation. That being said, correlation is a fairly decent conclusion to reach in this case. For the purposes of understanding how and why MidSchoolMath makes a difference, it may be enough to simply consider use of MidSchoolMath (MSM) a characteristic of higher performing schools. Even if it’s hard to definitively prove the higher scores are a result of MidSchoolMath (MSM), it’s hard to deny these schools are performing better.

National Science Foundation Randomized Controlled Trials

Three randomized controlled trials have been conducted under National Science Foundation funding. All three trials followed gold standard research designs.

General Methodology

Students took pre-tests, post-tests, and delayed post-tests in their regular math classrooms containing experimental and control students, seated in typical seating to control for any cheating. Not only were students randomized to various conditions, all teachers involved in the studies taught control and experimental conditions, which were assigned to each group randomly. Outcome evaluations in Studies 3 & 4 were developed by a third party, and analysis was conducted by a third party (J. Blanden & Associates), and every effort was made by MidSchoolMath to ensure that implementation of the program occurred in a pattern consistent with ‘business as usual’ methods.

Study 2 Summary

Participants in the evaluation were comprised of 435 students enrolled in a public school in the southwestern region of the United States. The sample was comprised of 51.8% Females and 48.2% males. Ethnic composition of the 6th grade classrooms reflected the district demographics of approximately 56% White, 39% Hispanic or Latino and 2% Black or African American.

Using a randomized controlled design, four teachers utilized a different type of curriculum in each of 3 classes designed to teach a single, complete CCSS standard. All materials were provided to teachers ‘as is’ from publishers. The three publishers in the study included Core Curriculum by MidSchoolMath featuring The Math SimulatorTM and two of the nation’s leading publishers.

Student interest level, achievement gain, and teacher’s perception of ease of use were all measured.

In all categories, MidSchoolMath posted significantly higher results (p < .05). Most notably, on the delayed post-test, MidSchoolMath was the only publisher to post gains with an effect size higher than zero. The full study is in review for publication.

Study 3 Summary

The purpose of Study 3 was to determine the magnitude of effect of Core Curriculum by MidSchoolMath featuring The Math SimulatorTM on a CCSS standard.

Participants were comprised of 265 7th grade students enrolled in a public school in the western region of the United States. The sample was comprised of 51.8% females and 48.2% males. Ethnic composition of the reflected the district demographics of approximately 56% White, 39% Hispanic or Latino and 2% African American students.

A total of 265 7th grade students were randomly assigned to one of two conditions. In the experimental condition students interacted with The Math SimulatorTM involving an activity that posed a series of questions about the properties of the cylindrical granary. Students worked individually and collaboratively to solve the problems. The control group worked individually and collaboratively on an alternative mathematical activity of comparable duration and difficulty. All students completed a 5-item pretest and post-test concerning the properties of the volume of cylinders in their regular classroom the prior to and the day following instruction, respectively.

The outcome of the experiment is displayed in Table 1. Assessment items were sored 0, 1 or 2 to give partial credit (1) for answers that were correct within 1/10 of the answer key. Given the hierarchical structure of the data, students nested within teachers, a multilevel modeling approach was used to analyze the data (Raudenbush & Bryk, 2002). The model included one level-1 predictor; students’ pretest scores centered on their classroom mean, and one level-2 predictor; a dichotomous variable indicating students’ treatment condition (0 = Control, 1 = Experimental).

Posttest scores showed a moderate level of correlation with the pretest scores and the magnitude of the relationship was stronger for the control group than the experimental group. The most important finding however is that the magnitude of the treatment effect, Hedges’ g (Hedges & Olkin, 1985) is substantial (g = 1.24, 95%CI = .972, 1.507). Thus, 89.5% of the students interacting with The Math Simulator™ have a higher level of achievement than the average student in the control group (Cohen, 1988). Put differently, a student at the 50th percentile could be expected to move to the 89th percentile after interacting with the simulator.

Study 4 Summary

Study 4 investigated the optimal placement of direct instruction (before or after) relative to students’ attempts to solve complex mathematical problems, and additionally, was designed to confirm the magnitude of effect of Core Curriculum by MidSchoolMath featuring The Math Simulator™ on a different CCSS standard.

Participants were comprised of 353 6th and 7th grade students from four teachers’ classrooms enrolled in a public school in the western region of the United States. The sample was representative of the school population: 49% females and 51% males with an ethnic composition of approximately 89% White, 5% Hispanic or Latino, 4% Asian and Pacific Islander, 1% African American and 1% Native American students; and 20% eligible for free or reduced lunch.

A total of 353 students in each of four teachers’ four 7th grade classrooms completed a 5-item pretest concerning the calculation of irrational square roots. The four intact classrooms of four teachers were randomly assigned to either a productive failure (PF) or a direct instruction protocol (DI). Students in the PF condition interacted with The Math SimulatorTM module “Treasure Hunt.” The simulator drew on students’ prior knowledge of the Pythagorean Theorem to find the distance to the treasure. A right triangle was overlaid on a map that displayed the square root symbol over a number on the hypotenuse between the present location and the location of the treasure. The simulator asked students to approximate the square root of the number via successive clicks on a number line to find the distance.

First students were asked to locate the whole number portion of the irrational root between two whole numbers on the number line. Next, the simulator zoomed in to the tenths level and asked students to choose between two values on the scale. Finally, values for hundredths were displayed and students were asked to choose the best value to approximate the square root. Students submitted their answers and the result of the distance traveled based on their answer was displayed on the map. No other feedback was provided. Students attempted to approximate 3 different irrational square roots. After completing the simulator activity students watched a video providing direct instruction about how to approximate irrational square roots. All students completed a post-test the following day. Students in the DI condition received the same components of instruction except that the order of The Math SimulatorTM activity and instructional video were reversed. The following day all students completed a post-test concerning the approximation of irrational square roots.

The results of the experiment are displayed in Table 2. Students in experimental and control conditions were essentially identical with respect to achievement on the pretest. The two conditions also appear similar on the posttest with an increase in achievement from the pretest to the post-test. Initial results from fitting an unconditional hierarchical linear model to the data indicated that the magnitude of clustering within teachers’ classrooms was trivial (i.e., .35%). Accordingly, we assumed that student achievement is independent within-classrooms and utilized a simpler general linear model to analyze the data.

Results from a 3-way repeated measures analysis of variance, where treatment conditions (PF and DI) and students’ sex served as between-group factors and the time of test served as the within-group factor. The results are consistent with visual inspection of Table 2. There was a statistically significant increase in achievement between the pretest and the post-test (F(1,349) = 28.1 p < .001); however, there was no main effect of the order of treatment (F(1,349) < 1) on achievement or differential effects of the treatment conditions between the pretest and post-test (F < 1). There was, however, a statistically significant difference between the male and female students (F(1,349) = 8.65, p < .004) where boys (M = 1.26) exhibited higher achievement than girls (M = .99). No other interactions were statistically significant.

Although the increase in achievement from the pretest to the post-test is small in the absolute sense, the magnitude of the change is quite large. Calculation of the effect size of the change from pretest to post-test, from the F-ratio, indicated an effect size of g = 1.18. Thus, regardless of the placement of direct instruction, students interacting with The Math SimulatorTM show a sizeable increase in their ability to estimate irrational square roots. It is important to note that the absolute increase in achievement is less than what would be desired given the alignment of the problems with instruction. Nevertheless, students can be expected to move from the 50th percentile to 88th percentile after interacting with the simulator.

Discussion

Early results of Core Curriculum by MidSchoolMath featuring The Math Simulator™ are promising. A correlative study of schools in New Mexico that utilized MidSchoolMath curriculum showed statistically higher mean math scores. MidSchoolMath also outperformed two of the nation’s largest publishers on a short term, randomized controlled trial of one standard, and was the only publisher to have lasting retention effects from the intervention. Yet it is the magnitude of effect of The Math Simulator™ which is most compelling. On two randomized controlled trials, The Math Simulator™ has led to achievement gains that are in the very top tier of educational research (ES = 1.24).

The magnitude of effect size for The Math Simulator™ is made more compelling as early indications show that both low and high performing student show equivalent achievement gains. In both trials against baseline performance, a student at the 50th percentile would move to the 88th-89th further solidifying the expected results.

Not only the are the magnitude of the results compelling; the quality of the research exceeds that of criteria required for ESSA (Every Student Succeeds Act) Tier 1 compliance. The following four requirements are required for Tier 1 qualification of a program that Shows Strong Evidence of Effectiveness:

At least one randomized controlled trial was deemed by experts to be well-designed and well-implemented and did not have problems with attrition (students dropping out) to meet Tier 1.
Statistically significant positive effects to meet Tier 1.
At least 350 students to meet Tier 1.
At least two educational sites to meet Tier 1.

Core Curriculum by MidSchoolMath meets and/or exceeds Tier 1 criteria, with 3 randomized controlled trials, all deemed well-designed and well-implemented under National Science Foundation gold standard protocols, each of which with statistically positive results at p < .05, p < .01 and p < .0001 values, in multiple educational sites.

Two areas of future research are essential: 1) To determine if the large effects in The Math Simulator™ are related to narrative aspects. 2) To determine if the large effects translate to long-term broad scale efficacy outcomes. Both areas of future research are in process.

Works Cited

Organization for Economic Cooperation and Development (2010, Dec. 7). Presentation of the PISA 2010 Results. Gates, B. (2007, March 7). U.S. Senate Committee Hearing. Washington, D.C. Trends in International Mathematics and Science Study (TIMSS), 2009. Program for International Student Assessment (PISA). (2009) Learning Mathematics for Life: A Perspective from PISA. Organization for Economic Cooperation and Development. Schmidt, W. (2003, February 4). “Presentation to Mathematics and Science Initiative.” Schmidt, W.H., McKnight, C. & Raizen, S.A. (1997) A Splintered Vision: An Investigation of U.S. Science and Mathematics Education. U.S. National Research Center for the Third International Mathematics and Science Study. Laidlaw, S. (2009) Personal observation, comparing Addison-Wesley’s Mathematics: Grade 7, 1995 edition to Holt, Reinhart and Winston’s Math Course 2, 2007 edition. Jobrack, B. (2012). Tyranny of the Textbook: An insider exposes how educational materials undermine reforms. Lanham, MD: Rowman and Littlefield. Common Core State Standards Initiative. (2011) Common Core State Standards for Mathematics. Common Core State Standards Initiative. (2012) Common Core State Standards In the States. www.corestandards.org/in-the-states Zimba, J. (2012, April 26). Common Core Convening. Chicago, IL. Laidlaw, S. (2012) Personal observation, reviewing Pearson’s Algebra I iBook 2. Liu, O.L. (2009). An Investigation of Factors Affecting Gender Differences in Standardized Math Performance: Results from US and Hong Kong 15 Year Olds. International Journal of Testing, 9, 215-237. Anderman, E., Winne, P. Alexander, P., Corno, L. (2006) Handbook of Educational Psychology, 2nd Ed. Lawrence Erlbaum Associates. Stipek, D. J. (1996) Motivation and Instruction. Handbook of Educational Psychology. New York: Simon & Schuster/Macmillan. Kohn, A. (1993) Punished by Rewards: The Trouble with Gold Stars, Incentive Plans, A’s, Praise, and Other Bribes. Boston: Houghton Mifflin. Pink, D.H. (2009) Drive: The Surprising Truth About What Motivates Us. New York: Riverhead Books Bottoms, G. & Timberlake, A. (2012) Improved Middle Grades Schools for Improved High School Readiness: Ten Best Practices in the Middle Grades. Southern Region Education Board. Lombardi, M. (2007) Authentic Learning for the 21st Century: An Overview. Educause. Habgood, M.P.J. & Ainsworth, S.E. (2011). Motivating children to learn effectively: exploring the value of intrinsic integration in educational games. Journal of the Learning Sciences. 20(2), 169-206. Devlin, K. (2011) Mathematics Education for a New Era: Video Games as a Medium for Learning. Natick, MA: AK Peters/CRC Press. Gee, J.P. (2007) What Video Games Have to Teach us about Learning & Literacy. New York: Palgrave MacMillan Squire, K. (2011). Video Games and Learning: Teaching and Participatory Culture in the Digital Age. New York: Teachers College Press Engeser, S. & Rheinberg, F. (2008). Flow, performance and moderators of challenge-skill balance. Motivation and Emotion, 32(3), 158-172. Riecks, M. (2013 Jan. and prior) Personal observation, comparing online math game websites. Bruckman, A. (1999). Can educational be fun? Game Developers Conference 1999. San Jose, CA. Lave, J. (1988). Cognition in practice: Mind, mathematics, and culture in everyday life. New York: Cambridge University Press. Nunes, T. Schliemann, A. & Carraher, D. (1993) Street Mathematics and School Mathematics. New York: Cambridge University Press. Steen, F.F. & Owens, S.A. (2001). Evolution’s Pedagogy: An Adaptationist Model of Pretense and Entertainment. Los Angeles: University of California. Bandura, A. (1977). Social Learning Theory. Englewood Cliffs, NJ: Prentice Hall. Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice-Hall. Bandura, A. (1997). Self-efficacy: The exercise of control. New York: WH Freeman. Mayer, R. E. (2009). Multimedia Learning, 2nd ed. New York: Cambridge University Press. Sarbin, T. (1986). The narrative as a root metaphor for psychology. Narrative psychology: The storied nature of human conduct. (pp. 3-21). New York: Praeger. Olson, D. R. (1990). Thinking and narrative. In B. K. Britton & A. D. Pellegrini (Eds). Narrative thought and narrative language. (pp. 99 - 112). Hillsdale, NJ: Erlbaum. Thorndyke, P. W. (1977). Cognitive structures in comprehension and memory of narrative discourse. Cognitive Psychology. 9, 77-110. Wiseman, S. & Neisser, U. (1974). Perceptual organization as a determinant of visual recognition memory. The American Journal of Psychology. 87(4), 675-681. Bransford, J. D. & Johnson, M. K. (1972). Contextual prerequisites for understanding: Some investigations of comprehension and recall. Journal of Verbal Learning & Verbal Behavior, 11(6), 717-726. Polya, G. (1957). How to solve it. New York: Doubleday. Polya, G. (1981). Mathematical discovery. New York: John Wiley & Sons Woodward, J., Beckman, S., Driscoll, M., Franke, M., Herzig, P., Jitendra, A., Koedinger, K. R., & Ogbuehi, P. (2012). Improving mathematical problem solving in grades 4 through 8: Practice Guide. (NCEE 2012-4055). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. DOE. Roediger, H. L., & Butler, A. C. (2011). The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences, 15, 20-27. Ebbinghaus, H. (1885). Memory: A contribution to experimental psychology. Translated by H. A. Ruger & C. E. New York: Teachers College, Columbia University. Karpicke, J. D., & Roediger, H. L. (2008).The critical importance of retrieval for learning. Science, 319, 966-968. Roediger, H. L., & Karpicke, J.D. (2006). The power of testing memory: Basic research and implications for educational practice. Perspectives on Psychological Science, 1, 181-210. Mayer, R. E., & Johnson, C. (2008). Revising the redundancy principle in multimedia learning. Journal of Educational Psychology, 100, 380-386. McDaniel, M. A., Agarwal, P. K., Huelser, B. J., McDermott, K. B., & Roediger, H. L. (2011). Test-enhanced learning in a middle school science classroom: The effects of quiz frequency and placement. Journal of Educational Psychology, 103, 399-414. Riecks, M. (2013 Jan. and prior) Personal observation, comparing online math game websites. Riecks, M. (2013 Jan. 21) Online search of What Works Clearinghouse. www.whatworks.edu.gov Van Voorhis, V. & Lammers, J. (2013, Jan. 10). Educational Games for Girls: Collaboration and Context. Edweb.net Schell, J. (2008). The Art of Game Design. Burlington, MA: Morgan Kaufmann Publishers Shadish, W. R., Cook, T. D. & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton-Mifflin. Usher, E. L. & Pajares, F. (2009). Sources of self-efficacy in mathematics: A validation study. Contemporary Educational Psychology, 34 (1), 89-101. Tapia M. & Marsh, G. E. (2004) An Instrument to Measure Mathematics Attitudes. Academic Exchange Quarterly, 8(2), 16-21. Ericsson, K. A. & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Revised edition. Cambridge, MA: The MIT Press. O’Donnell, C. L. (2008). Defining, conceptualizing and measuring fidelity of implementation. Review of Educational Research, 78(1), 33-84. Wale, C. M. & Cooney, J. B. (2011). Reliability and validity of a 20-item scale to assess common core learning standards for 7th grade mathematics. Unpublished manuscript. Krathwohl, D. (2002). A revision of Bloom’s taxonomy: An overview. Theory Into Practice. 41(4), 212-218. Raudenbush, S.W. and Bryk, A.S. (2002). Hierarchical Linear Models: Applications and data analysis methods 2nd ed. Thousand Oaks: Sage Publications Inc. Schoenfeld, A. (1985). Mathematical problem solving. New York: Academic Press. Ries, A. & Trout, J. (1993). The 22 Immutable Laws of Marketing. New York: Harper Business Miller, G. (2012, Jan. 9). Conversation during SRI International Evaluation Site Visit, Taos, NM.