A Bad Question

This isn’t about math this time, it is about reading. Actually it is about a question on a reading test, in my opinion a very bad question. I will probably compare this experience to math later, but for now this is about reading and language. Recently, my son’s 3rd grade class took a reading assessment, something they do every week like clockwork in preparation for the big assessment, the FCAT, which is Florida’s version of the NCLB testing regime. Following is one of the questions from the test.

“For many years guest workers in the United States suffered bad working conditions.”  What does the word suffered mean in this sentence?“

• A. Ignored
• B.  Felt pain
• C. Lived through
• D. Were damaged

Needless to say, the class did poorly. Most students chose (B). The teacher emailed the parents to give them a heads up (before we saw the results later that evening) that the test was hard and the class did poorly. My first thought was that this was an exceedingly difficult question for 3rd graders. In fact, to make sure it wasn’t just me, I showed the question to several colleagues and the consensus was that this question was difficult, not only for 3rd graders, but even for us. To be fair, this question was from a story that they read, so there was more context than just this question, but the poor results of the students and the poor results of my colleagues indicate that there is something wrong with this question.

My first reaction to this question, and questions like it, is that they are ambiguous. This is not an uncommon reaction and I am sure that many who read this have the same reaction. And the test itself states in the beginning that you are to choose the best answer, not just the one that seems correct, which not only implies ambiguity, it implies intentional ambiguity. But what is the nature of this ambiguity? After studying it for a bit I realized that the question is not only ambiguous in its choices but in the nature of the question itself.

Words have no meaning by themselves. They only acquire a meaning when they are used in a sentence or dialog, and even then, it is the sentence or dialog that has the meaning, not the individual words. Even though we take the meaning of the whole sentence and divide it amongst the words and even though we create dictionaries listing possible uses of words, we only do this because it helps us organize all of these labels (words) into a vocabulary. The mechanism of natural language takes off when we string these words together into sentences and dialogs and create context. This is not a simple sequential process where we place one word after the other and then add up the individual meanings of the words to get the whole meaning. It works in the reverse, we experience the whole meaning of the sentence (somehow) and then, if asked, we divide that meaning out amongst the words. And this division does not have clear and distinct boundaries, in fact, dividing the meaning to the word level is usually not even possible, we generally divide it down to the phrase level. For example, in this question, the word “suffered” is not enough to assign a portion of the meaning to. We really need to be assigning it to “suffered bad conditions”. If I said “The workers suffered” that implies an event, but if I said “The workers suffered bad conditions” that implies on period of time. This subtle difference is paramount to what they author is trying to convey. You don’t know what the meaning is until you see the words that follow “suffered” and thus, they are part of the label for that meaning.

The first issue with this test is that it is intentionally ambiguous in what it is actually asking. This question could be taken three ways.

1. What is the meaning of the word “suffered”?
2. What is the meaning of the word “suffered” in the sentence?
3. If we were to replace the word “suffered” with the choices, which choice would retain the original meaning of the sentence the best?

We know that the question is not asking (1) because “suffered”, like any word, has many meanings. The question is literally asking (2) but that is still very much like asking (1) unless the student understands that (2) is actually asking (3). Unfortunately, this test emphasizes “word meaning” so much, even in the nature of of its answers, that the students (and adults) tend to answer (1) instead of (3).

This question would be better stated if it asked “If we replaced the word “suffered” with these choices, which choice would result in a sentence with the closest meaning to the original?” Or, instead of listing the meanings as they did, they could have listed four versions of the sentence and asked the student to choose the one that was the closest in meaning to the original. The fact that the question (and the test) puts so much focus on the meanings of individual words is counter what actually occurs in language. The meaning lies in phrases and sentences, not the words.

Getting past how meaning is formed in sentences is doable, although this test does everything it can to confuse that notion. But even after the student gets that sorted out, they are in an even more ambiguous position. If I were to rewrite the sentence in this question it would be something like…

“The workers lived with pain through bad working conditions.”

Or maybe something like…

“The workers lived with hardship through bad working conditions.”

I would not write…

“The workers lived through bad working conditions.”

Because that does not imply the effect on the workers. That is why the author used the word “suffered” instead of something like “enjoyed” or instead of simply saying “lived through”. So now the student is confronted with choices (B) “felt pain” and (C) “lived through”. I have looked at this question a dozen times, sometimes (B) seems correct and sometimes (C), in fact they together are the answer. I can not say with any certainty that upon my first exposure I would choose (C). In fact, even after all of this analysis, I cannot come up with a semantical reason that either of these two choices are the “best” choice. They are both equally implied in this sentence. But notice that I said I couldn’t come up with a “semantical reason”, and the question does literally ask what does the word “suffered” mean in the sentence. From a semantical point of view it means most nearly, (B) and (C), at the same time. But from a grammatical point of view, only one choice has the right grammatical form to replace “suffered” and that is definitely (C). The other choices sound silly (grammatically speaking) in place of “suffered”. And maybe this one of the keys to these poorly made questions. When in doubt, substitute the choices into the sentence and pick the one that doesn’t sound silly, grammatically speaking.

I guess a good part of this story is that now these students know what “suffered” means. They have suffered through this test. They didn’t just live through it, they suffered through it, with the pain and hardship that “suffered” implies.

Bob Hansen

Kahn Academy

Most are probably aware of Kahn Academy by now. I came across it a couple years ago in a discussion blog and it has received quite a bit of press lately. At its core it is (as it began) a large collection of short (10 minute) lessons on topics in math and now in many of the other subjects taught in school, from elementary school to high school. The founder and key person behind Kahn Academy is Salman Khan and he is rather prolific. He has created 1000’s of these video lessons thus far and made them available on YouTube as well as via the Kahn Academy site. But Kahn Academy has grown beyond its roots of just a large (now very large) collection of short tutorial videos. It also has a practice component where the student can practice the math displayed in these videos and like the vast collection of mini topics in the videos, it too is a vast collection of mini topics connected in a tree like hierarchy. And it also has a section that shows how well the student is doing as they progress through these mini topics.

So what is Kahn Academy and can it be effective?

First off, the videos are squarely tutorials, each aimed at a specific type of math problem or in many cases just a single skill. For example, there is a tutorial on adding single digit numbers, another one on adding multi digit numbers, and another on adding negative numbers. There is nothing new here in terms of the types of problems or skills covered. What is new is that someone (Salman Kahn) put in the considerable effort to make a video tutorial on each and every type. There are 1000’s of tutorials available. The upside is that the coverage of all the itty bitty bits of mathematical exercise is very complete. If you can think up a skill there is probably a video for that. The downside is that it is difficult to navigate and sequence but the Kahn Academy site does have an index arranged by phase and topic. And when we think about it, the topics in textbooks are usually not followed in the strict order they are printed and some are skipped entirely. So in the end I guess it is always the parent’s or teacher’s prerogative to decide which tutorials are important and how to sequence them.

But how effective are tutorials? Actually, I think when the student is ready they are effective, at least as effective as any other good instructional method, and in this case they are free. But they are not effective when they student is not ready. Unfortunately, and Kahn causes a lot of this in the way that he and his benefactors (Bill Gates et al) promote the site, people are led to believe that this is a viable alternative to live interactive instruction, actually an alternative to school itself, essentially, an online school. This is really not a good analogy. Kahn’s tutorials do very little to explain the steps or to explain the math and purpose that connects all of these videos together. Don’t get me wrong, it is obvious that Kahn himself knows all of that and the confidence he exudes while performing the math in these videos is a big plus and exactly what a tutorial should look like, for students that are ready. But the vast majority of students, actually I’ll just say all students, can’t pull all of the rest of that purpose and underlying connectivity out of thin air. Math is an art that has evolved and been perfected over thousands of years by a countless many great minds. To advance a student into all of that it takes a combination of direct instruction, exercise and guidance by a teacher who has themselves successfully made the same journey. This is why teaching exists.

The Kahn tutorials (I think we should call them tutorials rather than videos) are a useful part of that journey, but only a part. Tutorials cannot replace a live teacher that is able to watch a student and see what they are having difficulty (the student certainly won’t know why) and address those difficulties directly. Tutorials can’t be there when the students ask the big questions that we so eagerly anticipate and wait for. And tutorials certainly can’t answer those questions with an explanatory interactive dialog that only occurs between two persons.

I personally use the Kahn tutorials with my son when we have made it through the understanding phase and the initial doing phase. I use them when he is entering the gaining fluency phase (which continues for quite some time). I don’t use them a lot in the understanding phase because there usually are elements of practice of the math we already covered contained in the new math we are embarking on. We are continuously rehashing previous conquests. But when we get far enough along, I do find the Kahn tutorials to be great reaffirmations and a welcome change of pace that builds fluency and confidence in applying the math they know. And the learning process does continue during these videos when the student is ready and engaged.

There is also a collaborative aspect to the tutorials. I am not talking about the goofy version of collaboration that some educationalists try to fool people with, where one or two students do all the cognitive work and learning for a whole group of students. I am talking about the real version of collaboration that occurs in the real world where you are expected to cognitively follow the work and explanations offered by your peers. Kahn’s tutorials are in this fashion collaboration, but as I have said before, the student must be ready.

One thing is for certain. When your child can follow these tutorials and the subtle choices that Kahn makes and talks to while he is doing the math, give yourself a pat on the back. Your child has arrived in math land. Where they will explore next in that magical land is up to you and them, but they have arrived. And that is a very precious achievment.

Bob Hansen

Devlin on Spreadsheets

Keith Devlin recently posted “What is Algebra” which in part talks to the relationship between algebra and spreadsheets. I too think spreadsheet math is under represented in public schools but not for the same reasons as Mr. Devlin suggests. His misinterpretation of the reality of spreadsheets and how they are used underlies the often cited misinterpretation that technology aids the acquirement of mathematical ability, be it arithmetic or higher.

I have said many times before that spreadsheets are the most used math in the world today and if all a computer could do was spreadsheets, there would be one beside every typewriter. If our task is words we use Word and if our task is numbers we use Excel. But when we use spreadsheets do we use algebra? Or more specifically, does the typical spreadsheet user use algebra?

First off, the typical user uses a spreadsheet to layout arithmetic operations on numbers in rows and columns. I would characterize this “spreadsheet math” as pseudo algebra in that the user is certainly choosing the order and layout of the calculations. They might take one value that is a product of two terms and show the two terms individually as well as the product. So they are factoring in a way, albeit a more arithmetical form of factoring. And this is all that the vast majority of spreadsheet math is. Reordering, refactoring and reformatting sequential series of arithmetic operations across and down. The user does does all of this in order to tell a story, a number story.

Is there algebra in all that? I think there is certainly some basic algebraic reasoning, and some users are more ingenious than others, but the chief source and inspiration of all this is arithmetic and sense of number. Indeed, when a user comes to me and tells me that their spreadsheet does not work, it isn’t because they recognize an error in their pseudo algebraic manipulation of the rows and columns. They come to me because the totals don’t add up. And that is what spreadsheets are good at. Organizing arithmetic in useful ways that balance out and tell a number story.

Devlin states…

“The important thing to realize is that doing algebra is a way of thinking and that it is a way of thinking that is different from arithmetical thinking. Those formulas and equations, involving all those x’s and y’s, are merely a way to represent that thinking on paper. They no more are algebra than a page of musical notation is music. It is possible to do algebra without symbols, just as you can play and instrument without being ably to read music.”

Before I go after the error in this paragraph, let me first state that he is correct in saying that algebra is a way of thinking. It isn’t just symbols like good writing isn’t just words. However, and this is a big however, once you arrive at that level of thinking the symbols should make perfect sense and are a natural extension that allows you to make use of and extend this reasoning. His example of being able to play music without being able to read music is an exaggeration at best. It is probably based on the misconception that there are two types of musicians in the world, those that can read music and those that cannot. The reality is that virtually all musicians can read music, even those that play by ear, for the same reason that all writers can read words. Where musicians differ is in how well they can “sight read” music, not simply read music, which means play the music directly from a score without practice. To put this in perspective, the majority of professional musicians (those who make music you pay for) can do this in one take. It is a requirement of the industry because, well because there are enough musicians that can do it and that sets the mark.

It is unreasonable to think that these abilities, algebra, music and writing to name just a few, are hindered by notation. The notation exists and was born of these abilities. The notation evolved and was perfected with these abilities. So how can this notation that was born of the ability be a hinderance to students? It can’t. Devlin already identified the hinderance. It is not the notation, it is the thinking. It isn’t that the student can’t get past the symbols because (as Devlin said as well) the symbols are not even the algebra. The student can’t get past, or even started is a better way to put it, the thinking!

Devlin goes on to say that algebra is important, more important than even arithmetic. Empirically speaking alone, I don’t see any basis for this statement. At most, according to test results, 15% of the population even makes a dent in elementary algebra, and except for the engineers, teachers and math lovers that use it after they’ve graduated, many of those 15% simply forget it. You can take a quick survey at work to know this is true, even if you work at a University. There really is very little algebra outside of school. I’ve always been curious why reformists never picked up on this fact. I suppose they might be out of a job if people all of the sudden came to the realization that the vast majority of people don’t use algebra. Not for recipes, not for balancing check books and not even for spreadsheets. In fact, they don’t even remember it! And for those few widely spaced instances when you have an actual algebraic question (which is almost always a weighted average) you go to the person in your department that still knows algebra.

I am a proponent of more spreadsheet math, but the type of spreadsheet math that people actually do in the world that is rooted in arithmetic and sense of number, not algebra. The number of users that even know what a macro is is very small and the number of people that use spreadsheets to “solve” problems is smaller yet. When you get into “solving” problems, even if your layout/computing tool is a spreadsheet, you do need more formal algebraic reasoning. But obviously, the vast majority of the world gets by without it and can meet their practical day to day math needs with enhanced (pre algebra) arithmetic and a tool like a spreadsheet.

In conclusion…

1. People use spreadsheets a lot.

2. They don’t use them algebraically for the same reason they don’t do well in algebra classes, because they don’t think that way.

3. The vast majority of math being done in the real world is arithmetic not algebraic and certainly not analytic. It is practical and especially suited for spreadsheets.

4. A majority of people have problems with algebra and they have these problems not because they don’t understand the symbols but because they don’t understand algebra or are simply not interested in understanding algebra.

And finally…

Even though Devlin states matter of fact that most people have problems with algebra and that algebra is a way of thinking, he falls (actually he jumps) right back into the trap of mandating that everyone to do it. We both seem to agree that spreadsheet math is important but I do not call it “algebra”, at least not the spreadsheet math that the vast majority of people actually use spreadsheets for. The reality here is that since the majority of math being done outside of school is spreadsheet math (not algebra) then maybe there should be tracks other than algebra in school, like say, business math, that teaches a variety of techniques using spreadsheets that even when they have algebraic underpinnings, they just work without having to know of or understand those underpinnings.

I think that if we attempt to teach algebra using spreadsheets we would (and do) run into the same wall we run into when we teach algebra itself. We would have to teach the students to think algebraically. As Devlin states, algebra is not symbols, and neither is it spreadsheets, it is a way of thinking. Once you catch on to that way of thinking and only if you catch on to that way of thinking then you will see the algebra in symbols, in spreadsheets, in arithmetic and in just about everything else having to do with number. Fortunately, for the rest of the world, spreadsheets have enormous usage potential without the need of algebra. Indeed, that is the source of their popularity.

Bob Hansen