Sylvana Novilia Sumarto/ F112807
The Analysis on Methodological Aspects of Roth & Bowen (2003): When Are Graphs Worth Ten thousand Words? An Expert-Expert Study
Introduction
This analysis will discuss about the methodological aspects which were used in the study conducted by Roth and Bowen (2003) titled “When Are Graphs Worth Ten Thousand Words? An Expert-Expert Study”. The study analyzed how scientists interpreted the familiar and unfamiliar graphs. The analyses showed that the scientists read the familiar graph transparently and they linked graphs to possible worlds by means of complex inferences when read the unfamiliar graphs. In this study, the researchers also proposed the two stage model of graph interpretation.
Purpose and Research Question
The purpose of this study is stated explicitly in this article, “This study was conducted to better understand graphing expertise” (p. 430). It is mentioned more specifically on the other part of the article, “This study was designed to provide a better understanding of how scientists interpret unfamiliar graphs” (p. 442) and “This study was designed to better understand reading of familiar and unfamiliar graphs by professional scientist” (p. 466). Moreover, the researcher also mentioned that they particularly interested in the contributions of scientists’ experience when they read the graphs (p. 430). Here, the experience will influence the level of scientists’ familiarity to the graphs which we knew it is relative for each person. For instance, the ecologist will be more familiar to the three graphs on task design than the physicists or forest engineer because the three graphs are standard in the ecology course. The university-based scientists were more familiar with doing the interpretive task than the nonuniversity public sector scientists because they teach their students to do the similar thing (p. 442).
The research questions are not explicitly stated on the article. I think the research questions should be more than an initial question as proposed in the conclusion part, which is also used as the title of this article, “When are graphs worth 10.000 words?”. In my opinion, the research question will always related with the result of the study. Generally, we will analyze the data to answer our research question. So, I can say that the research questions are the questions from the result of empirical data, which are “(a) that scientists cannot be taken as graphing experts in general and (b) that existing models of graphing expertise do not explain scientists’ readings of unfamiliar graphs” (p. 430). From the first result, I think the possible question is to what extent the scientists can understand and interpret the graphs. The second one is how the existing models of graphing expertise fit the scientists’ interpretation of the graphs. However, the first question is the main question research on this study because it is directly related with the purpose of this study.
Research Approach and Methods
The research approach that used in this study is ethnography (p. 438). There are some characteristics of ethnography that can be seen on this article. First, there is special attention given to the way the people being studied see their world (Denscombe, 2010, p. 80). In this case, the researchers want to find out how the scientists interpret the graphs. Second, the ethnographers are required to spend considerable time in the field among the people are being studied. They need to share in the lives rather than to observe from a position of detachment (Denscombe, 2010, p. 80). In this case, because of the researchers and the participants are all scientists so there is no possibility that the researcher will observe from a position of detachment.
However, the statement on page 438, “As a result of this study, we developed a model that …” indicates that the researchers might be not only use the ethnography as their approach but also used the other approach to facilitate the development of the new model. There are some aspects that I proposed to support my argumentation that the second approach that used in this study is grounded theory. First, the data analysis is not only descriptive but also interpretative and categorized, which is one of the characteristic of grounded theory. Second, the starting point of this research based on empirical data (data collection) to propose the new model named 2-stage model of graph interpretation (p. 429). Third, in this study, the existing models of graphing interpretation from the previous studies are treated as provisional. “They are not fixed, they are not necessarily right. They are simply a tentative starting point from which to launch the investigation.” (Denscombe, 2010, p. 111). Fourth, the sample choosing is appropriate with the key features of grounded theory (Denscombe, 2010, p. 112-113). The last, unstructured interview that is used in this study is also the appropriate method for grounded theory.
Moreover, I think the grounded theory may be the appropriate approach for this study. As stated on Denscombe (2010, p.6), grounded theory is good approach for the research which clarifies concepts or produce new theories and also to explore a new topic or provide new insights. In this regard, the grounded theory is used to develop the two stage model of graph interpretation.
The method which is used in this study is unstructured interview. Related to the goal of this research that is to understand how scientists interpret the graphs, I think the interview is the most appropriate methods for this study. It is different with using open questionnaire; the researchers do not only knew the participants’ thinking from their explanation but also can see their gestures when they were giving their explanation.
Population and Sample
The population of this study is scientists. The researchers took 16 scientists (8 from university and 8 from public sectors) as sample of this study. There were 15 men and 1 woman in this sample. It was not balance in numbers but I think the gender is not the issue for the representativeness of the sample in this study. The sample is balance for the number of participants who work in university and public sector, but I believe that this is also not the issue for the representativeness of the sample. The number of participant is also not the issue for the representativeness of the sample because there is no statistical representative issue in this study. However, the sample is not representative enough for scientists because 13 of 16 participants are ecologists. The three others are 2 physicists and 1 forest engineer.
Related to the particular interest on the contribution of scientists’ experience, I think the sample choosing is appropriate because all of them had experience in research even received the national and international awards for their publications which indicates that all of them had experience with graphing (p. 433).
Data Collection
The only one method that was used in this study is unstructured interviews. From the interviews, the researcher had the transcripts, videotapes, and artifacts. The researcher did not mention about data triangulation in this study. It means there is no guarantee for the validity of the data collection.
In this article, the researchers explained the task design and the procedure of interview clearly in the “Task Design” and “Procedure” part respectively (p. 436-437). This made the data collection process more transparent.
The three graphs that used in instrument of the interviews are valid because the three types of the graphs are standard in introductory ecology courses and appear with considerable frequency in introductory ecology textbooks, which can be said have credibility (p. 434). It is also same with the participants’ own graph because they chose the graph from their research articles or reports or printouts of individual graphs they had published (p. 436). But, I wonder about the reason why the researchers choose the graphs that related with ecology for task design and not about the other science, such as from physics or biology.
Data Analysis and the Researcher’s Interpretation
The researchers give a clear explanation about how they analyzed the data, related to their concern on features, contextual constituents, and referent/ content domains. They also presented the levels of performance on the table and categorized incorrect interpretation in three section, namely “Difference in Contextual Constituents and Salient Features, Differences in Referents, and Dialectic and Iteration” (p. 442) . This analysis also was supported by proposing some parts of transcript. This description increases the transparency of data analysis.
The researcher also mentioned about the coding of the transcript in the same way (e.g. tracking across) and also the long term collaboration between them. These two aspects increase the interrater reliability. However, I do not see the explanation about the 3 extensive analysis, how they choose the three interviews and also the reason in choosing.
Conclusion
The conclusion describe about how the scientists interpret the familiar and unfamiliar graphs clearly. The researchers gave explanation about the steps that scientists did when they faced the unfamiliar graphs and also about the transparent reading which also was explained clearly on part “Familiar Graph: Transparent Reading”. Unfortunately, the researcher did not mentioned clearly about their new model, two stage model of graph interpretation, which can be stated as the answer of the second research question (how the existing models of graphing expertise fit the scientists’ interpretation of the graphs).
They also answered their initial question, “When are graphs worth ten thousand words?”. At the previous section, the amount of the words were just mentioned on part “Procedure” that “The total word production across the four tasks ranged from a low of 5.400 words to a high of 10.000 words” (p. 437) but in the conclusion the researcher mentioned that when the scientists interpreted unfamiliar graphs, they just spent 1.500-3.000 words (p. 470). I think it should be explained at the previous section too.
Consistency
In general, the purpose of the research that is to analyze how the scientists interpret the graphs fit with the ethnography as the approach and the unstructured interviews as the method. The approach that used in this study also had an appropriate theory background namely the literature on ethnomathematics (p. 431). However, there are some aspects which are not clear enough in this article. First, I would like to discuss about the purpose and research question of this study. The purpose of this study is to provide better understanding about how scientists interpret the familiar and unfamiliar graphs. But, later it is also mentioned about the existing models of graphing expertise which do not explain scientists’ readings of unfamiliar graphs and a new model that developed as the result of this study. I think it’s kind of giving a new insight on the previous theory. So, that’s why I think there is the possibility of using the second approach in this study namely grounded theory in order to develop the two stage model of graph interpretation.
Second, the conclusion gave the clear explanation about how the scientists interpret the familiar and unfamiliar graphs. Nevertheless, I wonder about the two stage model that was not mentioned in this part whereas it also appeared in the discussion part. I think this should be included in the conclusion. In this part, the researchers also answered the initial question of this study. But, I wonder to what extent the initial question is made. In my opinion, the most important is about the sense of the interpretation and not about the amount of the words.
References
Denscombe, M. (2010). The Good Research Guide (4th edition): For small-scale social research projects. London: Open University Press.
Roth & McBowen. 2003. When Are Graphs Worth Ten Thousand Words? An Expert-Expert Study. Cognition and Instruction, 21 (4), 429-473.
Sylvana N S 