Skip to content

Commit

Permalink
COMP: removed excesss generic questions
Browse files Browse the repository at this point in the history
  • Loading branch information
@kevindeland
kevindeland committed Oct 18, 2018
1 parent e0e89c1 commit 5937c1f
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 15 deletions.
14 changes: 1 addition & 13 deletions comp_questions/src/main/java/cmu/xprize/comp_questions/CQn_ViewManagerASB.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2415,20 +2415,8 @@ public void setRandomGenericQuestion(){
File[] allfiles = dir.listFiles();*/

String[] genericQuestions =
{ "Has this happened to you",
"What part of the story are you reading now",
"What could you learn from this",
"When does this take place",
"What do you think this story will be about",
"Where does this story happen",
"What do you think will happen next",
"Where does this take place",
"What does this story teach you",
"Who does this talk about",
"What happens in this story",
"Who is this story about",
{"Who is this story about",
"What has happened so far",
"Whom did you like best in this story",
"What is this story about"};

Log.wtf("COMPREHEND", "files: " + genericQuestions.length);
Expand Down
4 changes: 2 additions & 2 deletions comp_reading/src/main/java/cmu/xprize/rt_component/CRt_ViewManagerASB.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1474,7 +1474,7 @@ private PointF broadcastActiveTextPos(TextView text, String[] words){

/**
* This is where we process words being narrated
*
* VMC_QA why does this get called twice for the last word???
*/
@Override
public void onUpdate(String[] heardWords) {
Expand All @@ -1489,7 +1489,7 @@ public void onUpdate(String[] heardWords) {

while (mHeardWord < heardWords.length) {

if (wordsToSpeak[mCurrWord].equals(heardWords[mHeardWord])) {
if (wordsToSpeak[mCurrWord].equals(heardWords[mHeardWord])) { // VMC_QA these are not equal. one of these is out of bounds (probably wordsToSpeak)

nextWord();
mHeardWord++;
Expand Down

23 comments on commit 5937c1f

@JackMostow
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@judithodili - These questions are stems from a published intervention with experimental evidence that they scaffolded comprehension when presented in a multiple choice format with similarly generic answers.

  1. Why were most of them removed?
  2. Why this particular subset? I recall mention that some generic questions confused kids, but asking one of only the same 3 questions after EVERY page is inhumane. We agreed to reduce the probability to 20%.

@judithodili
Copy link
Collaborator

@judithodili judithodili commented on 5937c1f Oct 21, 2018 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JackMostow
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@judithodili - I agree that the generic questions are currently way too frequent. That's why we recently agreed to reduce their frequency to about 1 page in 5 (a compromise between 3 and 10) by inserting them with probability 20%. This solution is easiest implementationally (1 line of code) because it doesn't require any additional state. Unfortunately the reduction didn't make it into 2.7.0.1 -- so it inhumanely asks the same 3 questions over and over.

In the ITS2004 study I mentioned, Project LISTEN’s Reading Tutor randomly inserted "three kinds of multiple-choice questions in children’s assisted reading.... To compare their effects on story-specific comprehension, we analyzed 15,196 subsequent cloze test responses by 404 children in grades 1-4. Wh- questions significantly raised children’s subsequent cloze performance...."

The stems of the Wh- questions analyzed in the study were the "what, when, and where" questions in the list above. The multiple choices were similarly generic. They addressed the possible incompleteness of the choices or irrelevance of the question by including a "none of the above" option like "I don't know" or "This is a silly question!", as in "When does this take place? in the present; in the future; in the past; It could happen in the past; I can’t tell."

Dropping the multiple choices and instead using just the stem plus "Tell RoboTutor" ("Semea RoboTutor") to elicit a spoken response makes the questions much shorter and simpler, and eliminates the issue of incompleteness albeit not of irrelevance.

Please restore all of the original questions to cover more of the kinds that kids may encounter in EGRA, and address the irrelevance issue by appending the already narrated prompt "Does this make sense?" ("Je, hii inaeleweka hapa?"). This addition should be beneficial in itself by helping kids by monitoring their own comprehension -- a crucial metacognitive process in reading.

Thanks. - Jack

@judithodili
Copy link
Collaborator

@judithodili judithodili commented on 5937c1f Oct 21, 2018 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amyogan
Copy link
Collaborator

@amyogan amyogan commented on 5937c1f Oct 21, 2018 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JackMostow
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@judithodili and @kevindeland - OK, let's:

  1. Most importantly, reduce insertion probability to 20% (1 page in 5) as planned.
  2. Leave out these questions that make (more) sense only:
    a. At the start of the story:
    "What do you think this story will be about",
    "What part of the story are you reading now", [requires knowing what parts a story has]
    b. At the end of the story:
    "Has this happened to you",
    "What could you learn from this",
    "What does this story teach you",
    "What happens in this story",
    "Whom did you like best in this story",
  3. For variety, add back in these 5 generic questions, for a total pool of 8 to pick randomly from each time:
    "When does this take place",
    "Who does this talk about",
    "Where does this story happen",
    "What do you think will happen next",
    "Where does this take place",
  4. Don't add "Does this make sense?"; it adds metacognitive complexity and makes unclear which question to answer. Instead, just assume that the selected questions make sense often enough to work ok.
  5. Add "Tell RoboTutor." to elicit a spoken response.
    Thanks. - Jack

@JackMostow
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@judithodili and @kevindeland - OK, let's:

  1. Reduce insertion probability to 20% as agreed.
  2. Keep these 7 generic questions, because they typically make sense anywhere in a story and exercise the information retrieval, summarization, and inference skills used in reading and tested by EGRA questions:
    "What is this story about",
    "Who is this story about",
    "Who does this talk about",
    "Where does this story happen",
    "Where does this take place",
    "What has happened so far",
    "What do you think will happen next",
    3. Omit the other questions, because they typically make sense only at the start or end of a story.
    4. After asking a generic question, add "Tell RoboTutor" to elicit a spoken response.
    Thanks. - Jack

@JackMostow
Copy link
Contributor

@JackMostow JackMostow commented on 5937c1f Oct 22, 2018 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@judithodili
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Who is this story about",: What if this story is not about a person? E.g Animals? How is that different from "what is this story about?"

"Where does this story happen": What if the place is not listed (happens quite often with these stories)? Do you mean the city? House? Farm? I dont know what a possible answer to this question is. Same issue as "Where does this take place",

"When does this take place": Do you mean what time in the day? What season e.g fall spring? What year? Again.. I can't think of a possible answer to this question.

"What do you think will happen next": What if this gets randomly asked at the end of the story?

@JackMostow
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Who questions are somewhat genre-specific (narrative fiction, biographies, news stories, ...) but all the stories I've seen are fiction. Animals are characters. They talk, so "who" seems fair.

"Where" and "when" can mean whatever the kid thinks. The point of all these questions is to elicit active processing.

"What will happen next" is fine at the end of a story -- it asks about the characters' future. Teachers love it.

@judithodili
Copy link
Collaborator

@judithodili judithodili commented on 5937c1f Oct 22, 2018 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JackMostow
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The questions have answers in the context of a story.

@amyogan
Copy link
Collaborator

@amyogan amyogan commented on 5937c1f Oct 22, 2018 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JackMostow
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Costs and benefits:
Worst case cost: kid does nothing for a few seconds or bails out of the story.
Benefits: From the metacognitive perspective, even just thinking about the question is beneficial.

It would be nice to reassure kids who don't know how to answer, but how?
"Does this make sense?" might help, but adds to complexity and confusion.
Adding "What do you think?" might help, as in "What do you think will happen next?".

Ideas for what to say -- preferably already narrated?

@amyogan
Copy link
Collaborator

@amyogan amyogan commented on 5937c1f Oct 22, 2018 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JackMostow
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, let’s do this:

  1. Keep these 8 generic questions for variety, because they typically make sense anywhere in a story:
    • "Who is this story about",
    • "Who does this talk about",
    • "Where does this story happen",
    • "Where does this take place",
    • "When does this take place",
    • "What is this story about",
    • "What has happened so far",
    • "What do you think will happen next",
  2. Omit the other questions because they typically make sense only at the start or end of a story.
  3. After each page, insert a generic question with probability only 12%. 2.7.2.1 has 25%, which feels too high.
    a. Make the probability a configuration variable to make it easy to change.
  4. Don’t ask “Does this make sense?” because it would add metacognitive complexity and make unclear which question to answer.
  5. Prompt “Tell RoboTutor” after the question to elicit a spoken response without using the first person.
  6. See how this revised version works out in practice. If it flops, we can change the probability to zero in config.json without changing the apk.
    Thanks. – Jack

@judithodili
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just so we are clear on what the problem is... here are a few examples of the stories that are currently in Robotutor.... please use them as a reference and try to answer the generic questions (because the kids have to answer them anyway) ... Perhaps that will help to select questions that are more appropriate.

These 2 are the exact same question - can you pick one:

• "Who is this story about",
• "Who does this talk about",

Colours_of_a_rainbow.pdf
Colours.pdf
A_very_tall_man (1).pdf
Amaro_Kisoma (1).pdf

@amyogan
Copy link
Collaborator

@amyogan amyogan commented on 5937c1f Oct 23, 2018 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@judithodili
Copy link
Collaborator

@judithodili judithodili commented on 5937c1f Oct 23, 2018 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JackMostow
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two questions are not identical:
• "Who is this story about",
• "Who does this talk about",

The answer to the first one is the main character(s) of the story.
The answer to the second one is whichever character(s) the page mentions.

@JackMostow
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No files yet.
The answers to a question at the end of a page may (and should at least sometimes) vary by page, and they may depend on the page illustration. But since you asked, here are some possible answers based on story.txt:

• "Who is this story about",
A very tall man

• "Who does this talk about", [the same because this is a single-character story]
The very tall man

• "Where does this story happen",
• "Where does this take place",
Inside, outside, at the man's home, ...
• "When does this take place",
In the past

• "What is this story about",
Problems of being too tall

• "What has happened so far",
the man's hoe/doorway/bed/bike was too small; the man made...,; the man sat on...

• "What do you think will happen next",
During the story: the man will have trouble with ... because he's too tall
At the end: the man will be lonely and make friends with the animals.

Relating specific text to general who-what-where-why-when questions is an important part of listening and reading comprehension. Its NLP analogue is "information extraction."

@JackMostow
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Colors story (which BTW works great with the pic questions!):
• "Who is this story about",
the girl in the pictures

• "Who does this talk about",
the girl in the pictures

• "Where does this story happen",
at home

• "Where does this take place",
inside/outside

• "When does this take place",
anytime / now

• "What is this story about",
colors

• "What has happened so far",
saw some colors / particular objects

• "What do you think will happen next",
see another color

@JackMostow
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A higher priority now should be to find and fix bugs in activities and data sources, such as missing introductory prompts for Parrot activities.

Please sign in to comment.