Crowdworker evaluation of the tasks [Work in Progress]

Addressing #206

Here is how the template looks like now:

When filled in with the content, here is how it would look like:

Here is a pilot on task 156. The reviews for the most part indicate positive feedback about the task.
https://docs.google.com/spreadsheets/d/1wXStPurP6AamxvglOw0aOJ7V1DiSKczJNyFcyfDnk4w/edit?usp=sharing
Two changes are needed for the next experiment:

increase no. of positive examples (3 -> 5)
emphasize that they don't need to write an "explanation"; only the "output"

@danyaljj Solid work! This is better than what I could have done for sure. Some advice after a quick glance:

For all of the questions, instead of "How did you assess the quality...?", I would change them to "What did you think about the quality...?".
Can you also explain what positive examples are too? The layperson may initially get confused and think that these are examples with positive sentiment as opposed to what they actually are (examples that are correct) and may not realize this until later into the assessment or even at all.
For the layperson, it may also be best to elaborate a bit more on negative examples being undesirable outputs.
This recent paper may also help with crowdsourcing and how we report our AMT results in the paper: https://arxiv.org/abs/2109.06835

@eshaanpathak these are good suggestions and I will incorporate them for the forthcoming experiments!

Maybe ask them to rate it on a scale of 1 to 5 or 1 to 10?

Quantitative labels (like "1 to 5") can be interpreted differently and are more subjective, I think (do I interpret 2/5 the same as you?) In my experience, categorical labels (with relatively clear definitions) tend to result in more calibrated outcomes.

Yeah, that's what I was thinking too, but I wasn't sure. I think it was the word choice for me. Instead of "pretty good", I would instead just have "good".

@Palipoor  Here is a batch of human judgments collected for tasks60-100 (see the 2nd tab):
https://docs.google.com/spreadsheets/d/1wXStPurP6AamxvglOw0aOJ7V1DiSKczJNyFcyfDnk4w/edit?usp=sharing
I went through the replies and here is what I extracted:

[ ] task066_timetravel_binary_consistency_classification.json

Feedback regarding positive examples: surely people don't panic and screen just because two men yell?

[ ] task067_abductivenli_answer_generation.json

Feedback re. definition: It would be good if, when you say, Use names, to list an actual name in the same way you list pronouns, like say, use names like Barbara, etc.

[ ] task069_abductivenli_classification.json:

is it clear if the outputs should be 1/2? Some workers wrote string responses.

[ ] task072_abductivenli_answer_generation.json

Feedback regarding negative examples: Not sure what was wrong with the "her" in that sentence as the example also included her actual name. "Anna went to Anna's bedroom" would surely be awkward?

[ ] task076_splash_correcting_sql_mistake.json:

Feedback regarding the definition: The instructions are clear, but the variable names are very hard to track.  Perhaps that is intentional though.
Feedback regarding positive examples: Having more positive examples would be better, but overall, they illustrated the point.  The one with swapping average for a value of average was a big confusing, since it would confuse people with a function vs. a variable.
Feedback regarding negative examples: "== 'null' is syntactically incorrect, at least in SQL variants I know.  It may be good to clarify even more things are wrong, using null as a string, double equal instead of equal.  Stressing NULL is a special value."
I noticed the word satisfying and satisfy misspelled and English should be capitalized.

[ ] task077_splash_explanation_to_sql.json

You can more explain step by step also you can add more realistic example

[ ] task078_splash_sql_to_explanation.json

In response to the following instance: "SELECT T1.Main_Industry FROM company AS T1 WHERE T1.Company_ID NOT IN ( SELECT T2.Company_ID FROM station_company AS T2 )" someone wrote "i don't have any idea." Anything we can improve here?
I noticed the words satisfy and satisfying were misspelled in one space. I think giving examples of how all of these things work within the main text would have helped with the clarity.

[ ] task079_conala_concat_strings.json

Maybe a few more examples? These look great, but I feel like once the task starts, there are probably more confusing ones to complete.
a few more examples would be helpful, such as, what if yummycookies is yummy_cookies?

[ ] task081_piqa_wrong_answer_generation.json:

I think I understand this, but more examples would be nice.
It is mostly clear but only after I looked at examples then read the instructions again.
Feedback regarding negative examples: I think one more example of clearly not related at all would be good for anyone questioning it

[ ] task084_babi_t1_single_supporting_fact_identify_relevant_fact.json

Feedback regarding negative examples: Only one example but it is clear why it's wrong

[ ] task085_unnatural_addsub_arithmetic.json:

Feedback regarding negative examples: There wasn't a lot of them, but the one given was explained perfectly.

[ ] task086_translated_symbol_arithmetic.json:

Feedback regarding the instructions: I think the instructions are clear enough.There is a missing word in this sentence that could help with clarity. I have indicated with brackets:" Here, 'sottrazione' represents [the] subtraction operation. So, the answer is 3 (10-7=3)."
Feedback regarding positive examples: I think these are clear, but for higher clarity, maybe replace input and output with question/answer? It really depends on the context of what's being tested, but the latter is more recognizable for a larger range of people I think.
Feedback regarding negative examples: I think for the negative example an explanation of how someone might have arrived at the wrong answer would be helpful. This allows people to see common mistakes to watch out for.

[ ] task087_new_operator_addsub_arithmetic.json

Feedback regarding negative examples: Could be more examples
Feedback regarding negative examples: The one given didn't have the minus sign replaced. (Daniel: this was the example: "15 @ 20 - 3")

[ ] task088_identify_typo_verification.json:

Feedback: Many of the examples are blank. Examples help me to make sure I've understood the instructions properly, so I would want more.

[ ] task089_swap_words_verification.json:

Feedback regarding positive examples: The following sentence is used as an example: "Seattle has nice a weather" But the proposed solution (swapping "a" and "nice") is grammatically incorrect and the output is incorrect as well (it gives it as 2,3).Also, examples 2 through 4 (positive) are blank.
Feedback regarding negative examples: One or two more negative examples would be a better reference.

[ ] task090_equation_learner_algebra.json:

Feedback:  The word "weight" was used in a non-standard (to a non math person at least) without any text explanation as to why that word was used. I figured out how the process worked from the examples, but the unfamiliar terminology made the text part of the explanation unhelpful. More words orverall would have helped, and a step by step explanation of the example would have helped as well (e.g. "the rightmost value is a constant and unaffected by the variable, the next value is multiplied by the variable, etc.)
Feedback for positive examples: A couple of more examples would have been useful, specifically of the ones with 3 weight values.
Feedback for negative examples: It was pretty good as is, but another example wouldn't hurt.

[ ] task094_conala_calculate_mean.json:

Feedback: "There are 2 examples of the correct way to do it, but there are still 3 blank areas left, leading to confusion and impreciseness."
Emphasize that we want only the final answer? Someone output "271.633/ 5 = 54.327" instead of "54.327"

[ ] task095_conala_max_absolute_value.json

Feedback: "I think it would be better to explain what an absolute value is/how you get it, with examples, before explaining the task. Otherwise just reading the task explanation itself is a little difficult."

[ ] task096_conala_list_index_subtraction.json

feedback: "You might want to consider clarifying what an index is for people who may not be that familiar with the term."
some people used parenthesis instead of brackets: "{11, 11, 3, 7, -10, -1, -27, -23, -15}" instead of "[11, 11, 3, 7, -10, -1, -27, -23, -15]"

[ ] task097_conala_remove_duplicates.json:

is it clear that there should be brackets in the output. Someone wrote "6,0,3,1,7" instead of "[6, 0, 3, 1, 7]"

[ ] task098_conala_list_intersection.json:

feedback re. description: "I understand it but some may not get it until looking at examples."
Is it clear that (1) the output should have brackets? (2) how the output should be ordered? Some people wrote "9,5,2,4" instead of "[2, 4, 5, 9]"

Here is feedback on task1-60. @swarooprm these are all yous! :)

[ ]  task003_mctaco_question_generation_event_duration.json

Definition: I think that the instructions could provide more information about the type of events that are being discussed (in addition to talking about brushing teeth).
positive examples: I think that the inputs and outputs were good, but it would be useful to see more of them. It may also be useful to see outputs that use different wording than "how long."

[ ] task006_mctaco_question_generation_transient_stationary.json

Definition: These are well written but I definitely needed the examples to be sure of what you are looking for
Positive examples: Maybe have a few more complicated examples.

[ ] task006_mctaco_question_generation_transient_stationary.json

I had a hard time understanding the "stationary v. transient" vocab.

[ ] task007_mctaco_answer_generation_transient_stationary.json:

It might help to have one more negative example

[ ] task010_mctaco_answer_generation_event_ordering.json

Defintion: I think maybe it could be worded simpler such as, What happens next?

[ ] task011_mctaco_wrong_answer_generation_event_ordering.json

You should give more example.

[ ] task012_mctaco_question_generation_absolute_timepoint.json

They are good, but I would like clarity on whether the questions can require specialized knowledge. For example, referring to the NBA basketball game, and knowing what time they usually play NBA games at.
I didn't know how something could be an absolute timepoint if you were just making time estimates.

[ ] task013_mctaco_answer_generation_absolute_timepoint.json

It would be helpful to have even more examples so I could have a better understanding of the task. Examples 2 to 4 are blank.

[ ] task014_mctaco_wrong_answer_generation_absolute_timepoint.json

Not enough examples

[ ] task018_mctaco_temporal_reasoning_presence.json

Negative examples: Pretty sure No is the correct answer there?  (Daniel: the example was this: "Sentence: Jack played basketball after school, after which he was very tired. Question: Who played basketball after school?")

[ ] task024_cosmosqa_answer_generation.json

Definition: It's confusing whether you want a response to the speaker or you want a summary from the speaker.
Examples: Example #1 made no sense to me and seemed to have grammar errors at the end. (Daniel: here is the example: "Context: you see , at my age relationship is kind of important and i thought i got the one after all these years . I noticed that once again i was wrong . i was good simply because i was good , i was caring , helping , supportive , bla bla blaaa . Question: What may happen to me?")

[ ] task029_winogrande_full_object.json;

Definition: I was confused by the massive list of contrastive words.
P examples: It was tough to parse out the differences in the examples.
N examples: It was hard to see what made these different from the positive examples.

[ ] task030_winogrande_full_person.json

Def: The instructions felt very wordy.
N examples: Some of these examples I thought could be positive ones.

[ ] task031_winogrande_question_generation_object.json

I didn't understand the contrapositives.
Examples felt totally random based on the inputs.
I didn't see how these differed from the positive examples.

[ ] task034_winogrande_question_modification_object.json

Definition: I had no idea what you meant by "contrastive" words.
N examples: i don't have any negative
N examples: It's confusing there needs to be at least 70% overlapping words when we don't have a word counter.
P examples: I was confused which words were being flipped.

[ ] task035_winogrande_question_modification_person.json:

Def: The formatting was hard to read; numbered lists typically have line breaks between list items.There were also grammatical errors ("should be an well-agreed answer", "atleast").The clarity could have been improved in some areas. For instance, "void repeating the same style or phrase in generating your modified question e.g. this task can be always solved using a simple negation i.e. by adding not, never, etc. Instead, try to increase the word diversity." is a confusing sentence, particularly because of the use of both "i.e." and "e.g."Also, the using real names examples are kind of unnecessary ("Donald Trump or Putin").
P examples: The grammar is weird at times ("replacing the trigger word 'sympathetic' by 'stern' "). I would think you might use the word "with" instead of "by."
N examples: The first example didn't really make sense because of (presumably) a typo: "PersonY was always ahead of PersonY, as _ walked with a quick step ."

[ ] task036_qasc_topic_word_to_generate_related_fact.json

Def: Virations sound should be "vibrations."
P examples: I wasn't sure how you were coming up with the topic words that you were.
N examples: Again, I didn't know how you were coming up with the topic words you came up with.

[ ] task044_essential_terms_identifying_essential_words.json

Def: The inclusion or exclusion of nouns in the question can be better explained.
N examples: An example showing more words that necessary being given would have been helpful.

[ ] task045_miscellaneous_sentence_paraphrasing.json

P examples: if they were not all the same example

[ ] task046_miscellaenous_question_typing.json:

Listed 4 examples, but last 2 examples were actually blank. A few more examples might be nice. Would have liked to see an answer that was "other". Examples listed were pretty obvious, would be nice to see some more complicated ones.

[ ] task049_multirc_questions_needed_to_answer.json

The questions weren't made clear
The instructions were understandable, but the example questions weren't clear.
The same as above. The example reasoning made sense even without the questions.

[ ] task053_multirc_correct_bad_question.json

Instructions were a bit circular.

[ ] task053_multirc_correct_bad_question.json

Do all questions need to be corrected or are some correct the way they are written?

[ ] task057_multirc_classify_incorrect_answer.json

Well, when it said, "an incorrect answer should be of the same semantic type of the given correct answer (e.g., both can be names of locations)", I thought it meant that, for example, "not very hot" would not be a good incorrect answer for a question like, "What was the temperature outside?" because a correct answer would be something like, "well over 100 F" and so I thought that a good incorrect answer should also mention a specific temperature, but the examples cleared that up for me, and actually "not very hot" would be a good incorrect answer, so it's a good thing the examples were there or I would have been confused.
It is a tad odd that the examples are numbered starting from "0", and I actually read three examples, went back to check something in the task definition, and then tried to return to the third question and got confused because that wasn't the question I was on, and obviously it turned out that the numbering started at "0", so I should have gone to the one numbered "2".
These are numbered starting at "0" too, though in this case I personally was not confused. Also, I believe that if the answer I'm evaluating is correct, that does not count as a good incorrect answer, and it would have been nice if there was just a simple example showing an output of "Yes" when the input showed an actually correct answer, unless of course I'm missing something and I actually am supposed to say, "Yes" in that situation.

Feedback for task 119-200:
Some of these were my tasks, I will fix them first, and then move on to the other ones.

@danyaljj  Do the "\n"s in the input text appear as a "new line" for the crowdworkers? There's this review for one of my tasks, although it has "\n"s before questions:

@danyaljj Do the "\n"s in the input text appear as a "new line" for the crowdworkers? There's this review for one of my tasks, although it has "\n"s before questions:

Yeah, we translate \n symbols to a new line in HTML. If there is already a new line, you can ignore this comment.

Here is feedback on task1-60. @swarooprm these are all yous! :)

Does this mean I shouldn't pick tasks 1-60 to fix?

@Palipoor Feel free to fix those. I am focusing on other priorities of this project.

Tasks 200-300:

[ ] task201_mnli_neutral_classification.json

regarding instructions: It's alright but there seems to be some room for interpretation based on common knowledge.
regarding p examples: They were adequate. The actual phrases are not as clear cut.
regarding n examples: More negatives might be helpful.

[ ] task202_mnli_contradiction_classification.json

regarding instructions: It might be helpful to say that 2 of the 3 choices agree with the statement but one disagrees, select the one that disagrees.
regarding p examples: Though M and M has a motto about not melting, in fact they really do, if you hold them long enough, perhaps the example shouldn't depend on motto knowledge.
regarding n examples: ``

[ ] task203_mnli_sentence_generation.json

regarding instructions: The actual instructions themselves are good! I think that the problem is in the next section
regarding p examples: Should define entailment
regarding p examples: I do not think that they were answered in a correct way. There is nothing about these examples that made sense
regarding n examples: They were all over the place

[ ] task204_mnli_same_genre_classification.json

regarding instructions: Need definitions of genres because I don't know what they all mean. What's slate and oup?
regarding instructions: Too many genres and a misspelling "oup"
regarding p examples: ultimately would want an example of each genre
regarding n examples: same as above, impossible to say what some of the genres will read like

[ ] task205_remove_even_elements.json

regarding n examples: the first negative example could further clarify by stating it is supposed to be odd numbers returned

[ ] task207_max_element_lists.json

regarding instructions: explain the maths terms better and that the output should be the maximum of each set
regarding p examples: they're fine if you understand sets, but need more if you don't
regarding n examples: the same, they're fine if understand the maths
regarding instructions: Possible identify the numbers in each sequence with an error that you put into the output with a caption of why the number was chosen.     I found the bad examples confusing.
regarding p examples: I thought the positive examples were easy to follow.
regarding n examples: What are the input examples pulled from?   Is it only the ones that show the example or many more?

[ ] task208_combinations_of_list.json

regarding instructions: Its just the answers not the questions.
regarding p examples: I still does not understand the task
regarding n examples: They were really confusing since no instructions were given.

[ ] task210_logic2text_structured_text_generation.json

regarding instructions: I don't understand computer coding or programming and felt like these instructions were trying to teach that to me.
regarding p examples: I had a hard time matching up what the inputs were to the outputs given the long strings of texts in the chart.
regarding n examples: As above, I had difficulty following the chart.
regarding instructions: I really had a very difficult time interpreting the list of commands that was given.
regarding p examples: I wish the examples were shorter so that I could more easily map the commands to the parts of the inputs.
regarding n examples: The examples were too long to be able to efficiently map on the list of commands provided to the inputs in the examples.
regarding instructions: pretty good
regarding p examples: pretty good
regarding n examples: okay

[ ] task211_logic2text_classification.json

regarding instructions: Context. The instructions really need context with how the commands are used and what can be done with them. Think of the reader like they're a young kid who needs the most basic explanation. Use real world comparisons and examples to make it easier. Use photos.
regarding p examples: See previous answer. The instructions provide no point of reference for a person to "imagine" how these commands are used and how they would interpret them. I have a background in teaching English to speakers of other languages. To be blunt, your instructions and examples are harder to read than some foreign languages.
regarding n examples: See previous statements.
regarding instructions: I was totally confused by the instructions given how lengthy the list of commands was. I had a hard time matching up which commands corresponded to which outputs.
regarding p examples: In theory the examples should have been a simple mapping exercise but I found the list of commands to be overwhelming and was unable to map the examples to the list.
regarding n examples: As indicated above, in theory the examples should have been a simple mapping exercise but I found the list of commands to be overwhelming and was unable to map the examples to the list.

[ ] task212_logic2text_classification.json

regarding instructions: The above instructions are good and very clear message.
regarding p examples: The positive examples are gave very good and positive thought.
regarding n examples: The negative examples are gave negative thought.
regarding instructions: Spaces between lines would be helpful, along with an example of what a table/row/heading looks like
regarding p examples: It would help to include the extra detail of what command indicates what is described (for example, in the first example, what word indicates the sum of rows).
regarding n examples: Example 3 and 4 are blank

[ ] task215_rocstories_incorrect_answer_generation.json

regarding instructions: The instructions are very clear and to the point. The only suggestion I have is to separate the two things listed. Displaying them in bullet form would be better. Would make it easier to quickly see the two points separately.
regarding p examples: The positive examples are great. Very straightforward with simple scenarios. My suggestion is to start with 1 and not 0. I thought starting with 0 was kind of unnecessary and odd. I count starting with 1 not 0.
regarding n examples: No real issue with the writing quality. But again, start with 1, not 0.

[ ] task220_rocstories_title_classification.json

regarding instructions: The directions are brief and clear. it's too the point without any added confusing terms.<newline><newline>NOTE that below you have the same title for both a and b in the last part of the hit???
regarding instructions: not sure why the sentences need to be numbered but otherwise fine.
regarding instructions: They seem comprehensible and indicate that a recap is not needed for the explanation.
regarding p examples: I think the positive examples were self explanatory. No issues.
regarding n examples: The negative examples were easy to understand. No issues.
regarding instructions: It couldn't have been explained any more simply than the explanation was and how it all gives you the chance to figure it all out.
regarding p examples: Nothing can be improved, it sounds fine as is. Maybe just have us state who or what the story is mainly about or what is focused on specifically.
regarding n examples: They clearly listed one good reason which was fair and to the point.

[ ] task221_rocstories_two_choice_classification.json

regarding instructions: They include al of the simple and easy words to make the instructions clear, even stating which order to put the letters in.
regarding p examples: There was one of them that I thought could have all three as an answer so that was slightly confusing.
regarding n examples: Totally makes sense to me with just two answers making the story sound connected.

[ ] task223_quartz_explanation_generation.json

regarding instructions: It's pretty straight forward, but still a little wordy. Maybe just plain Jane, but then some people would still get confused.
regarding p examples: They're great. Some are generic, but they do a great job at showing what kind of rating you're interested in.
regarding n examples: They are short and vague. I would include a description explain why, specifically, each example is negative.
regarding p examples: I think that some better examples could be used, in some spots it is hard to tell where the blank that I am supposed to fill in goes
regarding n examples: It seems like the answers in the negative examples are actually the correct answered one

[ ] task225_english_language_answer_generation.json

regarding instructions: State whether we should include citations if we're using sources like Merriam Webster's. Should we be copying and pasting from grammar sites and dictionaries or do we need to paraphrase everything?
regarding p examples: Some of these seemed like copy and pasted answers from dictionaries or grammar websites and I wasn't sure if that would be allowed.
regarding n examples: It's not clear how complex of a response is sufficient for what you're looking for.
regarding instructions: I thing every instruction is good orbed
regarding p examples: I thing positive is good

[ ] task226_english_language_answer_relevance_classification.json

regarding instructions: The negative examples are a bit simple and extreme.  Maybe show an example of one that is borderline so we can really understand.
regarding p examples: no issues with positive examples
regarding n examples: Give better negative examples where the answer could be positive if they had changed a small thing.

[ ] task231_iirc_link_classification.json

regarding instructions: It would be better to be more concise given the underwhelming pay.
regarding n examples: There is too many details to support the facts.
regarding p examples: example sentences are not labeled a b c d so impossible to tell how to correctly answer questions.
regarding n examples: same as above, example sentences are not labeled a b c d so impossible to tell how to correctly answer questions.

[ ] task233_iirc_link_exists_classification.json

regarding instructions: Easy to understand the task
regarding p examples: While easy to understand, huge walls of text are not helpful examples.
regarding n examples: Same as above, huge walls of text

[ ] task234_iirc_passage_line_answer_generation.json

regarding instructions: No real problems here but why not make it so that a sentence can be selected by clicking on the passage. That would be much more clear than providing a text box to be typed into (considering you just want us to select sentence(s) directly from the passage.
regarding p examples: In the examples, I'd have the selected sentence highlighted or bolded in the context passage.
regarding n examples: In the negative examples, I'd explicitly point out the good sentence that could have been selected (assuming there was one) to provide a full picture of the task.

[ ] task235_iirc_question_from_subtext_answer_generation.json

regarding instructions: Is the answer already given or must I come up with it? Why isn't that more clearly stated.
regarding p examples: The example explains the hit here, directions are lacking but examples tell you how to do it.

[ ] task236_iirc_question_from_passage_answer_generation.json

regarding p examples: All examples made sense.
regarding n examples: Answer in Example 0 on negative Examples is unclear.

[ ] task240_tweetqa_question_generation.json

regarding instructions: where are the questions coming from, what are they supposed to be about, what's the point of the exercise, anything really, idk wtf it's about
regarding p examples: more of them, how you arrived at those questions
regarding n examples: what the questions should have been like/about

[ ] task241_tweetqa_classification.json

regarding n examples: `More could be added.

[ ] task242_tweetqa_classification.json

regarding instructions: a couple more example wouldn't hurt
regarding p examples: more of them
regarding n examples: they don't make any sense, misspelling of sweep or swept, they seem to contradict the positive examples
regarding p examples: Only downside is some of the tweets are difficult to understand
regarding n examples: Same as above, semi-difficult to understand example tweets

[ ] task244_count_elements_in_set_union.json

regarding instructions: I understood it better after looking at examples
regarding p examples: Once I saw the examples I understood the task
regarding n examples: It clearly explains why it is wrong

[ ] task245_check_presence_in_set_intersection.json

regarding instructions: maybe explain what a set/intersection is for non-math people
regarding p examples: same as above, explain what a set is for those who might not know/understand
regarding n examples: same, explain set and intersection
regarding instructions: Elaborating more would probably help workers.
regarding p examples: In both positive examples, the explanation is wrong. There is no 10 in either of the Set 2s.

[ ] task246_dream_question_generation.json

regarding instructions: Overall quite good, you should clarify that we do NOT (or do if that's the case) need to answer the question.
regarding p examples: The first example is a bit obtuse, but it IS answerable, could you use a more clear example instead though?

[ ] task248_dream_classification.json

regarding instructions: give clearer category titles
regarding n examples: give more examples

[ ] task264_paper_reviews_accept_or_reject_classification.json

regarding instructions: doesn't say whether we should run spanish text through a translator or just use our best judgment.
regarding p examples: doesn't mention anything about how to deal with spanish. one example just switches halfway between spanish and english
regarding n examples: large walls of text

[ ] task266_paper_reviews_reviewer_perspective_classification.json

regarding p examples: would need to see an example of each rating from very neg to very pos

[ ] task268_casehold_legal_answer_generation.json

regarding instructions: the level of difficulty is very high and it's hard to ensure a correct answer

[ ] task270_csrg_counterfactual_context_generation.json

regarding instructions: You could bold some of the key words in order to come up with some good outputs because it does not feel natural to read about the task, instead it feels a bit heavy and I think it could be a little easier than that.
regarding p examples: For me, those were a bit unclear or too general, I would have liked some examples with more noticeable solutions.

[ ] task274_overruling_legal_classification.json

regarding instructions: Many of us are not familiar with the terminology used in the court s so a definition of overruling and non-overruling being included in the instructions will be helpful.

[ ] task275_enhanced_wsc_paraphrase_generation.json

regarding instructions: better examples
regarding p examples: put on red or capital letters the differences
regarding n examples: They were basically the same.

[ ] task276_enhanced_wsc_classification.json

regarding instructions: This is probably too much, unless you have a huge batch with a qualified passed by workers to get it.  It's confusing and tough.
regarding p examples: Wow, only 2 examples but so very many possible iterations? The instructions indicate many possible things but you only offer two possible ones.
regarding n examples: Really, we need more than 2 examples to parse all this out. This is truly a lot. Really.

[ ] task277_stereoset_sentence_generation_stereotype.json

regarding instructions: They are as clear as it can get
regarding p examples: The examples are clear what the task is looking for
regarding n examples: Good explanations of why those answers are bad
regarding instructions: Good instructions but it does entail some knowledge of the context of the sentence too so people may not like that part of it.
regarding p examples: Good and it shows that the sentence makes sense because of the knowledge you need too.
regarding n examples: It shows that describing things randomly isn't right and that you have to make it look valid too.

[ ] task278_stereoset_sentence_generation_antistereotype.json

regarding p examples: The examples for 2, 3, and 4 are blank.

[ ] task280_stereoset_classification_stereotype_type.json

regarding p examples: The first example seemed like it wasn't a positive statement.
regarding n examples: Question 0 seemed to be more about race than gender.

[ ] task283_dream_incorrect_answer_generation.json

regarding instructions: The initial instructions are clear
regarding p examples: Example 0 explanation is a whole mess. I'm super confused by that one. I've read it several times and still don't understand what happened to it. Example 1 is clear.
regarding n examples: This one is very clear

[ ] task284_imdb_classification.json

regarding p examples: I think example 0 is wrong.  It says that this review is negative:  Working with one of the best Shakespeare sources, this film manages to be creditable to it's source, whilst still appealing to a wider audience. Branagh steals the film from under Fishburne's nose, and there's a talented cast on good form.<newline><newline>I think this example should be positive and I don't see any expressions of "hateness" as the example states.

[ ] task285_imdb_answer_generation.json

regarding instructions: the 2nd example says there's hate but I don't see that at all
regarding p examples: they contradict each other
regarding n examples: they have the same problem of saying there's hate where there is none

[ ] task288_gigaword_summarization.json

regarding instructions: example 2 through 4 are blank
regarding p examples: all of the listings are horribly typed out.
regarding n examples: also very lame explanations and typing methods.
regarding instructions: Perfect. The directions are short and to the point. Whoever did this did it perfectly. Very clear.
regarding p examples: Examples give clarity and are what would be expected.

[ ] task289_gigaword_summarization.json

regarding instructions: I don't think the instructions need improving, I think it is very clear what the goal of this task is, and what your job as the worker is. Maybe where it says to generate a label as the worker, you can specify that you are generating an "output", so that it is explicit what your label is.
regarding p examples: I think the positive examples are very clear and so they make sense and are not confusing.
regarding n examples: I think you could maybe say that these examples are "incorrect" instead of negative. But it might make the worker feel better if they are not told they are incorrect and instead said they performed negatively, I suppose that would just rely on more feedback. But personally, I wouldn't mind if I was told that these examples were incorrect.
regarding instructions: Very straightforward and easy to understand.
regarding p examples: clear and obvious examples
regarding n examples: absolutely no problems.

[ ] task290_tellmewhy_question_answerability.json

regarding instructions: Pictorial tips
regarding p examples: Simple and elaborate examples
regarding n examples: Clear hints to identify

[ ] task291_semeval_2020_task4_commonsense_validation.json

regarding instructions: Easy to understand.
regarding p examples: Straightforward examples
regarding n examples: Barely even needs a negative example, very easy to understand
regarding instructions: Slight grammatical error.
regarding p examples: Some examples are missing.
regarding n examples: No suggestions at this time.

[ ] task292_storycommonsense_character_text_generation.json

regarding instructions: Everything was clearly explained.
regarding instructions: example 3 and 4 under positives were missing.
regarding n examples: Having a pronoun, then the Name was helpful and clarifying here.
regarding instructions: Very well said in one sentence. I am not sure if there are characters that can be included as a group though.
regarding p examples: Isn't Mormon missionaries a character (group) too?
regarding n examples: Good as can be as it even explains that pronouns do not work for this sort of task.

[ ] task293_storycommonsense_emotion_text_generation.json

regarding instructions: I think it was pretty clear but I would give more context on what exactly someone would be doing. I feel that the answers are pretty ambiguous since anyone could be feeling anything and people interpret things differently
regarding p examples: I think the examples were excellent and gave a clearer picture to the task
regarding n examples: I think the examples were perfect and gave a clearer picture about what is needed

[ ] task294_storycommonsense_motiv_text_generation.json

regarding instructions: The directions were clear!
regarding p examples: The examples were helpful.
regarding instructions: The directions were very clear in my opinion.
regarding p examples: The examples were super helpful!
regarding n examples: The explanation for each negative example was very helpful.

[ ] task295_semeval_2020_task4_commonsense_reasoning.json

regarding instructions: super clear and easy to understand
regarding p examples: fine examples
regarding instructions: Instructions are clear and brief.  I wish there was a clearer explanation of how "against common sense" is defined.
regarding p examples: ``
regarding n examples: I wish the examples centered more on content rather than simply substituting another ordinal direction for a letter, only one type of these examples is really needed.

[ ] task296_storycloze_correct_end_classification.json

regarding instructions: The instructions are good because they are simple, uncomplicated and understandable. Bravo on this one. perfect.

[ ] task298_storycloze_correct_end_classification.json

regarding instructions: I have no suggestions at this time.
regarding p examples: Some examples were missing.
regarding n examples: I have no suggestions at this time.

[ ] task299_storycloze_sentence_generation.json

regarding instructions: The instructions should indicate that you expect a number to be used. It should also be mentioned that the added phrase or detail takes OVER a numbered position.
regarding p examples: I think a couple more good examples would help
regarding p examples: It wasn't clear what restrictions were in place to limit our creativity on the last sentence of the first example.; Daniel: first example: Sentence1: Sarah had been dreaming of visiting Europe for years. Sentence2: She had finally saved enough for the trip. Sentence3: She landed in Spain and traveled east across the continent. Sentence4: She didn't like how different everything was.
regarding n examples: I couldn't tell how some of the last sentences really differed in terms of the creative liberties that the author was taking.
regarding n examples: Example 1 missed that it also repeats a step already existing so it's not actually a missing part; Daniel: Sentence1: She had finally saved enough for the trip. Sentence2: She landed in Spain and traveled east across the continent. Sentence3: She didn't like how different everything was. Sentencce4: Sarah decided that she preferred her home over Europe.

[ ] task300_storycloze_order_generation.json

regarding instructions: a bit confusing but not bad after a couple of reads
regarding p examples: they were kind of jammed together and hard to read
regarding n examples: they could be formatted better
regarding instructions: Very clear and to the point
regarding p examples: The examples are clear
regarding n examples: There is nothing confusing

Tasks 300-399:

[ ] task301_record_question_generation.json

regarding p examples: Far too much reading required in the examples
regarding p examples: all examples have way too much reading involved

[ ] task303_record_incorrect_answer_generation.json

regarding p examples: The write up was a bit too complex in its writing.
regarding p examples: Overly long examples for both positive and negative
Daniel: surprisingly, most people answer the instances incorrectly (probably not noticing "incorrect"). Wondering if we can bring attention to "incorrect" by changing it to "incorrect".

[ ] task305_jeopardy_answer_generation_normal.json

regarding instructions: Changing "all answers should be lowercased" to "answers should be in all lower cased letters" could improve clarity.
regarding p examples: The examples were good and easily understood but examples 3 and 4 were blank. Just a personal opinion but perhaps examples that use abbreviations (e.g. "Dec. of Indep.") should be avoided as they could be ambiguous to participants. In terms of presentation, it would be useful to start off with "Example 1" and not "Example 0." Additionally, setting "Category:" and "Clue:" as subheadings could also help with readability.
regarding n examples: As mentioned previously in the positive examples field, starting with "Example 0" is a bit odd.

[ ] task308_jeopardy_answer_generation_all.json

regarding instructions: Instructions could be a bit more thorough.
regarding p examples: Could use more examples
regarding n examples: Could use more examples

[ ] task309_race_answer_generation.json

Daniel: somehow our annotators answer the instances incorrectly. Wondering if there is a systemic bug in the instance labels.

[ ] task311_race_question_generation.json

regarding p examples: way too much text for all examples
regarding instructions: I think it would be easier to just ask for a sentence that basically sums up what the article was about.
regarding p examples: I got a pretty good understanding of what kind of information  you are looking for.
regarding n examples: I think the negative examples were good, but they all just had one small problem with them like saying British instead of American, I wonder if there would be an example that didn't just get a small fact wrong, but an example of a bad summary for the article.

[ ] task316_crows-pairs_classification_stereotype.json

regarding instructions: The instructions make it clear that the acceptable output is either stereotype or anti-stereotype yet the questions should sometimes not be answered without any explanation. For example, the statement "Anna's mother was proud of her for behaving in such a strong masculine way"; I would lable it stereotypical if I was talking about a strong way being considered masculine, but I might lable it anti-stereotypical if I was talking about Anna's mother being proud of her daughter for acting manly because I don't think mothers want to see their daughters acting manly (even though they want them to be strong). I don't know how you would know if I thought the group being stereotyped was mothers or men (without an explantion). I am probably over thinking the task but I wanted to provide some feedback.<newline>
regarding instructions: I think it could use some examples of common stereotypes or what stereotypes usually are making statements on. For example: race, ethnicity,  gender, ability, etc.
regarding p examples: The examples are clear, but only rely on one form of stereotyping (race or ethnicity).

[ ] task319_stereoset_classification_profession.json

regarding n examples: Definitely needs a few more examples

[ ] task320_stereoset_classification_race.json

regarding n examples: Example 1 is listed as a negative example, yet the provided answer seems to agree with the answer given in the explanation (anti-stereotype).

[ ] task322_jigsaw_classification_threat.json

regarding instructions: I think anyone can tell what to do here given the examples too and the explanation for it all. I didn't have any issues with it.
regarding p examples: I can tell how and why the options were chosen as they were, it is a no brainer.
regarding n examples: Easily understood by me without much reading.

[ ] task324_jigsaw_classification_disagree.json

regarding instructions: wording on what agree means could be improved.  (rather than lack of disagreement, should explain you're seeking agreement.  maybe add a neither option if it appears neutral?)
regarding n examples: explanations on examples make no sense....?
regarding instructions: The comments are lacking context, which makes this a bit confusing, but still understandable. Negative examples don't entirely make sense to me.
regarding p examples: Just lacking some context, as I mentioned above.

[ ] task326_jigsaw_classification_obscene.json

regarding instructions: not sure if obscenity refers just to sex, or if negative language too
regarding p examples: Example 1 lists a comment about the weather as obscene
regarding instructions: " materials and acts of pornography" not sure what this means by materials.
regarding p examples: One of the examples says damn in it, and it says to label it obscene because it refers to sexual content. it doesn't though.
regarding p examples: i want to improved that quality
regarding n examples: i don't have any negative example

[ ] task328_jigsaw_classification_insult.json

regarding instructions: Instead of listing examples one by one, it's better to list first the examples of either insult or non-insult followed by the other.
regarding p examples: Some of the output and definitions do not match. For example, the example says "non-insult" but in the explain, it says it is an "insult".
regarding n examples: There was only one negative insult that I could see. There could be more.

[ ] task329_gap_classification.json

regarding n examples: The word "refers" was written as "efers" twice.

[ ] task330_gap_answer_generation.json

regarding n examples: typos in the instructions (efers instead of refers)

[ ] task331_gap_incorrect_answer_generation.json

regarding instructions: Doesn't really explain what an ambiguous pronoun is

[ ] task332_tellmewhy_answer_generation.json

regarding instructions: Could use a better definition of what exactly is considered a "complete: answer
regarding p examples: explanations need to be expanded upon
regarding n examples: could use more examples

[ ] task333_hateeval_classification_hate_en.json

regarding instructions: It might be helpful to have some sort of content warning in the instructions for the task. Also to ask about demographics since that might influence what people identify as hateful or not. Also, since there are only two options a radio button or dropdown would be clearer and indicate that only those two outputs should be used.
regarding p examples: They are clear, but more examples could be given.
regarding instructions: Change "sex" to "gender"<newline><newline>Make it clear that we are supposed to type "Non-hateful" or "Hateful"<newline><newline>Should we click on links? That can completely change the answer (like the Trump tweet on this page) and also ups the work time. It isn't mentioned anywhere.
regarding p examples: Include an example with a link that changes the context.
regarding n examples: The negatives are showing wrong word choices not just bad examples. The tweet below this about Trump would be a good type of example to use because if I click the link it is definitely hateful, but if I don't click the link, it isn't

[ ] task334_hateeval_classification_hate_es.json

regarding instructions: Doesn't state how we should translate the spanish posts
regarding instructions: doesn't mention whether we're supposed to infer if the spanish is offensive or not or use a translation service
regarding p examples: both positive and negative examples provide translations but the actual task does not
regarding instructions: Feel like hatefulness could be a wide spectrum that depends on each individual's personal attitudes.
Daniel: if this task involves Spanish, is this mentioned in the input_language tags?

[ ] task335_hateeval_classification_aggresive_en.json

regarding instructions: Bit of a grey area separating hate, aggression, and abuse
regarding instructions: aggressiveness and abusiveness seem kind of like grey areas

- Daniel: if this task involves Spanish, is this mentioned in the `input_language` tags?

[ ] task336_hateeval_classification_aggresive_es.json

regarding instructions: Doesn't state how/if we should translate the spanish
regarding p examples: pos and neg examples translate the spanish for me but the actual task does not
regarding instructions: This question says goodbye and heard menu<newline>
regarding p examples: The liquid word is one of the most important for everyone the person.
regarding n examples: Heard speech should not be spoken to in public or to anyone.
Daniel: if this task involves Spanish, is this mentioned in the input_language tags?

[ ] task338_hateeval_classification_individual_es.json

regarding instructions: I honestly don't understand any of it given the examples.
regarding p examples: I don't understand why one is positive and the other is negative when they're both hateful so I can't even say if it's good or not
regarding n examples: I don't understand why one is positive and the other is negative when they're both hateful so I can't even say if it's good or not
regarding instructions: Straightforward but doesn't mention if/how we should translate spanish to english

[ ] task339_record_answer_generation.json

regarding p examples: All examples are giant walls of text

[ ] task340_winomt_classification_gender_pro.json

regarding p examples: could use more examples (altho not many needed for something so straightforward)
regarding n examples: could use more examples (altho not many needed for something so straightforward)

[ ] task341_winomt_classification_gender_anti.json

regarding instructions: "The coreference link with gendered pronouns", sounds like you have a chart or something. Please, either define what coreference link is clearly or just omit it, it's easy enough to figure out from the examples. You're wasting time of those taking the survey because they will be looking for a "chart". Mention clearly that you only really want Male/female though.
regarding p examples: Examples make clear what's ok.
regarding n examples: You say woman is wrong but, you hadn't really clearly defined this before. it's good you're doing it here, but what about "Man", Trans, etc?

[ ] task343_winomt_classification_profession_anti.json

regarding instructions: No idea what a coreference link is

[ ] task344_hybridqa_answer_generation.json

regarding p examples: Please be more clarify like if there is a positive correlation between question and answer.
regarding n examples: Again more clarify to be the negative correlation between question and answer
regarding instructions: Doesn't say what we should do if we have no knowledge of the question being asked

[ ] task345_hybridqa_answer_generation.json

regarding instructions: Gigantic list of speech tags that no one would be able to memorize

regarding instructions: Gigantic wall of text. Over 20 different tags I'm supposed to memorize, and I have to tag every single word

regarding p examples: ``

regarding n examples: ``

[ ] task346_hybridqa_classification.json

regarding instructions: Gigantic wall of tags I'm supposed to memorize.
regarding instructions: The instruction may be quite easier
regarding p examples: Lack of clarities
regarding n examples: It may be clear more

[ ] task347_hybridqa_incorrect_answer_generation.json

regarding instructions: I think it would be better to put the tags in a listing going from the top to the bottom. Having them bunched together in paragraph format makes it a bit hard to read.
regarding p examples: Examples 2, 3, and 4 are completely blank. They should either be filled in or removed.
regarding n examples: I think giving more examples would make the instructions more clear.
regarding instructions: giant wall of tags to remember, but since we have to give an implausible one, maybe it's easier.

[ ] task348_squad2.0_unanswerable_question_generation.json

regarding instructions: very intresting one.
regarding p examples: it also a positive one.
regarding p examples: for both pro and con, the examples are way too long to read

[ ] task350_winomt_classification_gender_identifiability_pro.json

regarding p examples: Why is there always some missing?

[ ] task351_winomt_classification_gender_identifiability_anti.json

regarding p examples: could use more examples
regarding n examples: could use more examples
regarding instructions: The examples were simple and I could understand the good examples but the negative examples confused me.
regarding p examples: These were simple and fairly easy to understand.
regarding n examples: These were a little confusing.

[ ] task352_coda-19_classification.json

regarding instructions: Doesn't explain what each category means
regarding instructions: Too many categories for paragraphs that are too long. Doesn't explain what each category is.

[ ] task354_casino_classification_negotiation_no_need.json

regarding instructions: There were not many good examples and non-examples.
regarding p examples: I really needed more examples.
regarding n examples: I needed more examples.

[ ] task357_casino_classification_negotiation_small_talk.json

regarding instructions: it's too wordy. don't need to know what they're negotiating about.

[ ] task359_casino_classification_negotiation_vouch_fair.json

regarding instructions: pretty vague explanation for what vouch-fair is.

[ ] task360_spolin_yesand_response_generation.json

regarding instructions: The examples and negative examples were good.
regarding p examples: The explanations were good and clear.
regarding n examples: They helped me understand what would be a bad answer.
regarding instructions: If these HITs keep running I would really like the instructions to show up again after the feedback boxes so I don't have to scroll all the way back up

[ ] task362_spolin_yesand_prompt_response_sub_classification.json

regarding instructions: it's a bit too wordy but it's mostly clear

[ ] task365_synthetic_remove_vowels.json

regarding instructions: Examples 3 & 4 are missing from the good examples. I would also bold the actual instructions of removing the vowels, not everybody reads the directions clearly.
regarding p examples: They were fine, just reiterating that they were missing 3 & 4.

[ ] task367_synthetic_remove_floats.json

regarding instructions: Because I am dumb, I had to google to remember that an integer just means a whole number. Should probably include that integers be separated by a comma
regarding instructions: I would suggest defining what an integer is for those that don't recall.
regarding p examples: Examples 2, 3, and 4 are blank. Examples 0 and 1 are fine.
regarding n examples: The negative examples are very good. I like that they covered all sorts of different types of incorrect output.

[ ] task369_synthetic_remove_odds.json

regarding instructions: It made it very clear that odd numbers are not included in the output.  I do think it could clarify what to do with the number 0 though.
regarding p examples: It clearly showed examples of only even numbers being included in the output and the odd numbers remaining in the input.  I liked how it clarified that 0 would be included with even numbers.  Examples 2,3, and 4 could have been filled in because it was a bit awkward that they were blank.  They could include examples with negative numbers.
regarding n examples: I think a couple more negative examples could have been shown just to let people know for sure what isn't right.

[ ] task371_synthetic_product_of_list.json

regarding instructions: Suggest changing positive and negative examples to correct and incorrect examples respectively to make them more clear.
regarding n examples: the explanation on negative example 2 is unclear because the output does not look like a list of lists even though they are in brackets.
regarding instructions: I don't think there should be a comma after "of lists" in this sentence: In this task you will be given a list, of lists, of integers.<newline><newline>It makes it confusing and I had to reread it a few times.
regarding p examples: They seemed okay but there were not enough of the examples.
regarding n examples: There are enough examples. I am not really grasping the tasks though.

[ ] task373_synthetic_round_tens_place.json

regarding p examples: In the sentence "The output correctly round each integer in the input list to the nearest ten.", "round" should be "rounds".

[ ] task374_synthetic_pos_or_neg_calculation.json

regarding p examples: In the sentence "After the rule is applied 6 and -4 are equal, because that are multiplied by different numbers.", "that" should be "they".

[ ] task375_classify_type_of_sentence_in_debate.json

regarding instructions: this should be more informative
regarding p examples: the example should be more easy to understand
regarding n examples: there is no examples in this instructions

[ ] task376_reverse_order_of_words.json

regarding instructions: Explanation of the answer can be shortened such that the last sentence can read : So, the output is the correct answer.
regarding p examples: Article is missing such that "A" should be added before "luggage" in example 0.
regarding n examples: Cut off the "instead of...." part in the answer to shorten the answer.
regarding p examples: In the sentence "The order of words in the given sentence is 'lugguage', 'surrounds', 'a', 'vehicle', 'in', 'an', 'underground', 'parking', 'area'.", "lugguage" should be "luggage".
regarding instructions: doesn't say if we should include punctuation

[ ] task377_remove_words_of_given_length.json

regarding instructions: Noting that we could be critical is VERY helpful. I don't want to always nitpick on typos or phrasing because we're often asked to ignore that.
regarding p examples: Just add a couple more examples.
regarding n examples: It seems straightforward and the 2 examples are good, but once I do a few HIT's I may notice something different.

[ ] task378_reverse_words_of_given_length.json

regarding instructions: It was easy to understand especially with the examples.
regarding p examples: Simpler sentence examples.
regarding n examples: Simpler sentence examples.

[ ] task379_agnews_topic_classification.json

regarding instructions: Doesn't mention what we should enter if the input doesn't match any of the 4 categories
regarding p examples: Should probably contain an example of each category

[ ] task381_boolq_question_generation.json

regarding n examples: In the sentence "There's no information in the passage to help answer this question, this is not a valid output.", maybe there should be a "therefore" before "this is not a valid output".
regarding p examples: could use more examples

[ ] task383_matres_classification.json

regarding instructions: The word "anchored" didn't click with me. The phrasing on the instructions sounds much more confusing than the task. Also, fix the line break issue after "verb separated with a"<newline><newline>Most importantly, do I just type "yes" or "no"?<newline><newline>Switching from one HIT that was "remove words of 7 letters" to this task is very confusing and overwhelming. They should be different batches.<newline><newline>The time it takes to read the instructions and do the task is much longer when they are intermixed like this and end up being underpaid or make me want to return. I'm already over 13 minutes on this one!<newline>
regarding p examples: I definitely need more examples. Why isn't "invite" anchorable? It is a verb and it happened in the past because they decided to invite at last years (in the past) convention. Or is it only anchorable if it was "invited". This is really confusing!
regarding n examples: I liked that the first example was longer and the second shorter.  But, I think we need a few more long examples where the word doesn't come at the end, which makes me want to skim.

[ ] task384_socialiqa_question_classification.json

regarding p examples: In the sentence "Because Through context, we show that Sydney Sydney walks past homeless women and the answer is 'Sympathetic' which is an emotion.", Through shouldn't be capitalized.
regarding n examples: Example 3 has a confusing 'answer' - is the incorrect answer all three?

[ ] task386_semeval_2018_task3_irony_detection.json

regarding instructions: These instructions were better than some others. There was bold font and I knew I had to label as  'IRONIC' or 'NOT' and not "NOT IRONIC" - also, do they need to be in all caps?<newline><newline>But, so ambiguous! I'd hate to be judged for my quality of work because I didn't think something was ironic that other people did think was ironic.<newline><newline>Do we have to follow a link to see what the tweet is referring too as is the second test on this hit? I can't tell if they are being ironic unless I follow the link.
regarding p examples: Example 2 seems Ironic to me! They aren't dying as their lifeblood leaks from a stab wound. They are ironically 'dying' because there are only 3 episodes left.
regarding n examples: You used the same bad example for a negative as a positive and I still feel like it IS ironic. And this is why i'd greatly worry about the task. It's too ambiguous and
regarding p examples: could use more examples
regarding n examples: Negative examples definitely benefit from covering a wide variety of potential screwups.  Hard to do that with just 1 example

[ ] task389_torque_generate_temporal_question.json

regarding instructions: More information on what is considered a temporal relation. It seems too similar to the bad examples.
regarding n examples: I could take the response and still make it temporal so I am confused.
regarding instructions: Please mention in the instructions that a event must be mentioned in the output as a temporal comparison.
regarding p examples: The instruction says a temporal relation describes the relation between two things with respect to time, but the output in example 2 does not mention any event.

[ ] task390_torque_text_span_selection.json

regarding instructions: Does not explain what temporal relation is
regarding instructions: Written in a confusing manner but made clearer by the examples. Sort of.

[ ] task391_causal_relationship.json

regarding p examples: There is space for 5 examples, but the last 3 have no text. Either add actual examples there or remove the blank spaces. There is also no period or punctuation at the end of the Example 1 input sentence.

[ ] task392_inverse_causal_relationship.json

regarding p examples: not good enough examples to understand
regarding n examples: there is no examples there in some questions
regarding instructions: 'possible causation' sounds more technical than needed. The phrasing makes me feel like I need to pull out a dictionary.
regarding n examples: definitely need a couple more examples

[ ] task397_semeval_2018_task1_tweet_anger_detection.json

regarding instructions: Differentiation of positive and negative examples can be made even clear.
regarding p examples: The instruction quality is good enough to understand and to make a clear response.
regarding n examples: It should have been given in a better quality.

[ ] task399_semeval_2018_task1_tweet_sadness_detection.json

regarding instructions: Make the instructions bulleted and larger font. "You must judge whether the author of the tweet is sad or not." <newline>"Label the instances as 'Sad' or 'Not sad' based on your judgment."<newline><newline>That's really all I need to see to know that I need to change my focus from verbs to sadness. But really, this is TOUGH when you are going through so many different styles of tasks
regarding p examples: Knowing that we can go off of a hashtag makes a BIG difference, because so many hashtags are purposely sarcastic and the person isn't sad at all.
regarding n examples: Need more examples. This is so different from one persons view to the other. And tweets can be so ambiguous.

[ ] task400_paws_paraphrase_classification.json

regarding n examples: Needs more than one example

Feedback regarding tasks 600-850:

[ ] task607_sbic_intentional_offense_binary_classification.json

regarding instructions: I'm not really sure what "non-offensive statements that are underhandedly offensive" means, an example would be nice.<newline>

[ ] task608_sbic_sexual_offense_binary_classification.json

regarding n examples: Some of the language in the negative examples are REALLY offensive to read.

[ ] task609_sbic_potentially_offense_binary_classification.json

regarding instructions: "Offensive language" seems a bit like a grey area that would mean different things for different people.
regarding instructions: It saying input and output was confusing, why not just have it say what they are like "example" and "judgement" or something.
regarding n examples: Examples were good, maybe a few more that aren't quite as simple

[ ] task611_mutual_multi_turn_dialogue.json

regarding instructions: Its fine as is. The letters f and m are not capitalized  in that part though unlike the other sentences, so that slightly confused me at first only.
regarding p examples: Very good and easy to read. Could have been somewhat of a shorter story.
regarding n examples: Shorter story would be enough.

[ ] task613_politifact_text_generation.json

regarding n examples: The output for example 1 in the negative examples should be taxes because the input didn't mention small businesses.
regarding instructions: Instead of "the sub-string" it would be more acceptable to use "a sub-string" in the last sentence. It would be helpful if the instructions included possible strings and explained whether an output could contain more than one word.
regarding p examples: The explanations have some questionable grammar and are a bit unreadable. It would help if they didn't simply restate the input and output and instead offered more context. Also, it's not clear why the broader category of "military" was chosen for example 2 instead of "weapons" or "nuclear".
regarding n examples: The explanations are very unreadable, especially example 3. It's not clear if the entire phrase "legal issues, supreme court" is the desired output.

[ ] task614_glucose_cause_event_detection.json

regarding p examples: I definitely think you should have more examples. I don't think that 2 are enough.
regarding n examples: As with the positive examples, I think you need to provide more than 2.

[ ] task615_moviesqa_answer_generation.json

regarding instructions: A little bit more than a couple sentences would help so I know I'm doing it properly.  You want me giving a rating for a movie, okay what website am I using, IMDB, Rotten Tomatoes, something else?  Am I just looking at user reviews or critics or both?  Do I just provide 1 rating and that's it or put it into a sentence?
regarding p examples: Do I just provide the answer or do I need a bit more?  Like if it says it's a war movie, should I just say "war" or write "X movie is a war movie about such and such"

[ ] task616_cola_classification.json

regarding instructions: Way too much leeway in determining whether a sentence is acceptable.

[ ] task617_amazonreview_category_text_generation.json

regarding instructions: These should have example categories; also they should explain the syntax (why is an underscore used? Is it required for all categories using two words?)
regarding p examples: The examples are clear, but it's not clear why an underscore is being used in 'home_improvement'.
regarding instructions: Very good at pointing out what to do and encouraging criticalness too.
regarding p examples: I saw some typos and that hyphen in between two words that really doesn't belong there but overall otherwise it makes good sense.
regarding n examples: I didn't see anything to critique about there.
regarding instructions: The instructions should list all of the possible categories.

[ ] task618_amazonreview_summary_text_generation.json

regarding instructions: Instructions specify Amazon food products though neither task of reviews to be summarized are for food.
regarding n examples: Layout design error, sentences hanging outside the tinted area. (Perhaps it's my screen?)

[ ] task625_xlwic_true_or_false_answer_generation.json

regarding instructions: Directions make sense. Just reading quickly I get the entire gist of it. No problem.  That's the key to good instructions NOT being overly wordy but being clear. You did this.
regarding p examples: examples make sense.
regarding n examples: Examples work in each case.

[ ] task628_xlwic_word_with_different_meaning_sentence_generation.json

regarding instructions: Don't need to have examples in instructions. Doesn't say what to do if selected word has only 1 meaning
regarding instructions: Doesn't say what to do if a word doesn't have multiple meanings.

[ ] task630_dbpedia_14_classification.json

regarding instructions: It's not that clear what to do when there's additional text in the title, other than in brackets, but the main portion of the title is present in the text.
regarding instructions: It's clear that text in brackets can be ignored, but what about other text, like what if there's a semi colon, or what if the entity in the text is part of a longer title?

[ ] task632_dbpedia_14_classification.json

regarding instructions: These could use some clarification about tasks that concern a person, but the person isn't the main subject of the text (like a text about a book, that mentions the name of the author)

[ ] task633_dbpedia_14_answer_generation.json

regarding instructions: Simplify 'You are given a question and options. You are expected to choose from the options and output the number of that option' to 'pick the correct number'

[ ] task641_esnli_classification.json

regarding p examples: There are some typos but they don't affect understanding of the task. More importantly, there are no examples of using the 'N' option.
regarding n examples: There are no examples using the 'N' option.
regarding p examples: Could use a couple more examples, especially for a natural case.
regarding n examples: Could use a couple more examples, especially for a natural case.

[ ] task642_esnli_classification.json

regarding p examples: Examples"<newline>Ex 1 is understandable given both refer to a boy wearing a blue jacket.  We assume it's the same boy agree/disagree = yes.  Ditto with the logic on Ex 2, makes sense.  However, I couldn't determine in the tasks below: Is it the same man/forearms being described?  The same woman bowling/sleeping?  So I answered No to both for I couldn't clearly determine if the actors were the same.  (note typo,  'collared' shirt, in first task below.) Do forearms in both sentences indicate a yes?  Very little to hang it on in the second sentence.  {{My head is spinning, lol}}

[ ] task649_race_blank_question_generation.json

regarding instructions: The instructions are very brief. It would help to know if there are guidelines that must be followed when completing the task.
regarding p examples: The examples are fine but fill in the blank type questions aren't usually about filling in a phrase or sentence but more about just filling in a word.  Your examples want people to fill in more than just one word, if that's the case you have to explain that.
regarding n examples: All the examples are way too long.  The entire article or passage doesn't need to be so huge.

[ ] task664_mmmlu_answer_generation_abstract_algebra.json

regarding p examples: Unnecessary line break in example 2
regarding instructions: It's good. Already your instruction was so good .
regarding p examples: Your example is very perfect.
regarding n examples: You are clear every doubt

[ ] task665_mmmlu_answer_generation_anatomy.json

regarding instructions: it was easy to understand.
regarding p examples: it was easy to understand.
regarding n examples: it was easy to understand.
regarding p examples: If all you're looking for is just a multiple choice type answer then the examples are good

[ ] task666_mmmlu_answer_generation_astronomy.json

regarding instructions: Am I only supposed to just choose a letter even if it's not the right answer?  There didn't seem to be a right answer for the questions provided, the second one just listed evidence for the Giant Impact Hypothesis.
regarding p examples: The examples were fine but they were nothing like the questions I had to do
regarding n examples: The examples were fine but they were nothing like the questions I had to do
regarding instructions: Doesn't say what to do if multiple answers are plausible.

[ ] task667_mmmlu_answer_generation_business_ethics.json

regarding instructions: The instructions could explain that there will be multiple fields per answer.
regarding p examples: I felt confident that I understood the task following the examples, but I'd recommend highlighting or otherwise emphasising the answers in the explanation text.
regarding instructions: it could have been simplified even more.
regarding p examples: it could have been simplified even more
regarding n examples: it could have been simplified even more

[ ] task668_extreme_abstract_summarization.json

regarding instructions: Should have specifics on how long the summary should be and what it must include
regarding instructions: Needs better instructions on generating a summary. Can it be a short phrase, multiple sentences, etc

[ ] task669_ambigqa_answer_generation.json

regarding instructions: It might be a good idea to clarify that the response should be brief and not in the form of a sentence.
regarding p examples: England, Washington, and DC should be capitalized.
regarding n examples: Typos:<newline>ad (should be and)<newline>Washigton<newline>conntrol

[ ] task670_ambigqa_question_generation.json

regarding p examples: The outputs in both examples 1 and 2 have grammatical errors. "elaborate" should be "elaborates". The output in example 2 is incoherent or grammatically incorrect.
regarding n examples: "ask" in the output of example 1 should spell "asks".
regarding instructions: Some parameters on what we can or cannot include would improve clarity.

[ ] task671_ambigqa_text_generation.json

regarding instructions: The last sentence in  the instructions is incomplete:"....should be separated by", by what?
regarding n examples: "elaborate" in the explanation in examples 2 and 3 should spell " elaborates".

[ ] task672_nummersense.json

regarding instructions: doesn't make sense at all. based on the examples I think the instructions should be changed to 'answer the word problem with a word'

[ ] task673_google_wellformed_query_classification.json

regarding instructions: The instructions could delve a little deeper into what exactly you're looking for. It's a big vague.
regarding n examples: The examples are just the flipped versions of the positive ones.
regarding instructions: If the query is well formatted and spelled correctly, but is wrong ("the sky is purple"), is it still "Good"?

[ ] task675_google_wellformed_query_sentence_generation.json

regarding instructions: The description of what you're doing in the task could be written in a better way with less words. I would tell people they need to find the query from the list that is the most well structured and formed. The way the instructions were written made you reread them a few times to really understand what was being asked.

regarding p examples: The examples are pretty good if you understand the instructions well. The explanation of the examples is simple which keeps it easy to understand.

regarding n examples: The examples make sense if you understand the instructions. Once the instructions are written a little better, you will understand exactly what the examples mean the first time. The explanations are simple but they should be because the instructions are simple.

regarding instructions: It says I'm given a set of queries separated by ''.  I don't know what that is referring to, it looks unnecessary.  Other than that the instructions are clear.

regarding p examples: Your very first example you chose said it was free of spelling and grammar errors but it wasn't, Haiti wasn't capitalized.  I also don't know why there's a space before the question marks, that shouldn't be there.

regarding n examples: ``

[ ] task686_mmmlu_answer_generation_college_biology.json

regarding p examples: examples based upon people knowing things that few turkers would know offhand
regarding n examples: examples based upon people knowing things that few turkers would know offhand
regarding instructions: maybe the need to be instructions clearer. Am I supposed to know the correct answer. Are we supposed to just answer (A,B,C,D) or our we supposed to type out the answer too?
regarding instructions: A lot of these are now similar and perfect and I hope that doesn't flag me for not typing much feedback

[ ] task688_mmmlu_answer_generation_college_computer_science.json

regarding p examples: Example 1 says coulb instead of could.
regarding instructions: It would be helpful to include how we will know which option is correct: whether purely through common knowledge or through a specific reference source. Also, it would be good to know what one should do if there is unclear language or are typos.
regarding p examples: There's a typo in the Explanation of Example 1. It would also be helpful if the explanations were less vague. In both cases, the example explanations seemed to just restate the question and answer.
regarding n examples: Both negative examples were undesirable due to rejected output formatting. It would be helpful to have a negative example where the answer/content makes it incorrect if that is something that can happen.
regarding instructions: It says "You need to answer the question by selecting the correct option."  This is vague, what is the correct option?  Choosing the actual answer or just choosing a letter or is it something else?
regarding p examples: ``
regarding n examples: If the examples are meant to be answered correctly and not just pick a letter, then there should be examples of choosing a letter but it being the wrong one.

[ ] task690_mmmlu_answer_generation_college_medicine.json

regarding p examples: Both + and - examples have sports questions embedded despite the instructions having to do with college medicine
regarding p examples: examples based upon people knowing things that few turkers would know offhand
regarding n examples: examples based upon people knowing things that few turkers would know offhand

[ ] task692_mmmlu_answer_generation_computer_security.json

regarding instructions: Doesn't say what to do on the off chance that two of the answers are plausibly correct.

[ ] task693_mmmlu_answer_generation_conceptual_physics.json

regarding n examples: "Muliple" in the output of example 2 in the negative example is misspelled. It should be "Multiple".
regarding instructions: maybe the need to be instructions clearer. Am I supposed to know the correct answer. Are we supposed to just answer (A,B,C,D) or our we supposed to type out the answer too?

[ ] task694_mmmlu_answer_generation_econometrics.json

regarding instructions: At first I thought I was grading the answer itself. How do I know what the correct option is

[ ] task695_mmmlu_answer_generation_electrical_engineering.json

regarding instructions: the explanation is easy to understand.
regarding p examples: the explanation is easy to understand
regarding n examples: the explanation is easy to understand

[ ] task697_mmmlu_answer_generation_formal_logic.json

regarding instructions: Would need a brief explainer on what exactly formal logic is.

[ ] task698_mmmlu_answer_generation_global_facts.json

regarding instructions: The instructions could be more brief and written in a more readable way. I.e. your task to select the correct multiple choice answer from the question shown below.
regarding p examples: The example was good but it would be nice to reiterate why the positive example is positive. If you read the positive example first, you won't really understand the task, until you read the negative example.<newline><newline>
regarding n examples: The examples are clear. However, there is a typo. "Muliple options can not be correct." Cannot is one word.

[ ] task699_mmmlu_answer_generation_high_school_biology.json

regarding instructions: It can be more clear
regarding p examples: Its more clear
regarding n examples: It can be better

[ ] task700_mmmlu_answer_generation_high_school_chemistry.json

regarding instructions: It would be helpful if the instructions included how we determine which answer is correct whether it's via common knowledge, verification from a specific source, or something else.
regarding p examples: Example 2 is missing a question. The answer given seems to conflict with the information present in the input.
regarding n examples: If it's possible for an answer to be incorrect based on content instead of just formatting, it would be helpful to include this as an example. If the output was B for Example 1, would it be incorrect and why?
regarding instructions: None. Easy to understand.
regarding p examples: None. Easy to understand.
regarding n examples: None. Easy to understand.

[ ] task702_mmmlu_answer_generation_high_school_european_history.json

regarding instructions: It's not entirely clear if answering these questions depends on existing knowledge, or just information contained in the texts.
regarding instructions: It's not clear if generating these answers depends just on knowledge in the passages, or if they require existing knowledge.

[ ] task703_mmmlu_answer_generation_high_school_geography.json

regarding instructions: Instructions are clear. The questions are straightforward enough.
regarding p examples: The examples help a lot, they give good scope, because I was worried when you mentioned geography.
regarding n examples: Examples take advantage of using positive sometimes and reversing, which is good.

[ ] task705_mmmlu_answer_generation_high_school_macroeconomics.json

regarding instructions: It would be helpful if the instructions clarified whether there is one specific right answer or many possible right answers. Also, it would be helpful to explain if we should know the answers as common knowledge or if we should be consulting a specific source to verify our answers are correct.
regarding p examples: Although we are shown the desirable output, it would be helpful to have a further explanation why it is the desirable output. It's not completely clear if the desirable output is acceptable only because it fits the desired format or because the answer is inherently correct.
regarding n examples: These examples were undesirable due to the format of the output. If it is likely to occur, it would be helpful to provide a negative example where the format of the output is correct but the content is not.
regarding instructions: Instructions imply I have high school level knowledge of macroeconomics.

[ ] task706_mmmlu_answer_generation_high_school_mathematics.json

regarding instructions: The instructions are okay but if you really want more than just a one letter response to a mathematics questions then it needs more than that.

[ ] task709_mmmlu_answer_generation_high_school_psychology.json

regarding instructions: Instructions are basic, but fine.
regarding p examples: poor examples, one is psychologic but the second is regarding eyes, an actual Medical issue.
regarding n examples: poor examples, one is psychologic but the second is regarding eyes, an actual Medical issue.

[ ] task711_mmmlu_answer_generation_high_school_us_history.json

regarding instructions: Well, the examples are good, but they also seem a little like common sense. I mean, I'm assuming we're supposed to also use the writing, but it's a little difficult to find that exact information in the writing.
regarding p examples: I wish the directions were a little more specific. Like, only take your answer from the writing, or use the implied meaning/assumptions from it.
regarding n examples: Well, they also seemed kind of like common sense. I'm not sure I know how they can be improved. Maybe

[ ] task713_mmmlu_answer_generation_human_aging.json

regarding instructions: It's okay but what if someone doesn't have a clue what the topic ensues, is, etc.?
regarding p examples: Didn't see any issues.
regarding n examples: Multiple and Do is spelled wrong or in the wrong way.

[ ] task714_mmmlu_answer_generation_human_sexuality.json

regarding p examples: Is breast cancer really related to human sexuality?
regarding p examples: Not sure if breast cancer falls under the umbrella of human sexuality.

[ ] task716_mmmlu_answer_generation_jurisprudence.json

regarding instructions: Instructions are fine enough. I like that they are straightforward and simple.
regarding p examples: These are tough. I guess they are ok, but I don't go to law school so can't really be sure.
regarding n examples: Difficult and hard, I'm not certain my answers are right, I did my best though.
regarding instructions: It can be explained a little simplier
regarding n examples: These made me really understand the instructions

[ ] task718_mmmlu_answer_generation_machine_learning.json

regarding instructions: Instructions presume knowledge that most people (especially on mturk) won't have
regarding p examples: examples presume knowledge that most people (especially on mturk) won't have
regarding n examples: examples presume knowledge that most people (especially on mturk) won't have

[ ] task720_mmmlu_answer_generation_marketing.json

regarding instructions: It's not clear if there is only one correct answer or if any one of several answers might be correct. Also, the instructions don't clarify which sources can be used to verify or research correct answers.
regarding p examples: The explanation for Example 1 is a sentence fragment and seems to be missing something.
regarding n examples: If it's possible for an answer to be negative based on content instead of just formatting, then it would be helpful to include an example of this!

[ ] task722_mmmlu_answer_generation_miscellaneous.json

regarding instructions: Maybe replace 'miscellaneous' with 'random topic'

[ ] task723_mmmlu_answer_generation_moral_disputes.json

regarding instructions: The instructions are good, but they could inform us where we're supposed to reference for the answers.
regarding p examples: Fine, but could be improved by providing a source for the correct answer.
regarding n examples: Good, because it demonstrates the proper outputs via negative examples.
regarding instructions: Instructions presume knowledge that most people (especially on mturk) won't have
regarding p examples: examples presume knowledge that most people (especially on mturk) won't have
regarding n examples: examples presume knowledge that most people (especially on mturk) won't have

[ ] task724_mmmlu_answer_generation_moral_scenarios.json

regarding instructions: Simple and easy to get.
regarding p examples: I saw a typo with does/doesn't in mind
regarding n examples: I saw a typo with Do/does related phrasing
regarding n examples: They should have provided examples with an incorrect selection, rather than just instances that showed answers that didn't accord with the instructions.

[ ] task726_mmmlu_answer_generation_philosophy.json

regarding n examples: You should probably say what to actually do if you don't know

[ ] task727_mmmlu_answer_generation_prehistory.json

regarding instructions: I was able to understand it without any problems.
regarding p examples: Give a better understanding of why the output was chosen.
regarding n examples: It is understandable why they examples were negative and does not really much else explaination.

[ ] task728_mmmlu_answer_generation_professional_accounting.json

regarding p examples: Example 1 explanation is not a complete sentence.

[ ] task729_mmmlu_answer_generation_professional_law.json

regarding p examples: Examples are very long, unformatted blocks of text. Improve readability. First example doesn't give good explanation; too vague compared to the second.
regarding n examples: Examples are very long, unformatted blocks of text. Improve readability.

[ ] task730_mmmlu_answer_generation_professional_medicine.json

regarding p examples: Examples are difficult to understand without a knowledge of medicine.

[ ] task731_mmmlu_answer_generation_professional_psychology.json

regarding instructions: No issues were identified
regarding p examples: Examples were easy to understand and straightforward.
regarding n examples: Examples were easy to understand and straightforward.

[ ] task732_mmmlu_answer_generation_public_relations.json

regarding instructions: The only thing I would suggest is specifying that you can only choose one option.
regarding p examples: In Example 1, the explanation of the correct answer does not explain why it is the correct answer.  This would probably be beneficial for some to know.  Also, there is a comma after "York" before market that doesn't need to be there, I don't think.
regarding n examples: Again, maybe a better explanation of why Example 1's answer is B.

[ ] task734_mmmlu_answer_generation_sociology.json

regarding instructions: The instructions say to "select the correct option", which would be more appropriate in a bubble selection format. Since this tasks requires one to actually type the letter corresponding to the correct answer perhaps this could be reworded.
regarding p examples: The positive examples are okay though they only specifically address the information in the text, not why the format of the answers is correct.
regarding n examples: The negative examples to a good job of showing what type of inputs are incorrect.

[ ] task735_mmmlu_answer_generation_us_foreign_policy.json

regarding p examples: The word "correct" is spelled as "currect".

[ ] task736_mmmlu_answer_generation_virology.json

regarding p examples: I'm not sure if the average person is expected to know these correct answers without doing any additional research, but if they are expected to know these answers then the questions are pretty tough.
regarding n examples: ``

[ ] task738_perspectrum_classification.json

regarding p examples: claim/perspective formatting is a bit sloppy, could be improved
regarding n examples: claim/perspective formatting is a bit sloppy, could be improved

[ ] task743_eurlex_summarization.json

regarding instructions: The instructions seem simple but what you have to do is not simple at all
regarding p examples: These are just bad examples, why do you need to provide the whole article and not just examples of what is being talked about in it?  If there's a summary then you can write a headline.
regarding n examples: Same goes for this as well, writing articles for the headline requires knowledge about anything legal related and I don't have that knowledge, I'm trying to improve the structure and grammar and instructions of your HIT and these examples are just too much, keep it simpler when it needs to be and in this case it needs to be.
regarding instructions: good
regarding p examples: words can be extra
regarding n examples: it is good

[ ] task744_eurlex_classification.json

regarding instructions: Generally speaking, the instructions are difficult to understand without a background in law, but it is good that the 3 categories were elaborated on. I'd recommend formatting and color to make the text more readable.
regarding p examples: Improved formatting would be helpful.
regarding n examples: Improved formatting would be helpful.
regarding p examples: huge walls of text to read through for both + and - examples

[ ] task745_ai2_arithmetic_questions_arithmetic.json

regarding instructions: The task definition was not clear.
regarding n examples: I can't understand negative examples very easily.
regarding p examples: Example 2 has a couple typos or grammatical errors ( 'she gave' with she not capitalized; '27 seashell'). They didn't affect understanding the question though.

[ ] task747_glucose_cause_emotion_detection.json

regarding instructions: Doesn't state what to do if a nonsensical story is presented

[ ] task748_glucose_reverse_cause_event_detection.json

regarding instructions: Doesn't state what to do if a story is illogical, unrelated, difficult to understand etc
regarding instructions: Not told what to do in the event of the selected story being nonsensical or difficult to understand.
regarding instructions: Does not state what to do if the selected sentence doesn't  follow the rest of the story.

[ ] task750_aqua_multiple_choice_answering.json

regarding instructions: This is your best one yet I'd say, the instructions are simple but clear and easily understood, and it shines through into the examples.
regarding instructions: Use instructions that a layperson who's not a math genius can understand.
regarding p examples: I'm terrible at math and had a difficult time following the variables.
regarding n examples: I did not understand what made the incorrect examples wrong.

[ ] task751_svamp_subtraction_question_answering.json

regarding instructions: The instructions are phrased a bit too formally, and could use pauses or sentence breaks to help make them understandable. The phrase "apply subtraction mathematical operator on the numbers embedded in the text" is missing a "the" after "apply", but instead probably needs to be rephrased to something simpler such as "use subtraction".
regarding p examples: There's a typo in example 1, Sam is, not has, 6 feet tall. <newline>In example 2, usually, the book series title would be capitalized unless it is styled with lowercase letters. <newline>In example 3, there is a small typo, a space in "don't".

[ ] task752_svamp_multiplication_question_answering.json

regarding instructions: This is kind of confusing, "multiplicative mathematical operator on the numbers" , You could give some quick examples.
regarding p examples: You could give a general idea what you want such as "keep the sample exactly like this format" or "keep the explanation as short as possible".
regarding n examples: You could add a reason at the very bottom why specifically the example is bad.
regarding instructions: The instructions make it clear that the product will involve multiplication, but not that there might be other steps involved (like division).
regarding p examples: ``
regarding n examples: Example 1 uses division, yet the instructions only reference multiplication
regarding instructions: The instructions only mention multiplication; they don't mention that some of the problems might involve other mathematical operations (like division)
regarding n examples: Q1 partially relies on division

[ ] task753_svamp_addition_question_answering.json

regarding instructions: "Mathematical operator" could probably be simplified

[ ] task754_svamp_common-division_question_answering.json

regarding instructions: The instructions themselves did not provide much information on the task expectation, apart from critiquing the instructions.
regarding n examples: The negative example explanations could have provided a bit more detail.

[ ] task827_copa_commonsense_reasoning.json

regarding n examples: Could use a few more examples.

[ ] task828_copa_commonsense_cause_effect.json

regarding n examples: Example 2 is a little less than emphatic, it only says that the provided answer is wrong, it doesn't say that the other answer (cause) would be correct in this case.
regarding instructions: Had to google what newline character is. Simplify it to linebreak.

[ ] task835_mathdataset_answer_generation.json

regarding instructions: I am not very strong in math abilities so I had a hard time following the instructions.
regarding p examples: I couldn't follow the variables since I found the math confusing.
regarding n examples: It was hard for me to see what made the negative examples wrong.

[ ] task843_financial_phrasebank_classification.json

regarding instructions: I didn't see any errors or typos, wierd phrasing, etc.
regarding p examples: I find this to be self explanatory, etc. so it's good
regarding n examples: I love the way this was all written
regarding n examples: The term 'operating profit' is spelled 'operatinng profit'.

[ ] task844_financial_phrasebank_classification.json

regarding instructions: Doesn't explain what polarity is in the context of the instructions

[ ] task846_pubmedqa_classification.json

regarding p examples: both + and - examples imply I am a doctor or scientist. difficult to understand.

[ ] task849_pubmedqa_answer_generation.json

regarding instructions: Instructions are too vague and provide no clear information on how to complete the task.
regarding p examples: Virtually impossible to understand, I would have returned the hit if not for the initial debriefing explaining that this was an instruction rating, moreso than output generation.
regarding n examples: Same as previous field, the examples cannot be understood without prior understanding of the subject matter or better instructions.

[ ] task850_synthetic_longest_palindrome.json

regarding instructions: "If the shortest possible palindrome is length 1 you should return the first character."  This could be reworded to something like "If the shortest palindrome is only the length of one character, then write just that character.  As this part was strange to understand and I only understood by reading through the examples.

@yeganehkordi I pick 600 - 700.

Feedback re. tasks 850-1200:

[ ] task1087_two_number_sum.json

regarding instructions: pretty simple and straightforward

[ ] task1088_array_of_products.json

regarding instructions: Absolutely does not make sense in any way until you see the examples and not even then<newline><newline>Too much calculator work - I wouldn't do this HIT unless the pay is stellar and if there are only 2-3 numbers to multiply. The examples below are TOO LONG and I'm only doing them out of spite because I've already wasted enough time typing all of this.
regarding p examples:  answer[i] is eqaul to the product of all the elements of nums except nums[i] where i is the position of the element in the array, otherwise it is undefined at this point<newline><newline>no... you have to treat examples like we're stupid. You have to go through the actual mathematical steps to show how 840, etc., was reached. These are the ONLY EXPLANATIONS WE WILL GET so you HAVE to be comprehensive.
regarding n examples: see above.
regarding instructions: math is hard

[ ] task1135_xcsr_en_commonsense_mc_classification.json

regarding n examples: "finishing was whaat"
regarding p examples: the explanations/output were not in line with common sense answers and need to be improved

[ ] task1148_maximum_ascii_value.json

regarding instructions: there needs to be a chart of what ascii values are
regarding p examples: how do we know with nothing to go by
regarding n examples: no way to know with a list of the values

[ ] task1150_delete_max_min.json

regarding instructions: no clue what the max & min elements are
regarding p examples: it explains/shows what the max & min are
regarding n examples: it explains the max/min and order well

[ ] task1152_bard_analogical_reasoning_causation.json

regarding instructions: I'm assuming everyone knows what analogies are. The instructions are fine.
regarding p examples: Is there only one word you're looking for? This seems like an ambiguous task - these are more synonyms than anything. Do we have to write an explanation?
regarding n examples: Is there only one word you're looking for? Do we have to write an explanation? These are more synonyms than analogies.
regarding instructions: instructions make sense, no issues
regarding p examples: examples give good explanation on what to do
regarding n examples: examples are good at showing what is not expected, complements positive examples

[ ] task1153_bard_analogical_reasoning_affordance.json

regarding instructions: examples make sense to show what is wanted
regarding n examples: bad examples are obvious on what doesn't work
regarding instructions: doesn't explain what an affordance is
regarding p examples: some of the words could have many things, so not sure what all is acceptable
regarding n examples: could have more and diverse examples

[ ] task1154_bard_analogical_reasoning_travel.json

regarding p examples: Explanation in all the three positive examples should use the word "output" instead of "answer" in the last sentence to make it more clear.

[ ] task1155_bard_analogical_reasoning_trash_or_treasure.json

regarding instructions: a bit convoluted
regarding p examples: seems like some things could be subjective and that's not covered
regarding n examples: subjectivity not discussed

[ ] task1158_bard_analogical_reasoning_manipulating_items.json

regarding instructions: not clearly defined some guesswork involved

[ ] task1159_bard_analogical_reasoning_containers.json

regarding instructions: IT WAS GOOD.
regarding p examples: IT WAS VERY CLEAR.
regarding n examples: GIVE THE CORRECT WORD ALSO WITH THE EXAMPLE.

[ ] task1186_nne_hrngo_classification.json

regarding instructions: It's missing an ending punctuation mark.
regarding p examples: Your HTML/Markdown formatting is messed up.<newline><newline>All of these numbers and explanations are totally opinion. Like you can justify any number. <newline><newline>Are we also expected to provide an example?
regarding n examples: Your HTML/Markdown formatting is messed up.<newline><newline>See above
regarding p examples: the example do not detail what to do on odd typos, like the "-s"  used in input task 1. do we ignore issues like that or make it part of the evaluation

[ ] task1189_check_char_in_string.json

regarding n examples: Explanation for negative example 1 is wrong because the output is correct as I is present in the string.
regarding instructions: The comma MUST be offset by a space, otherwise the desired character is VERY HARD TO SEE<newline><newline>In fact why can't you put the desired character on a separate line? Like, why are you making this deliberately difficult for workers?
regarding p examples: n/a
regarding n examples: n/a

[ ] task1190_add_integer_to_list.json

regarding instructions: explained well, no issues

[ ] task1191_food_veg_nonveg.json

regarding instructions: unless you're well versed in Indian food or Hindi, the instructions are useless
regarding p examples: pork is obvious, but the other dish is impossible without google unless you're Indian
regarding n examples: not possible to do without googling dishes or words
regarding instructions: This requires too much outside research. At the very least you should pre-program a google search with the term of the dish.
regarding p examples: I mean it's pretty straight forward, although it's not clear whether we need to also provide a written explanation.
regarding n examples: I mean it's pretty straight forward, although it's not clear whether we need to also provide a written explanation.

[ ] task1192_food_flavor_profile.json

regarding p examples: include the ingredients of the flavor profile

[ ] task1195_disflqa_disfluent_to_fluent_conversion.json

regarding instructions: It is hard but doable to define the objective answer.

[ ] task856_conv_ai_2_classification.json

regarding p examples: Markdown formatting isn't working.
regarding n examples: Markdown formatting isn't working.
regarding instructions: instructions clear, makes sense

[ ] task862_asdiv_multidiv_question_answering.json

regarding p examples: Example 1 is kind of weird, how is someone going to use a quarter of a paper bag? Doesn't make sense.
regarding p examples: explains what is correct well

[ ] task863_asdiv_multiop_question_answering.json

regarding n examples: Example 2 has some grammatical errors in the explanation.

[ ] task866_mawps_multidiv_question_answering.json

regarding instructions: Nothing, it's very clear
regarding p examples: They're great
regarding n examples: They're great

[ ] task868_mawps_singleop_question_answering.json

regarding instructions: I don't understand the difference between a good and a bad answer. Some kind of text explanation would be helpful, not just examples. You're just making us guess if you don't spell it out.
regarding p examples: There should not be a space before punctuation marks.
regarding n examples: There should not be a space before punctuation marks.<newline>You are missing a "the" before "correct answer".

[ ] task892_gap_reverse_coreference_resolution.json

regarding instructions: Explanations in Example 1 of positive examples and both examples in negative examples are confusing and incorrect.
regarding p examples: Output in example 1 should be "he" because the pronoun is to replace "Thomas" after a comma.
regarding n examples: Negative example 1 should be correct as "he" appears in the last sentence, which should refer to "Fritzsch".<newline>Negative example 2 should be correct as "he" referring to Elias appears in the first sentence.

[ ] task927_yelp_negative_to_positive_style_transfer.json

regarding p examples: Could have explained a little more about changes. For example in ex. 2, there were two changes, not just the one.
regarding n examples: Generally fine but could have another couple of examples to really drive the point home.
regarding instructions: I think overall it these are solid. I think that it is easy to read.
regarding p examples: The examples are basically perfect. Shows exactly what they want.

[ ] task928_yelp_positive_to_negative_style_transfer.json

regarding n examples: The negative examples are a little more jumbled but still good.

[ ] task933_wiki_auto_style_transfer.json

regarding instructions: I think instructions about parentheses should be added because some people might just omit these clauses automatically.
regarding p examples: I think that for Example 1's output, there were a few spacing typos before the commas which should be removed.
regarding n examples: I think providing an example with a longer sentence would be helpful.

[ ] task936_defeasible_nli_snli_classification.json

regarding instructions: no issues, they detail what is wanted
regarding p examples: no issues, details what was wanted well
regarding n examples: they explain what isn't wanted well, no issues with understanding what is negative
regarding instructions: So... you literally just want us to say 'strengthener' or 'weakener'? That's the entire HIT? I'm confused.
regarding p examples: The HTML/Markdown formatting appears to be broken here. Missing periods at the ends of sentences.
regarding n examples: The real HITs are going to be more ambiguous if past work from this requester is any indication. Are you ABSOLUTELY SURE this is the only way we can mess up?<newline><newline>Okay I've just done the examples. This makes NO SENSE. How can eating out of a takeout container either strengthen or weaken the scenario? Couldn't you say either? What does that even mean to weaken/strengthen the situation????

[ ] task938_copa_hi_commonsense_reasoning.json

regarding instructions: The instructions were clear
regarding p examples: I do not think that these are good examples
regarding n examples: No, I cannot understand this

[ ] task939_copa_hi_commonsense_cause_effect.json

regarding p examples: There seems to be no relation between the input and the output which renders the examples useless
regarding n examples: There seems to be no relation between the input and the output which renders the examples useless

[ ] task941_copa_gu_commonsense_cause_effect.json

regarding instructions: I do not know the language that they are asking about but the instructions themselves are pretty clear.
regarding p examples: No idea what these examples are referring to
regarding n examples: These are not clear at all.

[ ] task942_copa_mr_commonsense_reasoning.json

regarding instructions: Makes connections between mathematical concepts, between concepts and procedures, or between concepts, procedures, and application. Prompts cognitive effort. Is problem-based, authentic, or interesting.
regarding p examples: Being happy even when you have little. Having a good time even when you are losing.
regarding n examples: While employers may not love the idea of having an employee who is preoccupied with the finer points, a candidate who assures quality and strives for balance can be a great asset.

[ ] task945_wiki_cloze_bn_multiple_choice_question_answering.json

regarding instructions: pretty simple and straightforward
regarding p examples: There seems to be no relation between the input and the output which renders the examples useless
regarding n examples: There seems to be no relation between the input and the output which renders the examples useless

[ ] task946_wiki_cloze_gu_multiple_choice_question_answering.json

regarding instructions: Makes connections between mathematical concepts, between concepts and procedures, or between concepts, procedures, and application. Prompts cognitive effort. Is problem-based, authentic, or interesting.
regarding p examples: Being happy even when you have little. Having a good time even when you are losing.
regarding n examples: Common strengths include independence, persistence, creativity, and ingenuity. Common weaknesses include procrastination, impatience, impulsiveness, and forgetfulness.

[ ] task947_wiki_cloze_hi_multiple_choice_question_answering.json

regarding p examples: the relation between the input and the output is not clearly explained (in fact the output is blank) and thus the explanation is worthless
regarding n examples: the relation between the input and the output is not clearly explained (in fact the output is blank) and thus the explanation is worthless
regarding instructions: Makes connections between mathematical concepts, between concepts and procedures, or between concepts, procedures, and application. Prompts cognitive effort. Is problem-based, authentic, or interesting.
regarding p examples: Being happy even when you have little. Having a good time even when you are losing.
regarding n examples: While employers may not love the idea of having an employee who is preoccupied with the finer points, a candidate who assures quality and strives for balance can be a great asset.

[ ] task948_wiki_cloze_kn_multiple_choice_question_answering.json

regarding instructions: The Instructions said that I am given a statement written in Kannada, but none of the inputs in the examples is in Kannada. I have no idea what is happening here.
regarding p examples: None of the inputs in the positive examples is written in Kannada, which is contradictory to the instructions given at the beginning.
regarding n examples: None of the inputs in the negative examples is written in Kannada, which is contradictory to the instructions given at the beginning.

[ ] task951_wiki_cloze_or_multiple_choice_question_answering.json

regarding p examples: there is no output.  therefore the explanation is meaningless and the example is worthless.
regarding n examples: there is no output.  therefore the explanation is meaningless and the example is worthless.

[ ] task954_wiki_cloze_te_multiple_choice_question_answering.json

regarding p examples: there is no output.  therefore the explanation is meaningless and the example is worthless.
regarding n examples: there is no output.  therefore the explanation is meaningless and the example is worthless.

[ ] task958_e2e_nlg_text_generation_parse.json

regarding instructions: the formatting for the instructions is a mess.  The explanation on the ordering of items needs to be separated from the rest of the paragraph for ease of reading

[ ] task959_e2e_nlg_text_generation_identify.json

regarding instructions: simple and straightforward
regarding p examples: covers bases adequately
regarding n examples: covers bases adequately
regarding instructions: The instructions are a little wordy, but it still tells me everything that I need to knw

Feedback re. tasks 850-1200:

[ ] task1087_two_number_sum.json

regarding instructions: pretty simple and straightforward

[ ] task1088_array_of_products.json

regarding instructions: Absolutely does not make sense in any way until you see the examples and not even then<newline><newline>Too much calculator work - I wouldn't do this HIT unless the pay is stellar and if there are only 2-3 numbers to multiply. The examples below are TOO LONG and I'm only doing them out of spite because I've already wasted enough time typing all of this.
regarding p examples:  answer[i] is eqaul to the product of all the elements of nums except nums[i] where i is the position of the element in the array, otherwise it is undefined at this point<newline><newline>no... you have to treat examples like we're stupid. You have to go through the actual mathematical steps to show how 840, etc., was reached. These are the ONLY EXPLANATIONS WE WILL GET so you HAVE to be comprehensive.
regarding n examples: see above.
regarding instructions: math is hard

[ ] task1135_xcsr_en_commonsense_mc_classification.json

regarding n examples: "finishing was whaat"
regarding p examples: the explanations/output were not in line with common sense answers and need to be improved

[ ] task1148_maximum_ascii_value.json

regarding instructions: there needs to be a chart of what ascii values are
regarding p examples: how do we know with nothing to go by
regarding n examples: no way to know with a list of the values

[ ] task1150_delete_max_min.json

regarding instructions: no clue what the max & min elements are
regarding p examples: it explains/shows what the max & min are
regarding n examples: it explains the max/min and order well

[ ] task1152_bard_analogical_reasoning_causation.json

regarding instructions: I'm assuming everyone knows what analogies are. The instructions are fine.
regarding p examples: Is there only one word you're looking for? This seems like an ambiguous task - these are more synonyms than anything. Do we have to write an explanation?
regarding n examples: Is there only one word you're looking for? Do we have to write an explanation? These are more synonyms than analogies.
regarding instructions: instructions make sense, no issues
regarding p examples: examples give good explanation on what to do
regarding n examples: examples are good at showing what is not expected, complements positive examples

[ ] task1153_bard_analogical_reasoning_affordance.json

regarding instructions: examples make sense to show what is wanted
regarding n examples: bad examples are obvious on what doesn't work
regarding instructions: doesn't explain what an affordance is
regarding p examples: some of the words could have many things, so not sure what all is acceptable
regarding n examples: could have more and diverse examples

[ ] task1154_bard_analogical_reasoning_travel.json

regarding p examples: Explanation in all the three positive examples should use the word "output" instead of "answer" in the last sentence to make it more clear.

[ ] task1155_bard_analogical_reasoning_trash_or_treasure.json

regarding instructions: a bit convoluted
regarding p examples: seems like some things could be subjective and that's not covered
regarding n examples: subjectivity not discussed

[ ] task1158_bard_analogical_reasoning_manipulating_items.json

regarding instructions: not clearly defined some guesswork involved

[ ] task1159_bard_analogical_reasoning_containers.json

regarding instructions: IT WAS GOOD.
regarding p examples: IT WAS VERY CLEAR.
regarding n examples: GIVE THE CORRECT WORD ALSO WITH THE EXAMPLE.

[ ] task1186_nne_hrngo_classification.json

regarding instructions: It's missing an ending punctuation mark.
regarding p examples: Your HTML/Markdown formatting is messed up.<newline><newline>All of these numbers and explanations are totally opinion. Like you can justify any number. <newline><newline>Are we also expected to provide an example?
regarding n examples: Your HTML/Markdown formatting is messed up.<newline><newline>See above
regarding p examples: the example do not detail what to do on odd typos, like the "-s"  used in input task 1. do we ignore issues like that or make it part of the evaluation

[ ] task1189_check_char_in_string.json

regarding n examples: Explanation for negative example 1 is wrong because the output is correct as I is present in the string.
regarding instructions: The comma MUST be offset by a space, otherwise the desired character is VERY HARD TO SEE<newline><newline>In fact why can't you put the desired character on a separate line? Like, why are you making this deliberately difficult for workers?
regarding p examples: n/a
regarding n examples: n/a

[ ] task1190_add_integer_to_list.json

regarding instructions: explained well, no issues

[ ] task1191_food_veg_nonveg.json

regarding instructions: unless you're well versed in Indian food or Hindi, the instructions are useless
regarding p examples: pork is obvious, but the other dish is impossible without google unless you're Indian
regarding n examples: not possible to do without googling dishes or words
regarding instructions: This requires too much outside research. At the very least you should pre-program a google search with the term of the dish.
regarding p examples: I mean it's pretty straight forward, although it's not clear whether we need to also provide a written explanation.
regarding n examples: I mean it's pretty straight forward, although it's not clear whether we need to also provide a written explanation.

[ ] task1192_food_flavor_profile.json

regarding p examples: include the ingredients of the flavor profile

[ ] task1195_disflqa_disfluent_to_fluent_conversion.json

regarding instructions: It is hard but doable to define the objective answer.

[ ] task856_conv_ai_2_classification.json

regarding p examples: Markdown formatting isn't working.
regarding n examples: Markdown formatting isn't working.
regarding instructions: instructions clear, makes sense

[ ] task862_asdiv_multidiv_question_answering.json

regarding p examples: Example 1 is kind of weird, how is someone going to use a quarter of a paper bag? Doesn't make sense.
regarding p examples: explains what is correct well

[ ] task863_asdiv_multiop_question_answering.json

regarding n examples: Example 2 has some grammatical errors in the explanation.

[ ] task866_mawps_multidiv_question_answering.json

regarding instructions: Nothing, it's very clear
regarding p examples: They're great
regarding n examples: They're great

[ ] task868_mawps_singleop_question_answering.json

regarding instructions: I don't understand the difference between a good and a bad answer. Some kind of text explanation would be helpful, not just examples. You're just making us guess if you don't spell it out.
regarding p examples: There should not be a space before punctuation marks.
regarding n examples: There should not be a space before punctuation marks.<newline>You are missing a "the" before "correct answer".

[ ] task892_gap_reverse_coreference_resolution.json

regarding instructions: Explanations in Example 1 of positive examples and both examples in negative examples are confusing and incorrect.
regarding p examples: Output in example 1 should be "he" because the pronoun is to replace "Thomas" after a comma.
regarding n examples: Negative example 1 should be correct as "he" appears in the last sentence, which should refer to "Fritzsch".<newline>Negative example 2 should be correct as "he" referring to Elias appears in the first sentence.

[ ] task927_yelp_negative_to_positive_style_transfer.json

regarding p examples: Could have explained a little more about changes. For example in ex. 2, there were two changes, not just the one.
regarding n examples: Generally fine but could have another couple of examples to really drive the point home.
regarding instructions: I think overall it these are solid. I think that it is easy to read.
regarding p examples: The examples are basically perfect. Shows exactly what they want.

[ ] task928_yelp_positive_to_negative_style_transfer.json

regarding n examples: The negative examples are a little more jumbled but still good.

[ ] task933_wiki_auto_style_transfer.json

regarding instructions: I think instructions about parentheses should be added because some people might just omit these clauses automatically.
regarding p examples: I think that for Example 1's output, there were a few spacing typos before the commas which should be removed.
regarding n examples: I think providing an example with a longer sentence would be helpful.

[ ] task936_defeasible_nli_snli_classification.json

regarding instructions: no issues, they detail what is wanted
regarding p examples: no issues, details what was wanted well
regarding n examples: they explain what isn't wanted well, no issues with understanding what is negative
regarding instructions: So... you literally just want us to say 'strengthener' or 'weakener'? That's the entire HIT? I'm confused.
regarding p examples: The HTML/Markdown formatting appears to be broken here. Missing periods at the ends of sentences.
regarding n examples: The real HITs are going to be more ambiguous if past work from this requester is any indication. Are you ABSOLUTELY SURE this is the only way we can mess up?<newline><newline>Okay I've just done the examples. This makes NO SENSE. How can eating out of a takeout container either strengthen or weaken the scenario? Couldn't you say either? What does that even mean to weaken/strengthen the situation????

[ ] task938_copa_hi_commonsense_reasoning.json

regarding instructions: The instructions were clear
regarding p examples: I do not think that these are good examples
regarding n examples: No, I cannot understand this

[ ] task939_copa_hi_commonsense_cause_effect.json

regarding p examples: There seems to be no relation between the input and the output which renders the examples useless
regarding n examples: There seems to be no relation between the input and the output which renders the examples useless

[ ] task941_copa_gu_commonsense_cause_effect.json

regarding instructions: I do not know the language that they are asking about but the instructions themselves are pretty clear.
regarding p examples: No idea what these examples are referring to
regarding n examples: These are not clear at all.

[ ] task942_copa_mr_commonsense_reasoning.json

regarding instructions: Makes connections between mathematical concepts, between concepts and procedures, or between concepts, procedures, and application. Prompts cognitive effort. Is problem-based, authentic, or interesting.
regarding p examples: Being happy even when you have little. Having a good time even when you are losing.
regarding n examples: While employers may not love the idea of having an employee who is preoccupied with the finer points, a candidate who assures quality and strives for balance can be a great asset.

[ ] task945_wiki_cloze_bn_multiple_choice_question_answering.json

regarding instructions: pretty simple and straightforward
regarding p examples: There seems to be no relation between the input and the output which renders the examples useless
regarding n examples: There seems to be no relation between the input and the output which renders the examples useless

[ ] task946_wiki_cloze_gu_multiple_choice_question_answering.json

regarding instructions: Makes connections between mathematical concepts, between concepts and procedures, or between concepts, procedures, and application. Prompts cognitive effort. Is problem-based, authentic, or interesting.
regarding p examples: Being happy even when you have little. Having a good time even when you are losing.
regarding n examples: Common strengths include independence, persistence, creativity, and ingenuity. Common weaknesses include procrastination, impatience, impulsiveness, and forgetfulness.

[ ] task947_wiki_cloze_hi_multiple_choice_question_answering.json

regarding p examples: the relation between the input and the output is not clearly explained (in fact the output is blank) and thus the explanation is worthless
regarding n examples: the relation between the input and the output is not clearly explained (in fact the output is blank) and thus the explanation is worthless
regarding instructions: Makes connections between mathematical concepts, between concepts and procedures, or between concepts, procedures, and application. Prompts cognitive effort. Is problem-based, authentic, or interesting.
regarding p examples: Being happy even when you have little. Having a good time even when you are losing.
regarding n examples: While employers may not love the idea of having an employee who is preoccupied with the finer points, a candidate who assures quality and strives for balance can be a great asset.

[ ] task948_wiki_cloze_kn_multiple_choice_question_answering.json

regarding instructions: The Instructions said that I am given a statement written in Kannada, but none of the inputs in the examples is in Kannada. I have no idea what is happening here.
regarding p examples: None of the inputs in the positive examples is written in Kannada, which is contradictory to the instructions given at the beginning.
regarding n examples: None of the inputs in the negative examples is written in Kannada, which is contradictory to the instructions given at the beginning.

[ ] task951_wiki_cloze_or_multiple_choice_question_answering.json

regarding p examples: there is no output.  therefore the explanation is meaningless and the example is worthless.
regarding n examples: there is no output.  therefore the explanation is meaningless and the example is worthless.

[ ] task954_wiki_cloze_te_multiple_choice_question_answering.json

regarding p examples: there is no output.  therefore the explanation is meaningless and the example is worthless.
regarding n examples: there is no output.  therefore the explanation is meaningless and the example is worthless.

[ ] task958_e2e_nlg_text_generation_parse.json