Visual question-answering tasks

To capture “question makes sense”, we explained to the workers (and conducted qualification tests to make sure that they understood) that any premise assumed in the question must hold true for the image they select. For instance,the question “What is the woman doing?” assumes that a woman is present and can be seen in the image. It does not make sense to ask this question on an image without awoman visible in it