The Beauty of Lies has moved to KirkKittell.com (Subscribe to the RSS feed). This post has been copied to http://kirkkittell.com/2008/03/11/on-how-to-administer-bad-surveys.
As a general rule, I support the Space Generation Advisory Council (SGAC, below improperly referred to as SCAG) because it is their duty to represent the views of youth to the United Nations, which is a noble goal in my opinion. That's a very useful duty. However, it helps if you do it properly. Below is a bit of feedback from someone in the industry that was submitted to SGAC in response to their Near Earth Object survey and the more recent USA 193 shootdown survey. I would provide the links to the survey for you so you could make your own judgment, but neither are available online at their originally advertised addresses.
As a young person who works on various strategic planning projects at NASA Headquarters, might I ask why SCAG has developed a pattern of poorly executed surveys? Your results can not be used in serious policy discussions because of their methodological errors.
First, I'm basing my assessment on the two recent surveys regarding Near Earth Objects and the shoot-down of a satellite by the U.S. Navy. Both surveys claimed to attempt to assess the opinions of young people, and the NEO survey results were reported to the UN as being the opinions of young people. However, a basic understanding of the statistics behind surveys says that to accomplish your stated goal you need to survey a random sample of people. You do not have that, so you have failed at your stated goal and then misrepresented your results.
In fact, the disfunction goes deeper than this. The NEO survey was planned to "teach" people about NEOs, as per your chat forums. That is not the role of a survey, especially not one that the organization told outsiders was to assess existing opinions. You can't both assess existing opinions and try to change those opinions at the same time. Worse, the NEO and shoot-down survey questions are terribly biased and a casual observer can't help but see what opinions you want survey respondents to have.
Finally, you do not know your margin of error. All credible major surveys report how many respondents there were and the margin of error in the results. You report neither. You also have worse than a random sample. You have a sample which has been sent to your website through marketing efforts. You tell people you want them "to express themselves" or some such. You have a sample that is self-selected and by definition does not represent the wider population.
These are basic errors that have been made with multiple surveys. Please do something that achieves your real objective, such as a real survey if you want to report the views people already have, or a blog to let people vent on a topic, or a set of fact sheets if you want to give them SCAG's view on a controversial topic. I do not know which of these is SCAG's real objective, but please properly caveat the survey results that you have already collected.
For a little extra boost, surveys such as the one mentioned above look like this SGAC projects survey: link. (Actually, if you follow the original link to the NEO and ASAT surveys, this is the new destination.). My problem with this? Look at the link at the bottom to space-industry.com. This is a money-making venture for the SGAC executive director. Of course you won't get quality survey results when the purpose is to get quantity survey results that could be used to field test a product or entity that provides income for you.
