Recently the Interkula has received a few requests to have access to the full questionnaire data after publishing one of our reports on Parenting in Triratna. In part, this seemed to be due to a concern that without the full data, readers could not be sure that bias had not been introduced into the report by the InterKula. We hope that this post addresses such concerns.
We are a small group of people who are doing this work in our spare time with few resources. We also do not have access to address lists for the whole Sangha, so have to make do with reaching as many relevant people as possible through the internet. What is most important is that any potential biases are stated in the report so that people can read it bearing these in mind, which we have done.
RAW DATA
It is not good research practice to share raw data. Having collected it, the data ‘belongs’ to the Interkula. Great care has been taken in writing reports so that individuals cannot be identified by their free text responses, and some did not give permission for their responses to be shared in a report, which we have complied with.
BIAS
Some are concerned that our questionnaire or reporting is ‘biased’ but no-one has specified the type of bias they are thinking of. We will address some of the biases we consider relevant here.
QUESTIONNAIRE
We have started with certain hypotheses e.g. with the Parenting survey, that in the past, there have been problems with parents feeling welcome. Therefore, the questionnaire assumes there has/ is a problem in this arena, but allows for answers that take in the whole spectrum of views, including that there is no problem. Similarly, with the latest questionnaire, we are starting from the hypothesis or assumption that there has been sexual misconduct in the past, and we were wanting to enquire as to how people felt the restorative process was going, and what else, if anything, people felt would be helpful to address this issue in our community. Another group of people may have had different hypotheses and thus come up with different questions. We hope the range of questions with free text boxes has enabled everyone who has taken the questionnaires to be able to put their points of view across.
We have mostly designed the questions on a Likert scale (from 1-5), putting questions in the negative and positive which is a standard and trialled data collection method used to reduce response bias. We have minimised social desirability bias (people giving you the answers they think you want to here) by these being self- completed, anonymous questionnaires. We have piloted the questions prior to the surveys being released.
SAMPLE SIZE
We have had more than one research question within each questionnaire, and thus have not done sample size calculations. If we did only have one question, there would still be little point in doing sample calculations as we are unable to sample participants randomly (see below).
BIAS
QUANTITATIVE/ NUMBER RESULTS
If we were to aim to get as unbiased a sample as possible, then we would need to:
a) Decide on our population: for example, if we were to do a questionnaire around racism in the Order, it is likely we would target people of colour. With each questionnaire, so far, we have posted links to the surveys on the most relevant internet sites.
b) List the people in the population, and choose them at random.
c) Ask only those people to complete the questionnaire.
As we felt unable to use official address lists for ethical reasons to do with data protection, we have had to go via internet groups such as various Facebook sites and the Buddhist centre online to attempt to reach people. This introduces more selection bias that if we sent this out via centre and order address lists, but even if we had done this, there would still be the following biases:
a) Very few people respond to online questionnaires; the characteristics of those who respond are different to those who do not, and these are unknown
b) People are more likely to respond if they have an interest in the topic
c) Any online survey from us or via a larger address list will favour those on the internet, who are not visually impaired, speak English and are literate.
d) Some people are not on any address lists or their mail addresses are not updated
We are fully aware, for example, that should we wish to reach the Indian Sangha, then we would need to use a different method, such as telephone surveys.
FREE TEXT/ QUALITATIVE RESULTS
The aim of reporting these is to show the RANGE of responses, and here sample size is irrelevant; what is relevant is continuing the questionnaire until no new themes arise. Usually up to 40 responses is plenty for this, and we aim to report this full range, noting what is both typical and atypical in response.
RECOMMENDATIONS
No recommendations were made in the Parenting survey, as the intention was to share the results and for people to use this as a starting-off point for their own discussions. If any recommendations are made in future reports, these will be based on the results from that survey, not personal opinions of members of the Interkula.
