Working safely for your community
Appendix 1 - Methodology
Design framework
The evaluation used a postal survey questionnaire of the Office of the CVS's database of voluntary organisations and associated interested parties.
Ethical considerations
The survey was anonymous except for demographic information. However, if an organisation wished to receive a hardcopy of the evaluation report, they were asked to provide contact information. The evaluation form also noted that a website version would be freely available. It was also decided to record any additional comments, including negative comments, received on the forms.
Components of the survey method plan
The survey design
The self-administered questionnaire used a mixture of closed answer questions (including an 'other' option) and a section rating statements using a Likert scale. Advantages of this survey approach included the economy of the design and the rapid turnaround in data collection, due to the lack of resources and time to undertake indepth research.
Because the Guidelines were aimed at smaller organisations, the evaluation also aimed to include responses from small to large organisations so the option of interviewing umbrella organisations was not used.
It was noted that some organisations did not like postal surveys so to incorporate a wide range of responses, plenty of space was provided for additional suggestions of assistance or comments.
Respondents were given the maximum possible time as the survey was posted before Christmas. This was also to allow for organisations that do not meet regularly.
Data collection is cross-sectional, not longitudinal, with data collected at one point in time.
The population and sample
The population was the Office of the CVS's voluntary organisation database. Certain problems with the database contents meant the response rate can only be estimated. These problems included outdated names and addresses, organisations returning forms and noting they had not received the Guidelines, and duplicate entries.
It should be noted that returns on surveys from the Community and Voluntary sector are generally low, and databases of this type may become outdated as organisations change staff and addresses, or cease to exist, often without notifying the database administrator.
However there were some positive outcomes from this mail-out. For example, there were a number of enquiries from organisations that had received the evaluation form but not the Guidelines, and other organisations requested additional copies of the Guidelines. In these cases, the evaluation form was returned in case they had any comments to contribute.
Some forms were spoiled due to comprehension issues - misunderstanding the question or not answering certain sections. There was also some miscomprehension of the term 'Full Time Equivalent', despite an explanation on the form.
Therefore, the respondent group, which numbered one hundred and fifty-seven organisations, must be regarded as the self selected 'population sample'. Although single stage sampling was used, meaning everyone had a chance of being selected, the response is biased towards who replied.
Instrumentation
Data were collected using a self administered questionnaire. These asked for factual information, using closed answer questions with a range of options, including an 'other' option, as well as long answer questions. There was also a section using Likert-items based on a scale from 'strongly agree' through to 'strongly disagree', and including a 'don't know/non-applicable' option. The questionnaire also included open ended questions where respondents were asked to comment on various aspects of the Guidelines. The survey form also asked for demographic information to provide further information on the size and type of organisations that had used the Guidelines, and the nature of their past contact with OSH. All information used in this analysis was derived from questionnaire data.
Data analysis
Data was entered on to an access database, with each respondent assigned a unique identifier. Responses were coded and any new, additional comments or responses (including those to the long answer questions) were coded. Responses were sorted by workforce mix, and then by the number of volunteers and employees, and their work area. The results were then graphed or written into a general discussion of the findings and recommendations.
Threats to validity
As discussed in 'population and sample', there were repetitions of names and addresses, out of date names and addresses, and up to four persons in the same organisation on the mailing list. This problem was discovered after some of the mail out had been completed. In addition, some respondents contacted OSH or sent back their forms, noting they had not received the Guidelines. In as many cases as possible, the survey form was resent along with a copy of the Guidelines. Therefore there was no method of calculating an accurate response rate.