Back in the dark ages, some unfortunate folk tried to teach me about Statistics, which included how statistical surveys should be conducted. Some snippets of information, perhaps misremembered or misunderstood, came back to me as I read Waterways Ireland’s press release about its research into the contribution of boating to the economy and as I attempted to complete its online questionnaire.
WI commissioned a similar survey in November 2005 (with the results being published in 2006). That survey was carried out on WI’s behalf by the Tourism Research Centre at Dublin Institute of Technology [about whose current status I can find little information]. The TRC folk posted questionnaires to the 6682 boat-owners registered with WI; 54 respondents said that the questionnaire was no longer relevant to them so the population was reduced to 6628. Owners returned 718 questionnaires; 14 were excluded as being incomplete so the 704 were analysed, a valid response rate of 11%.
Participation
The current survey is online. It is said to be “open to all boat owners on Ireland’s inland waterways” but there is no way of restricting participation. Question 1 is “Are you a Boat Owner?” It does not specify that the boat has to be on Irish inland waterways run by Waterways Ireland. However, answering No brings up a page stating, inter alia, “This survey is solely for owners of craft on the inland navigational waterways.”
There is nothing to stop truthful people who own boats on other waterways from participating. There is nothing to stop liars who don’t own boats from participating. There is nothing to stop people from participating twice, or even more often, as long as they delete cookies after each session; that makes the survey open to manipulation by evil-minded interest groups.
And, most significantly, there is no way of assessing whether the average expenditures to be reported by the survey are in any way representative of the amounts actually spent by the body of boat-owners on Irish inland waterways. All that can be said is that the reported results will represent what some people, who may or may not be owners of boats on WI’s inland waterways, said about their spending.
Initial Questions
According to the press release
The survey should take 10-12 minutes to complete.
That may be so if you happen to have all your financial records to hand and if your classification happens to coincide with that used by Waterways Ireland. It took me considerably longer than that, even though my records are all in an Excel spreadsheet. There is no provision for saving your answers and returning to complete them later.
Furthermore, I cheated: the survey asks about spending “between August 2013 to August 2014”; I had figures for y/e 31 December 2013 so I used those.
Q2 asks about gangs whereof you might be a member and Q3 about your three favourite waterways activities; I think you can get away with listing only two, but not only one. Q4 asks “Do you own your own craft outright or shared?” but does not distinguish between sharing with a spouse (or other close family member), sharing with friends and sharing via a commercial syndicate (if there are any: it would be interesting to know).
The next page has five questions: whether you bought your boat new or secondhand (Q5), when you bought it (Q6), how long (in months) you expect to keep it before buying another (Q7), the cost in euro (Q8: I don’t know whether, if you log in from NI, you can answer in sterling) and the number of days each year you spend boating (Q9).
I had difficulty with QQ6 and 7. Q6, for some reason, insists on your entering the day, month and year in which you bought the boat. I know the year, I’m pretty sure of the month but I have no idea of the day. Q7 assumes that you are going to sell your boat: there is no provision for saying “I don’t intend to [or have no plans to] sell it”. I thought I’d get around the problem by saying “ten years”, which meant entering 120, but the survey rejected that, whereas it accepted 99 (and, on later trials, accepted 100). My answer to Q7 is thus inaccurate because the question is badly designed.
That lot is more or less equivalent to QQ 1, 2, 3, 4, 9 and 10 on the 2005 survey.
One other feature of the survey is that error messaages appear in red between two questions; at first you have to guess which answer the system dislikes. And the message I got for entering too large a number for Q7 …
The comment you entered is in an invalid format.
… is not at all helpful: you have to guess what you are to do in response.
Expenditure questions
The next page has the detailed questions on expenditure; they are almost, but not quite, the same as those used in 2005. While that might allow comparison (were it not that the two surveys might be completed by non-comparable groups), some of the initial questions were so idiotic that they should have been dropped. I can’t imagine that anyone other than Ebenezer Scrooge would record a year’s spending on “Sweets/snacks” bought while on board. But there is no provision for showing that you have no records for a particular category: all you can do is record expenditure of 0.
The authors of the report on the 2005 survey said:
Another limitation is that respondents were asked to recall expenditure over a twelve-month period, which is difficult to do accurately unless precise records are kept during the reference period and this leads to figures based on estimates. One way to address this issue in future studies and to ensure more detailed record keeping in future studies is to introduce an ‘Expenditure Diary’ for Waterways Ireland registered boat owners to track their expenditure.
I suspect that would not have been welcomed but the problem has not, as far as I can see, been addressed in any other way, apart from this instruction at the top of the page:
Please detail all expenditure for the last 12 months only, please ensure all expenditure, including credit card and cash payments.Where the exact value is not known,please give as close an estimate as possible and state clearly that it is just an estimate.
However, if you try to enter anything other than a numeric value (eg “Estimate”) in any of the fields for QQ10 and 12, you get this:
Please enter a positive number.
The questions are grouped in two major categories: Regular Expenditure and Irregular Expenditure. The latter provides for the purchase of a boat but it does not provide for improvements more significant than maintenance (which is under Regular). For instance, if you happen to have added three and a half tons of ballast, there is no suitable category for recording the expenditure.
Here, in flagrant breach of WI’s copyright, but in the interests of allowing folk to prepare their answers, are the expenditure questions.
10. Regular Expenditure (August 2013- August 2014)
Regular Expenditure (August 2013- August 2014)
Mooring Rental (per annum)
Insurance
Fuel
Winter Moorings
Maintenance (to include parts,repairs,servicing,paint.etc)
Equipment (Life Saving Appliances, Fire Extinguishers, Mooring lines etc)
Annual membership fees
Smart Cards
Permits
Lock Passes
Other
11. Please specify equipment purchased/or other expenses
12. Irregular Expenditure (August 2013- August 2014)
Purchase of craft/boat ( within the last 12 months)
Training
Navigational publications – charts, guide, GPS
Hire of boat (if applicable)
Internal transport cost associated with boating
Accommodation while using the waterways
Food in bars and restaurants while using the waterways
Drink in bars and restaurants while using the waterways
Shopping(food) – on board
Shopping(drink)- on board
Newspapers/magazines
Clothing associated with boating
Sweets/snacks
Souvenirs/crafts
Other purchases associated with boating (please detail below)
13. Please specify other expenses
There is some upper limit to the values you can enter in the fields for QQ10 and 12, but I haven’t discovered what it is.
Trips
If you survive that lot, you get asked about your trips.
14. How many boating trips do you take each year?
15. Average number of days spent on each boating trip?
16. Last Boating Trip
What waterway was visited:
How much did you spend on your last boating trip on Ireland’s Inland Waterways(euro):
Number of nights spent on your last boating trip:
Number of people in your party:
Adults:
Children:
17. 2nd most recent boating trip
[same questions][
18. 3rd most recent boating trip
[same questions]
19. Which of the above 3 boating trips was most typical/most accurately reflect your spending patterns?
There is a minor difficulty in that it is assumed that the same number of people was on board throughout. What I see as a more significant problem is that, for those who take their annual holidays “on Ireland’s Inland Waterways”, at this time of year the three most recent boating trips might include (say) a fortnight’s holiday, a weekend and a bank holiday weekend. Only the non-bank-holiday weekend is likely to be typical of trips within the year, but all three might be regular annual events. It is difficult to get that across in answer to Q19.
There seems to be no data validation on that page: ridiculous numbers are accepted, as are waterways that are not run by WI.
The rest
On the next page, QQ20, 21 and 22 ask about age, gender and personal status. Then, on the last, you can (but are not forced to) give your name and email address if you want a copy of the survey results or to get WI mailings.
The value of the survey
I am all in favour of gathering and publishing information. And I appreciate both the difficulty of doing so at low cost and the desirability of maintaining some sort of comparability with the earlier survey. But it seems to me that this survey is seriously flawed: the design of the questionnaire, the restrictions on the answers and the impossibility of drawing any reliable conclusions about the body of inland boat-owners are weaknesses that undermine the value of the exercise.
Having been one of the victims interviewees for WI’s other information-gathering exercise, the Survey of Waterways Users 2010 [PDF], I had reservations about that too, but the method might offer more control and, with better questionnaire design [tested on actual owners], might be applied to the “Economic Contribution” survey as well as to the user survey.
Irish waterways history 