Validating and reliability testing the descriptive data and three different disease diagnoses of the internet-based DOGRISK questionnaire


Since we have 43 disease diagnoses (117 when including all drop-down lists) in the DOGRISK questionnaire, it was not feasible to validate them all. However, as we are interested in using different diseases in our future analyses, we chose to validate one medicated disease that could be evaluated easily using a concurrent medication question (hypothyroidism), one diagnosis that could be compared with an external official register (canine hip dysplasia; CHD), and one disease that even for a specialist is considered difficult to diagnose with 100 % certainty (canine atopic dermatitis or atopy), which in the questionnaire was covered by a wider question: atopy/allergy (skin symptoms)(AASS). The internal consistency for hypothyroidism was excellent (??=?0.95, ?=?8081). This was expected since hypothyroidism is also a human disease and it can be speculated that people may remember it more easily because of this. It is also always diagnosed from a blood sample at a veterinary clinic, and it invariably requires medication. Discrepant answers to the two questions on diagnosis and medication were very few, but could arise from owners having just visited the clinic and awaiting results, which would have yielded a ‘yes’ for diagnosis but nothing for medication. Another possibility might be that the owner could not remember the name of the disease or the name of the medication, resulting in one of the questions left unanswered.

The reliability of the radiographic CHD results was also excellent (??=?0.95). Many dog breed associations in Finland take part in a national hip screening program (PEVISA) [12] where dogs’ hips are screened by radiographs according to the Fédération Cynologique Internationale (FCI) [11] at the age of 12-18 months. The screening result is expressed as a letter, according to their worst hip; A (healthy), B, C, D or E (severe CHD). The small differences in owner-reported CHD results and the register data might reflect the fact that some owners take new radiographs when their animals are older to see if they have improved (or worsened). This would yield an inconsistency in the validation, as we have old results in the questionnaire but new results in the FCI register. Also, owners might simply forget the letters. We only looked at dogs that had official hip radiograph results, meaning that they were pure-bred dogs aged over 18 months.

Atopy/allergy (skin symptoms) (AASS) is much more difficult to diagnose than hypothyroidism and CHD, and therefore, we expected its reliability to be much lower than for the other two diagnoses. The diagnosing protocol is time-consuming and includes numerous treatment trials, elimination diet trials, blood work, etc. This might be too extensive and expensive for many owners. Thus, many pruritic dogs are incorrectly believed to be atopic by owners or even veterinarians, who sometimes also find it difficult to diagnose atopy and allergies. Some dogs therefore probably have the disease but lack a confirmed diagnosis. The dog’s symptoms also might have disappeared due to environmental or dietary changes, or they may be so mild that the owners think that their dog is healthy. For this validation, we sent a short email/mail questionnaire to the owners to ask about their dogs’ diagnosis. Altogether 31 % of the owners reported that the diagnosis was not verified by a veterinary work-up. However, response consistency or disease repeatability was good: the PPV was 0.91 when comparing the answers in the DOGRISK questionnaire and in the email/mail questionnaire. Therefore, we may consider the owner-reported disease status of AASS quite reliable, bearing in mind the difficulties in diagnosing this disease. Because email/mail was not sent to owners reporting that their dogs did not suffer from AASS, we could not conclude how many true- or false-negatives we had.

Owners seemed to be diligent with basic information, as the internal consistency of two questions related to the dog’s age was excellent. Also data on gender, season of birth, and breed matched very well with the official register records. Because all dogs in these analyses were pure bred dogs, the new owner should have received the registration book from the breeder or previous owner in which this information can be verified. In fact, most of our data concerning descriptive information and diseases can be considered good or excellent. Only data for the time before the owner had the dog can be considered slightly more unreliable.

Data on web-based questionnaire validity and reliability among human subjects are vast, and a comprehensive review is not feasible in this context. Although between-study comparisons are complicated by differences in the study methods and statistical measures used, the DOGRISK questionnaire appears to compare well with results for humans. The reliability is highly dependent on the exact variable analyzed, as shown by Stanton et al. [13], who found Kappa coefficients ranging from 0.65 to 1.00 for sociodemographic variables and from 0.21 to 0.71 for health variables. McAlindon et al. [14] compared consistency between answers given to age and date of birth and found a discrepancy rate of 1.3 %, similar to our figure of 1.0 % (?=?3540). Upon comparing self-reported hip osteoarthritis with results of clinical examination, Ratzlaff et al. [15] reported a positive predictive value of 61 %, a negative predictive value of 98 %, a sensitivity of 81 %, and a specificity of 94 %.

The retest population was not a planned repeatability study population, but a subgroup of the group responding to the existing questionnaire twice of their own initiative. Utilizing the responses of the owners who had answered the questionnaire twice was a limitation in our study, as all data will not necessarily be the same. Answers for gender, color of coat, season of birth, and born in owner family are considered stable since they should not change with repeated response, and repeatability in these was indeed high, being very similar to that in Sallander et al. [8].

We provided an option in the questionnaire to save answers midway through the answering process without submitting the questionnaire and to continue answering later. For those that used this option, it probably had a positive effect on the reliability of the answers, as respondents could gather all information before submitting the questionnaire and they could fill in the questionnaire at an optimal time as opposed to completing it in a rush. However, we also had informed respondents who contacted us by email that they could resubmit a totally new questionnaire at will if they received new information about the puppy stage or if conditions changed. As the two answers will then not be similar by default, they will show weaker repeatability. The most logical reasons for why some owners answered the questionnaire twice are because they had reported something incorrectly the first time, because they had forgotten that they had already filled in the questionnaire, because something had changed in their dog’s lives, because they wanted to report extra information about something they had no knowledge of the first time, or because other person in the household had responded to the questionnaire at the second time-point. The questions on body condition score and time spent outside, both covering the period when dogs were under the age of two months, could have been difficult for the owner to report since during this period puppies typically reside with the breeder. Thus, owners likely first provided an “educated guess” on the topic, followed by the actual answer given to them by the breeder, or a second own “educated guess”. This would explain the low repeatability in the answers to these questions, with answers to the question on puppy vaccination being somewhat more reliable, as it is common in Finland to vaccinate all puppies, and this information is usually given in written form (in the passport or vaccination booklet) to new owners if they adopt an older dog.

Questions about the environment were also deemed ill-suited to repeated questioning since owners might have moved, taken new pets, stopped smoking, etc., but we nevertheless chose to compare the following non-stable questions: tidiness of household, adult dog’s vaccinations, and other dogs in household. The time factor will be the reason for any discrepancy in the answers here, e.g. tidiness of the household may vary according to the life situation. Also, the number of dogs in the household may vary with time, as old dogs die and new puppies arrive. For the future analyses these variables can still be used but should be interpreted carefully.