A systematic review of studies evaluating Australian indigenous community development projects: the extent of community participation, their methodological quality and their outcomes

Thirty-one studies evaluating Indigenous community development projects in Australia
were identified. Ten (32 %) were published in the peer reviewed literature 39]–48].

Extent of community participation

Table 2 summarises the level of community participation across the four phases of project
development for each study. Table 3 summarises the number of studies relevant to each of the seven levels of community
participation, separately for the four phases of project development. The highest
levels of participation (level 5 to level 7), were found in the Diagnosis phase for
ten studies (32 %) 39], 43], 44], 48]–54], in the Development phase for 13 studies (42 %) 39]–44], 46], 49]–52], 54], 55], in the Implementation phase for 17 studies (55 %) 39]–44], 46], 48]–54], 56]–58] and in the Evaluation phase for 7 studies (22 %) 39], 40], 43], 44], 50], 53], 59]. Four studies (13 %) had at least level 5 participation in all phases of the project
39], 43], 44], 50]. The participation of the community was described with insufficient detail to be
assessed (unknown category) for ten studies in the Diagnosis phase (32 %) 42], 45], 57], 58], 60]–65], seven in the Development phase (23 %) 47], 48], 57], 60], 61], 64], 66], four (12.9 %) in the Implementation phase 47], 61], 63], 66] and two (7 %) in the Evaluation phase 46], 67].

Table 2. Level of community participation in each phase of project development for each study

Table 3. Number of studies across the levels of community participation and phases of project
development

Methods used in studies

Twenty-one studies (67 %) used qualitative methods only 39]–41], 48], 49], 51], 52], 55]–57], 59]–69], two (7 %) used quantitative methods only 46], 47], and eight (26 %) used mixed methods 42]–45], 53], 54], 58], 59]. Qualitative data were collected using semi-structured interviews in 24 studies 39], 40], 42], 43], 45], 49]–53], 55]–58], 60]–69], document analysis (n?=?15 42], 49], 51], 52], 55]–57], 60]–66], 68]), focus groups (n?=?9 39], 40], 43], 45], 48]–50], 53], 58]), participant observation (n?=?6 42], 43], 53], 54], 58], 69]) and photovoice 70] (n?=?2 50], 53]). Quantitative data collection methods included surveys in three studies 45], 48], 58], hospital/clinical records (n?=?4 43], 44], 47], 53]), school records (n?=?2 42], 58]), police records (n?=?1 42]), store records (n?=?1 43]) and ABS census data (n?=?1 46]).

Methodological quality of studies with a qualitative component

All 29 studies with a qualitative component (including mixed methods studies) provided
some description of the evaluation methods used (Table 4). Twelve studies (41 %) gave detailed descriptions of the data collection process,
including participant recruitment, focus group procedures and a clear description
of which data were recorded 39], 41], 49]–51], 53], 58], 60]–63], 67]. Four of these twelve studies (14 %) provided the interview questions 51], 58], 60], 67] and one study (4 %) described in detail how the data collection methods were tailored
to ensure their cultural appropriateness 49]. The data analysis methods were described in detail in seven studies (24 %) 39], 42], 50], 54], 58], 67], 69]. The potential for researcher bias was described in seven studies (24 %) 39], 44], 45], 49], 53], 58], 69]. Three studies (10 %) did not discuss the implications of their findings 52], 59], 65].

Table 4. Critical appraisal of qualitative components of studies evaluating Indigenous community
development projects (n?=?29)

Methodological quality of studies with a quantitative component

The summary ratings for all ten studies with a quantitative component were classified
as weak (Table 5). The likely extent of selection bias was unclear for six studies (60 %) because
description of the participant and community selection procedures was absent or insufficiently
detailed 42]–44], 46], 47], 59]. Five studies (50 %) used a cohort design without a control group 43]–45], 53], 54], one study (10 %) used a time series design 47] and the evaluation design of the remaining four studies (40 %) was unclear 42], 46], 58], 59]. No study adequately controlled for confounding variables. None of the studies used
blinding procedures. Two studies (20 %) used validated outcome measures 45], 58]. No study discussed the validity or reliability of their outcome measures.

Table 5. Critical appraisal of quantitative component of studies evaluating Indigenous community
development projects (n?=?10)

One study (10 %) described withdrawals and drop outs at the community level 46]. One study (10 %) described withdrawals and drop outs at the participant level 53]. Drop outs at the participant level were not applicable for the six studies (60 %)
that used either routinely collected data or a one-off survey 42]–44], 47], 54], 58]. Two studies (20 %) did not report drop outs 45], 59]. No study described the fidelity of the project. Three studies (30 %) reported on
the exposure of participants to the project 45], 54], 58]. Inferential statistical analyses were reported by four studies (40 %) 42], 47], 53], 58].

Outcomes

A summary of the aims and key outcomes for each study is provided in Table 6. All studies using qualitative methods concluded that community members reported
positive project impacts for their community. Two studies (7 %) reported quantitative
outcomes that were statistically significant: a reduction in injuries 47] and a reduction in cannabis use among females aged 13–36 and males aged over 16 years
42].

Table 6. Aims and outcomes of studies evaluating Indigenous community development project (n?=?31)