Evaluation resources – Document analysis
This guide supports school planning and the evaluation of Strategic Improvement Plans (SIP) initiatives and implementation and progress monitoring (IPM) related to curriculum implementation. This resource includes a template schools can use to complete document analysis.
What and why
A document analysis is a systematic process of reviewing existing qualitative documentary evidence in order to answer specific evaluation questions. The types of documents reviewed will depend on the question being asked. Document analysis is rarely used on its own. Rather, it is used in combination with other data sources (such as focus groups or interviews) to substantiate the contents and use of the documents being analysed.
A document analysis allows:
- an efficient way to collect data as it requires minimal time or disruption to routines by using existing documents which are available and accessible.
- a focus on historical data or trends by using existing data available.
- a context or narrative base to be formed for quantitative data. For example, document analysis of a scope and sequence and teaching and learning program may provide information on why school-based assessment results have increased or decreased for a cohort.
Document analysis can be used:
- to determine evidence of activity
- For example, reviewing stage or faculty meeting minutes to identify how and what information about curriculum implementation (phases, journeys, role responsibilities, KLA/ subject matter details) is discussed.
- to determine evidence of process quality
- For example, reviewing the faculty, stage or school wide process for developing scope and sequences, teaching and learning programs or assessment materials. This can determine whether steps are relevant, concise and sequential for staff to follow.
- to determine evidence of impact
- For example, by triangulating professional learning attendance records and feedback with teaching and learning program documents, to demonstrate adequate planning to improve understanding of curriculum changes.
To undertake a document analysis, a three-step process can be undertaken:
- Step one – Identify: determine data relevant to your evaluation question; determine how you will record the relevant data you are looking for in each document, this could be through colour coding and highlighting pre-determined criteria; record in a template or use a consistent method of recording the frequency of statements related to the criteria
- Step two – Read: dig deeper to identify relevant patterns in the text; categorise or code words or phrases according to the agreed criteria
- Step three – Interpret: review the categorised data to reflect on what the documents are showing; summarise key themes or patterns; record any other relevant factors (author of the document, who contributed, who did not to determine any voices that may be missing); work collaboratively with colleagues to review interpretations of any words or phrases that are complex to code; annotate documents where necessary to show clear alignment to the coding process; tally the reference/s made to each criteria; consider overall frequencies or average frequencies of criteria to enable comparisons, and; note the facts and consider whether the data is providing new or emerging themes or aligning with existing knowledge.
- Plan evaluation questions and select relevant and existing documents to review carefully. For example: an open-ended question will result in deeper analysis, rather than a question that requires a yes/no answer.
- Keep the scope tight. Use specific criteria to evaluate relevant document/s that are aligned to your evaluation inquiry or question.
- Plan and allocate time to undertake document analysis with team members.
- Involve those undertaking the document analysis process in the planning to ensure inter-rater reliability (consistency of different individuals in measuring something).
- Collaborate with team members to determine a list of possible documents to review. These may include:
- scope and sequences
- learning sequences or units of work
- lesson plans or teaching and learning strategies
- work samples
- minutes of faculty/stage/executive meetings
- procedure documents (school) or policies (departmental)
- professional learning
- Consider using a sampling strategy if a large number of documents are to be reviewed. This may involve a representative sample across faculties or stages, or across student groups when reviewing work samples.
- Seek permission from staff prior to analysing documents and inform staff of the intention of the review and how the data will be used.
- Duplicate documents you have chosen to review, so that any annotations or coding is recorded on duplicated copies, not on originals.
Document analysis – Template
The following analysis template (DOCX 92.9 KB) provides a suggested structure schools may utilise to undertake a document analysis.