Methodology

A comprehensive approach was adopted to conduct the process evaluation of the 'Building Capacity in the Safe System' project. This evaluation was designed to provide a thorough analysis of the project's implementation and outcomes. The following subsections detail the data collection methods, data analysis techniques and limitations encountered during the evaluation process.

Data Collection Methods

To ensure a robust evaluation, several data collection methods were employed. The author was provided with relevant project documents for review including plans, milestone tracking records, deliverables and communication records between Transport Scotland and Agilysis. These documents provided valuable updates on the project's planned activities, progress, and adherence to timelines.

Structured interviews were conducted with both the client (Transport Scotland) and contractor (Agilysis), each lasting for approximately 45 minutes. The interviews were designed to gather qualitative data on stakeholder experiences, perceptions, and satisfaction with the project’s implementation and outcomes (See Annex A for client and contractor interview guides).

On commission, it was envisaged that other interviews and surveys would be held with a broader group of stakeholders to collect quantitative data on their experiences and perceptions of the project. The client and contractor were contacted for suitable contacts, but it was agreed by both that this additional data collection was not necessary as the client confirmed that they had secured good feedback themselves from partners as part of the overall project process.

Data Analysis Techniques

The data collected through the above methods were analysed using a combination of qualitative techniques. The qualitative data from the interviews and open-ended survey responses were analysed using thematic analysis, using NVIVO. This involved coding the data to identify common themes and patterns related to the project's implementation, client and contractor experiences, key successes and challenges. This analysis helped in understanding the nuanced perspectives of those involved with the project and provided in-depth insights into the project’s impact.

Limitations

While the evaluation was conducted rigorously, there are several limitations with the approach taken which should be noted. Given the nature of the project, the author was not privy to all project documentation as it would have been impractical for this to have been the case. However, importantly, the author was able to review milestone logs which provided critical insights into the project's progress. Originally, surveys with a broader spectrum of stakeholders were planned, but the client did not consider this necessary to follow-up. This broader survey may have revealed some different perspectives that were not captured through the interviews conducted with the client and the contractor. Despite these minor limitations, the evaluation methodology employed provides a worthwhile assessment of the ‘Building Capacity in the Safe System’ project, with actionable outcomes.