Impacts

This chapter explores the evidence, learning points and knowledge gaps from the MaaS pilots around the impacts of the MaaS pilots. The key themes explored are Monitoring and Evaluation of Individual Pilots, Evidence of Uptake, Modal Shift, MIF Themes and Pilot Specific Objectives. In each case the topics are considered in turn based on the evidence presented in the final Evaluation Reports provided for each pilot, perspectives from the consultation with MaaS leads, and engagement with transport operators and beneficiaries.

Key metrics relating to the uptake for each pilot are provided in Appendix C including the number of downloads, number of registered users, number of journeys planned and information about user demographics where this data was collected. The total number of downloads varied from 1,500 for GetGo Dundee to over 5,000 for Go-Hi. Tactran ENABLE and GoSEStran did not report the number of app downloads, instead measuring user sessions as evidence of uptake which ranged from circa 1,300 for myD&A College to 11,000 for the LLTNP app. During the engagement sessions, Go-Hi stated that they had an initial target of 5,000 users, with it noted that 3,500 users had registered on the app at the time of writing the self-evaluation report for the pilot in 2023 (4,750 was the final number of registered users with downloads of the app over 10,000). The initial target for GoSEStran was 1,000 users which was achieved a month in advance of the project closure date.. Most other pilot projects did not provide a target or benchmark for the number of users of downloads they hoped to obtain, with only the St Andrews MaaSterplan providing a target number of downloads attributed to different marketing channels. While the number of app downloads/users was reported over different periods after each app launched, there appears to be little correlation between the number of app downloads and the duration of the reporting period.

A potential reason for the high uptake of the LLTNP app, compared to all the others is that it caters primarily for visitors (over 4 million per year), and this is a group who are, by the nature of their relationship to the area, likely to actively seek travel information. Also there is a known lack of parking within the park, which may have contributed. Further assessment would be required to understand that extent to which the app could support the development of the local tourism economy, noting that apps targeted at visitors may have higher potential to influence these less frequent journeys than the more habitual journeys catered for by the other apps piloted.

Evidence of uptake differs for each pilot due to the differences in functionality of each developed platform. Some pilots measured uptake through the number of journeys planned, but others use the number of journeys booked/paid for through the platform where this function was available. Not all apps had user account/registration functions but could measure user retention by the number of returning user sessions. Statistics measuring repeated use are a better indicator of user retention and hence potential to influence travel behaviour.

Monitoring and Evaluation of Individual Pilots

Evidence from Reports

Each pilot undertook monitoring and evaluation of impacts according to their project specific monitoring and evaluation plan. In general across the pilots the following challenges and limitations limited the scope for robust monitoring and evaluation of the MIF pilots:

• App Functionality Limitations: not all apps include payment functionalities for all modes offered and could only infer measures such as mode shift from planned journeys, rather than paid journeys, with no means of verifying whether a journey actually took place by the looked up mode.
• Low Survey Response Rates: pilots which had aimed to use user surveys including Go-Hi and GoSEStran obtained low response rates which meant that quantitative analysis could not be confidently carried out in the case of GoSEStran. The richest data was gained from the LLTNP, though when cleaned for valid responses and segmented by users and non-users, the sample size was still low.
• Short Evaluation Periods: for GoSEStran the period between app launch and surveys was two and a half months, while other pilots had longer evaluation periods of up to 12 months, they suggest this was limited time to observe uptake in a new digital platform and any positive impacts of this. Tactran ENABLE said they needed to take a “pragmatic approach”, including the use of less robust or representative ‘convenience samples’ for user surveys due to the available budget and evolving nature and extent of the pilot.
• Representative Timepoint: The Go-Hi pilot noted the challenge of obtaining baseline data from which to evaluate the MaaS app against. In addition they state there was not a representative timepoint to assess the pilot, given their app was added to iteratively over time, a finding which was echoed by Tactran ENABLE who noted the “evolving nature and extent of the pilot”. The difficulties presented by delivering the project during the COVID-19 pandemic would have exacerbated this issue.

A variety of methods were used to gather data across the pilots and these each had their strengths and limitations. The full scope of monitoring and evaluation activities for each pilot is included in Appendix E including the response rates to surveys, focus groups held, and other evaluations undertaken as part of the pilot. An overview of the key methods used to gather data is provided below.

Dashboards/System Data

Data Dashboards were available through both Ember Technologies and Fleet on Demand Mobilleo and provided information on user uptake (number of downloads, accounts set up) and how the app was being used, (number/type of journey planned). Where user accounts had been created, further information was available, such as age and access to a car or not. User uptake data was used to monitor the success of marketing campaigns and to provide an indication of return users (most notably on the St Andrew’s MaaSterplan project and GetGo Dundee). Dashboard data also provided information on the kind of journeys that were being searched for, providing some insights into the way that people were using the app, the most common finding being that app users were looking for public transport information – specifically bus times, as opposed to booking taxis or other shared mobility.

The extent to which data was analysed in order to inform decision making on transport provision is limited. With over 10,000 journeys planned, the LLTNP app (within Tactran ENABLE) is the most popular of any of the MaaS apps trialled and data available through the dashboard made it possible to analyse where these users were based. This, and evidence from beneficiary discussion, provides some evidence of the ability for app data to provide insights into travel behaviours in order to adjust or tailor transport services. It is also worth noting for Go-Hi, additional data was available directly from operators such as car club and bike hire fleet deployment and utilisation, independent of the MaaS trial.

A key limitation of dashboard data as a metric of success is that it only considers the ‘captive audience’ of app users and is therefore an unrepresentative sample. The fact that most users were searching for bus times is therefore indicative of the type of audience who may already be engaged in looking for journey information, or who will have seen marketing materials posted at public transport related locations such as bus stops. The effectiveness of MaaS in inducing mode shift is therefore uncertain, based on dashboard data.

Surveys

User and non-user surveys were carried out by Tactran ENABLE, GoSEStran and GoHi.

GetGo Dundee conducted travel behaviour surveys on the day of each of their target events. StAndrews MaaSterplan promoted the wider GoSEStran survey.

Go-Hi conducted an Onboarding Survey, a User Survey and utilised results of the wider Highlands and Islands Travel Survey to gather experiences and attitudes towards the app. Despite a completion incentive, the sample sizes for the Onboarding Survey and User Survey were very low (35 and 54 respectively) impacting the confidence in the results and preventing segmentation. Responses to the Highlands and Islands Travel Survey related to attitudes towards MaaS in general and were non-specific to Go-Hi.

User and non-user surveys allow for greater understanding of the attitudes towards MaaS by different groups, provide a counter-factual in the absence of baseline data, and allow interrogation of the population in terms of attitudes towards MaaS and how likely it would be to change behaviour. Across all pilots which undertook user and non-user surveys, the sample size of non-users was significantly larger than those collected for users. Broadly, these surveys showed that there was a positive attitude towards the MaaS solution as a tool for behaviour change. However, as noted in all of the reports, the sample size and distribution of app users was consistently small and difficult to draw conclusions from.

Qualitative Data

Due to time and budget constraints, GoSEStran were the only one of the pilots to complete a user focus group as part of the post implementation evaluation, rather than as part of app development This found limited evidence (due to only four participants) that users planned their journey in the app, but then purchased tickets through other apps. End-user discussions were held with Tactran ENABLE projects although this was intended to shape the product at the start of the project, rather than gather evidence of effectiveness.

Evaluation Frameworks

All pilot projects developed objectives contained within their initial business cases and monitoring and evaluation frameworks. However in the cases of Tactran ENABLE and GoSEStran, these frameworks were not followed through into the final evaluation reporting. The Investment Fund did not prescribe a consistent or coherent framework through, for example, a theory of change model for which to objectively assess how well each pilot met the aims and themes of Transport Scotland. This led to a shortfall in evidence collected to support the central aims of Transport Scotland. Pilot programmes were therefore permitted freedom to set their own objectives which allowed for exploration of MaaS as a solution for a range of problems for which multiple learning streams have emerged.

Engagement with MaaS Leads

The conversation with St Andrews MaaSterplan elaborated on the challenges with monitoring which included that not all journeys facilitated by an app would be captured/counted in the number of journeys planned, an example would be a daily commuter who might use the app once to plan their journey, but only check real time information once they are familiar with their regular route, or ridesharing whereafter the initial contact between driver and rider, journeys may be planned out with the app.

Key Learning Points

From the evidence gathered, the following key learning points have been identified.

Finding a common framework that works has proved very difficult for MaaS across the world. As described in the introduction of this report, funding for MIF pilot projects was granted based on the submission of a business case which responded to the overall aim of the fund, one or more thematic areas, NTS2 priorities and several other policy drivers. In addition, each pilot project produced their own, separate objectives, which were approved via the submission of detailed monitoring and evaluation plans. A unified evaluation framework encompassing all these elements was not created. The development of such a framework in advance of future investment could assist with delivering more robust evidence in relation to Transport Scotland’s aims and objectives in the context of further MIF investment.

Some pilots reported that monitoring of the app took place over a limited period of time, in some cases only a few months after the app launched, either due to delays encountered earlier in the app development, the fluid nature of the app roll-out, or time constraints of the MIF programme. Some pilots claimed this did not allow enough time for robust timeseries data to be collected and the benefits of their work to be realised.

The quality of data used to monitor the apps was severely limited by the functionality, for example apps without comprehensive booking/payment functionalities could only infer metrics like mode choice from the last journey a user planned, with no means of verifying whether a journey actually took place by the looked-for mode. However, while the limited user-base achieved by the apps limited such benefits in the context of the pilots, the study highlighted opportunities to deliver insights relevant to policy priorities, including understanding rural accessibility and demand for travel as part of a wider roll-out.

All pilots generally found it difficult to obtain meaningful sample sizes from user surveys and may need to seek alternative methods of evaluation.

Modal Shift

Evidence from Reports

Positive evidence of modal shift associated with the use of any of the MaaS pilots is limited and should be interpreted with caution given the small survey sample sizes. However, Tactran ENABLE provides some evidence that the pilot was useful in supporting those making multi-modal journeys. The Dundee and Angus College Surveys conducted as part of this pilot showed platform users surveyed (n=24) were more likely to be making multi-modal journeys than non-users (15% of non-users (53 out of 351) journey to college had three journey parts, compared to 38% of app users (9 out of 24)) and around a third (8 out of 29 app users) stated that the platform had influenced (a great deal or a lot) how they chose to travel to campus.

More than half of Go NHS Tayside platform users surveyed (52 out of 88) stated that the platform would be more likely to make them consider using public transport and active travel modes with around a third stating they would have used the car if they had not had access to the platform. Similarly, over half of survey respondents (39 out of 69) who had used the National Park Journey Planner ‘Strongly agreed’ or ‘Agreed’ that using the National Park Journey Planner had made it more likely that they would use public transport, walk or cycle instead of drive. Both of these questions are self-stated for the respondent, with no further evidence on observed modal shift provided or comparisons with non-users.

There is evidence from the Tactran ENABLE pilot that MaaS apps gave users additional confidence when travelling by public transport. Only 4% of Go NHS Tayside users reported they were not confident they would reach their destination on time (3 of 76 users surveyed), compared to 21% of non-users (70 of 337 non-users surveyed). A further indication was provided by the responses to the GoSEStran user survey, with around two-thirds of respondents (16 out of 26 app users) saying that they strongly agreed or agreed that the app encouraged them to use public transport, walk or cycle instead of drive. Due to the small sample sizes of app users, the findings of both surveys need to be interpreted with caution and cannot be generalised to app users as a whole.

Engagement with MaaS Leads

MaaS leads were, in general, positive about the potential for mode shift due to MaaS, though conceded that the numbers involved in the pilots were too small to robustly quantify the modal shift potential or wider impacts of MaaS over the course of the pilot. It was also acknowledged that abstraction from car journeys was less likely due to the user profile being generally existing public transport users. Among public transport users, there were further challenges in introducing alternative modes. Go-Hi suggested that the uptake of e-bike hire was dampened in educational settings due to the Young Persons’ (Under 22s) Free Bus Travel Scheme.

Engagement with Operators and Beneficiaries

Operators in the Go-Hi region felt that MaaS has little impact on modal shift away from private cars, and any additional trips by shared mobility or micromobility are likely to come from public transport. One of the Go-Hi DRT operators saw no evidence that Go-Hi had directed new users to their service.

Measures which discourage car use (parking charges, access controls) were seen as the only viable ways to reduce car use. One operator stated that ‘carrots’ such as free/concessionary public transport do not have an impact and it is unrealistic to expect MaaS to achieve modal shift on its own.

Some of the intended added value of MaaS did not see as great an impact as hoped. For example, Dundee and Angus College students did not value incentives and rewards as part of the app because of the free bus travel they already receive.

There was a view that the app may , in some cases, be counterproductive. A key problem with MaaS apps as highlighted by beneficiaries was that it could unintentionally promote the uncompetitive journey times of public and active travel, in comparison to private car or taxi journeys. They noted that the app could adversely lead to car journeys being chosen over more sustainable modes in contrast with their own objectives.

Key Learning Points

From the evidence gathered, the following key learning points have been identified.

Some pilots reported that it was difficult to get users to sign up to create an account or register themselves on the app. This made personalisation more difficult. A reason cited for the lack of account creation was users being unaware/unclear of the additional benefits of registering.

Experience from the pilots found that MaaS users are, in general, already likely to travel by public transport, and that additional trips by shared mobility or micromobility are likely to come from public transport, as opposed to the private car. There was some evidence however that MaaS can increase the confidence of users completing a journey on time and without delays. MaaS can also have the unintended consequence of highlighting uncompetitive journey times by public or active travel when compared to private or shared car use. Complementary policy measures are required to generate the conditions for MaaS to have an impact on modal shift.

MIF Themes

Applicants were asked to address one of more of the thematic areas set out in the introduction of this report. While these were referred to in their initial business cases, the final evaluation reports focussed on their own evaluation criteria, rather than presenting evidence around each theme., Though some did report relevant pieces of evidence which could be mapped to one or more of the themes, such as the Making Connections audit.

The most well-reported on themes were “Rural, Island and Communities”, and “Tackling Inequality, Accessibility and Mobility Barriers”. Many of the pilots were delivered in rural areas (GoHi, Tactran ENABLE, GoSEStran), and some pilots reported findings which showed the impact of MaaS on social inclusion and accessibility to services (NHS Tayside, D&A College as part of Tactran ENABLE).

Overall the themes were not used consistently enough to guide the pilot projects implementation or subsequent self-evaluation. In this section, relevant evidence is presented against each theme, where available.

Rural, Islands and Communities

This theme was specifically referred to in MIF applications for:

• GetGo Dundee
• Go-Hi
• GoSEStran
• St Andrews MaaSterplan
• Tactran ENABLE

Evidence from Reports

Go-Hi stated that rural communities were a focus area for their pilot given “shared mobility services can complement public transport systems by improving accessibility in areas with infrequent public transport schedules or limited infrastructure” and that “rural areas with less public transport are being left behind by MaaS developments because of the importance of public transport to MaaS”. St Andrew’s MaaSterplan noted that ridesharing which was a key component of their MaaS project can help solve the lack of public transport in rural areas.

GoSEStran noted that the most successful DRT examples in rural areas have been where funding is available to provide “new branded rolling stock to accompany the change to DRT”. They also note that rural bus operators in particular do not often have the means for pre-booking and payment which restricts what is achievable through MaaS. Despite this, the analysis of origin and destination data for GoSEStran suggests that the app is supporting individuals in rural locations to plan their journeys and is “of value in addressing the transport poverty endemic in rural locations”. St Andrews’s MaaSterplan notes that challenges remain regarding the service models and available funding for rural DRT services.

Engagement with MaaS Leads

Discussion with GoSEStran suggested that origin-destination data generated by the app provided some detailed insights into rural transport demand, however the numbers of people involved were not statistically significant enough to influence matters such as bus route planning. Go-Hi, however, used a different operating system and felt the dashboard data returned was not sufficiently detailed to provide reliable insights into demand.

Go-Hi felt that MaaS provides an opportunity to create a more equitable transport system in rural areas, perhaps through the use of mobility credits, as existing schemes like free bus passes having a disproportionate benefit in urban areas where there are more extensive bus services.

Engagement with Operators and Beneficiaries

DRT operators linked with the Go-Hi app felt that the automated route optimisation of their service based on bookings led to indirect journeys and excessive journey times for some users depending on how many people the service was carrying and how dispersed stops were. This reflected that “one size does not fit all” when it comes to planning DRT services in urban and rural areas.

Tackling Inequality Accessibility and Mobility Barriers

This theme was specifically referred to in MIF applications for:

• GetGo Dundee
• Go-Hi
• GoSEStran
• St Andrews MaaSterplan
• Tactran ENABLE

Evidence from Reports

Tactran ENABLE reporting states that MaaS should be “seen as a tool for promoting social inclusion by helping people access services such as employment, education and healthcare”. The report states that user surveys provide “pointers” that a MaaS platform has the potential to enable students to access further education opportunities that they might otherwise miss out on, make it less likely that patients will miss or have to cancel medical appointments (49 out of 88 (56%) users agree “Using Go NHS Tayside planner has made is more likely I will not miss my medical appointment”), and enable people without access to a car to undertake more leisure activities.

Comparisons between the user and non-user subsamples for Tactran ENABLE show that app users tend to have less access to vehicles than non-users. Sixteen per cent National Park Journey Planner users reported having ‘no access to a car or driving license’ compared with 1% of non-users (n=57 users and 250 non-users), and 17% of GoNHSTayside users do not have access to a car compared to 13% of non-users (n=76 users, 337 non-users). App users were also found to be more sensitive to cost (31% of student myD&A travel users indicated they could not afford the alternative modes of travel compared to 17% for non-users) and more likely to have a disability (14% of myD&A travel student users had a disability which affects their travel arrangements compared to 5% of non-users (n=29 users and 361 non-users)). While these findings show the audience reached by the app are more likely to face barriers to transport, it does not provide evidence that the app contributed to the reduction of barriers to transport for these users.

Both Tactran ENABLE and GoSEStran user surveys also suggested that users of the app are less likely to own a car than the general population and are more likely to suffer from a disability or be sensitive to the cost of transport. There was also a suggestion from Go-Hi that a future application of MaaS could use mobility credits to correct for “spatially unequal” policies like free bus travel, which is of little benefit in rural areas where bus service provision is poor. Due to the small sample sizes of app users, the findings of both surveys need to be interpreted with caution and cannot be generalised to app users as a whole.

Inequality could also be tackled through the development process for the apps. The Tactran ENABLE project included a Making Connections audit which was stated to enable “the design of simple and easy to use interfaces” with a focus on people with disabilities and early onset dementia which improved the accessibility of MaaS.

Engagement with MaaS Leads

For the Tactran ENABLE Dundee and Angus College app, there were more searches for the cheapest travel option. This showed what is important to the specific audience of students.

Discussion with St Andrews MaaSterplan supported the finding from Tactran ENABLE that a MaaS app can facilitate access to education and was especially pertinent in St Andrews where accommodation pressures mean many students live outside of the town.

Engagement with Operators and Beneficiaries

Dundee and Angus College supported statements from the Tactran ENABLE self-evaluation report that the app helped to address mobility barriers for travel to college, and in particular gave pupils not used to travelling by public transport additional confidence.

One DRT operator felt that solely relying on MaaS as the means for customers accessing services could lead to exclusion for a key part of their target market, as some of their customers would struggle to use an app or even own a mobile phone. A MaaS app solution does not therefore replace the need to maintain phone booking.

Tourism

This theme was specifically referred to in MIF applications for:

• GetGo Dundee
• Tactran ENABLE

Evidence from Reports

A key audience for the LLTNP app were tourists and visitors to the park, and the app had the aim of promoting sustainable tourism and allowing visitors to easily explore options for travel to and around the Park that did not involve driving. User sessions were strongly associated with the National Park’s peak visitor season and the large majority of users were from outside the National Park including the central belt of Scotland, and various locations in England and Wales. The app was used to complement the closure of a previously problematic car park at Conich Hill with a MaaS tool which acted as a ‘carrot’ offering journey alternatives.

Go-Hi suggested that tourists have particularly made use of Brompton and Hi-Bike shared bicycles due to the seasonality of the uptake which has reduced their car use at their destination.

Outreach activities with the tourism industry as part of the St Andrews MaaSterplan pilot received little response due to a lack of interest of “engaging with a pilot project developing an app that has little to no immediate impact on their own business goals”.

Engagement with MaaS Leads

Tactran ENABLE discussed the work with LLTNP as being an example of the project fulfilling an unmet need in terms of sustainable visitor management.

Engagement with Operators and Beneficiaries

Loganair felt that tourists are a captive audience for MaaS as this group will typically be looking for onward travel in an unfamiliar environment, while lacking the convenience of a private vehicle. The airports they operate out of in the Go-Hi area are often small and lack clear information about onward connections. MaaS was described as “joining the dots” for the customer’s journey.

LLTNP project partners for Tactran ENABLE discussed the use of the app being a potential tool to manage demand and to direct tourists towards different, less well used parts of the park and relieve pressure on tourist ‘hotspots’. Frustration was voiced that the lack of personalisation of the app did not make this possible. The high profile of the LLTNP brand made marketing the app towards tourists more successful than other apps. The National Park are investing in new transport solutions, such as a trial shuttle bus between Aberfoyle and Callander. A positive feedback loop was set up, with the app used to promote new transport options and new transport options used to promote the app.

Urban Environments

This theme was specifically referred to in MIF applications for:

• Go-Hi
• GoSEStran

The theme was too broad to have drawn out any specific learning points. This was a theme in Round 2 reflecting that appetite for a MaaS app within urban environments where there are plenty of other travel options and provision of a number of zonal ticketing schemes, was not forthcoming at the time of the funding.

Evidence from Reports

GetGo Dundee reported that the implementation of MaaS “must be accompanied by broader urban transport policies prioritising safe access to sustainable travel and discouraging reliance on private car”.

Engagement with MaaS Leads

This theme was not raised in engagement with MaaS Leads.

Engagement with Operators and Beneficiaries

This theme was not raised in engagement with Operators and Beneficiaries.

Key Learning Points

From the evidence gathered, the following key learning points have been identified.

While the available evidence is limited and often anecdotal from a small sample of respondents, it nonetheless points towards the contribution of MaaS towards the achievement of social inclusion policy objectives, including facilitating access to healthcare, education and leisure activities for those without access to a car or more sensitive to the cost of travel, as these groups were found to have higher uptake of the pilot apps. Deployment of MaaS could also provide a platform to support the distribution of mobility credits through the national entitlement scheme. The fact that user surveys found, users were more likely to face barriers, which could be assisted through Maas tools, also represents useful learning in regard to the potential for reducing inequalities.

Pilot Specific Objectives

Evidence from Reports

As set out in the introduction of this report, each pilot had a number of pilot specific objectives which formed the basis of the self-evaluation report for each pilot. The performance of each pilot against their set objectives is set out in Appendix D.

Though the pilot specific objectives were created by each project team, the extent to which these have been used to evaluate the success of each pilot is mixed. The GoSEStran and Tactran ENABLE pilots both developed a strategic overview containing a goal, priority and objectives for their projects as part of a wider logic model which contained performance measures such as outputs, outcomes and impacts. However for both of these pilots, the pilot specific objectives developed at this stage were not returned to, or referred to, in the final report at the conclusion of the trial. Instead Tactran ENABLE used the NTS2 priorities to group their findings while Go-Hi and GetGo Dundee used a series of Performance Indicators broadly linked to their stated objectives.

Engagement with MaaS Leads

GetGo Dundee attributed the failure to meet some of their initial project objectives to the COVID-19 pandemic, as the app launched in September 2020. Their project had been based around public events, some of which were disrupted/rearranged and measures such as car sharing they initially wanted to facilitate, being discouraged at the time.

While not part of the St Andrews MaaSterplan objectives, they cited successful wider impacts such as raising the profile of MaaS within organisations, including St Andrews University and Fife Council, for which MaaS is now included in their transport plans and strategies written since the pilot.

Engagement with Operators and Beneficiaries

Engagement with beneficiaries did not provide evidence regarding whether the pilot objectives were met. Instead, it highlighted that the objectives of beneficiaries often differed from the objectives of the MaaS pilots. For example, during engagement, Dundee and Angus College stated that the app aimed to address parking pressures on campus and the mobility barriers faced by some rural based students. However, these aims were not directly reflected in the Tactran ENABLE pilot objectives.

While East Lothian Council do refer to MaaS in their local transport strategies, they felt that Journey/Mobility Hubs are an area of greater priority, and that the MaaS pilot could have better integrated with this policy area. Another objective for East Lothian Council as part of the GoSEStran project was to improve perceptions and awareness of a DRT service, and address practical difficulties such as keeping printed timetables at bus stops up to date.

Key Learning Points

From the evidence gathered, the following key learning points have been identified.

There was inconsistency between the pilot projects in the way that project objectives are reported on. Objectives referred to in initial applications differed from those in the approved monitoring and evaluation plans and these differed again from those reported on in evaluation reports. Future work should ensure the stated objectives are supported by an overarching evaluation framework, including statement on how outcomes will be measured and followed through into final reports. A constraint mentioned by all pilots was the time limits of the programme, which did not permit evidencing medium to longer term impacts. Engagement with key personnel highlighted that the pilots would have benefitted from more time to test and evaluate the apps, to realise the full benefits.