Google Analytics, Google Data Studio, Google Search Console, QuestionPro, Askable
Apolitical received a grant with the goal of kickstarting a number of new projects and initiatives, one of which was building a mobile app. As Apolitical’s first user researcher, I was asked to do research to determine what V1 of the mobile app should look like and what product features it should contain. After starting the research, however, I recommended cessation of the mobile app project altogether and successfully pitched redirecting the funding towards improving the web app experience instead.
Just me! 💁♀️
Prior to my start date, Apolitical received funding to build a mobile app. I was asked to conduct discovery research on Apolitical’s device usage to get a better idea of what type of product/content would perform best on V1 of the app.
Once I started the research, however, the challenge shifted to demonstrating to the senior leadership team (and, subsequently, the grant providers) why I felt this project would lead to financial losses rather than gains for Apolitical and should be aborted.
I conducted individual interviews with 5 stakeholders, followed by a kick-off workshop to ensure alignment prior to the commencement of research and design work. The stakeholders agreed on the following benefits:
Mobile app could provide a better platform experience
Starting fresh so fewer bugs
Bypass bandwidth/VPN/firewall issues that prohibit public servants from viewing course videos
Staying logged in (currently an issue on web experience)
People are more likely to build habits on mobile than desktop
Nudges could be used to prompt users back to the app
Potential to expand platform offerings in the future
Reaching critical mass of users on web platform
Stakeholders generally agreed that launching a V1 beta test of a single course on the mobile app would likely be the best approach for the following reasons:
Videos would be accessible - currently don't work well on government computers
Easier to beta test and allows us more room for making and fixing mistakes - free course in exchange for testing motivates users to forgive us if something goes wrong
Allows us to test nudges in a way we can be more confident is relevant for users
A course app could add more appeal to potential investors/partners
There were, however, arguments made in favour of an app hosting free content as well:
Mobile is a common conduit for bite-sized content and digital communities (forums, messaging, social media)
Shorter pieces of content are more likely to viewed on-the-go than courses
The survey was distributed to all subscribed Apolitical users as part of a weekly email. Participants were invited to complete the 5-minute survey with no incentive. 140 participants completed the survey.
The largest responses came from participants in Canada (n=59), United Kingdom (n=30) and Australia (n=16). The rest of the participants completed the survey from 17 other countries.
73% of survey participants said that they primarily use a work or personal computer for online learning and only 21% said they use a mobile device. The remaining 6% said they most often use a tablet.
The types of content most commonly viewed on mobile were:
The types of content most commonly viewed on a computer were:
When asked "Where did you most often do this learning?", the 2 most common use contexts were:
At work (in office or at home during working hours) - 60.87%
At home (outside of working hours) - 32.30%
This indicated to me that the most successful features on an app would actually be services that Apolitical hosts off-platform on pre-existing apps (eg. social media and podcasts), or features that don’t yet exist on Apolitical (eg. a forum). The 73% of users stating that they do online learning from a computer rather than mobile was a red flag, as Apolitical’s current features are centred around reading and learning.
In order to understand how the Apolitical website was being used, I analysed the site's Google Analytics records from January 1st, 2020 until June 3rd, 2021. It is worth noting that there was a global pandemic with associated lockdowns being imposed during this time, so user behaviour was likely to be different than it was prior to the COVID-19 pandemic. As this research was done in June 2021 when hybrid and remote working models were beginning to be imposed, I decided that the pandemic data would likely be a better predictor of post-pandemic behaviour than pre-pandemic data. I also knew I wanted to conduct user interviews at this point and that this analytics data would be easier to triangulate with recent qualitative insights.
I found that 30.16% of visits came from mobile devices across all countries during this time period, while 66.85% came from desktop computers. The remaining 3% came from tablets. The bounce rate for mobile (20.98%) was considerably higher than the bounce rate on desktop (13.41%). Likewise, the average page views and session times were notably lower on mobile devices, implying that there may be an issue with Apolitical’s mobile user experience.
Because Google Analytics could not differentiate known members from non-member users, I asked the Apolitical data analytics team to build a more in-depth report for me so I could see the device usage of our active members specifically.
The following graph shows the device usage of known members (whether they’re logged in or out). We can see that this number (~5% or less) is fairly consistent across this time period.
Although 30.16% of visits came from mobile devices, only 4.54% of member sessions (periods of active engagement from Apolitical members) occurred on a mobile device (compared with 6.18% on tablet and 89.28% on desktop).
So who are the rest of the 30.16% if not Apolitical members? 🤔
The countries that visit Apolitical the most (USA, Canada, UK and Australia) predominantly viewed Apolitical from desktop computers. Users in the remaining countries - India (2.69% of users), Philippines (2.41% of users), Mexico (2.41% of users), South Africa (1.82% of users) and Colombia (1.65% of users) - more often viewed Apolitical from their mobile phones. Notably, these are also the countries that constitute the majority of Apolitical’s search traffic (along with the USA).
I found that about 44% of Apolitical's search traffic over the last 3 months was from mobile devices, which can be compared to 20% mobile site traffic overall over the same time period. Since search is a significant acquisition source (48% of all traffic) over the same time period, we can infer that non-search (including email, the next biggest traffic source) traffic skews very heavily to the desktop.
Hypothesis: the majority of Apolitical’s mobile users are non-members coming to the site briefly (average mobile session time for all users is 1 minute 55 seconds) as a one-off visit. Acquisition of these people is likely largely through organic search traffic.
15-30 minute long qualitative interviews were conducted with 22 participants: 17 Apolitical members and 5 non-members recruited through Askable. All were working as public servants for governments in Canada or Australia.
The vast majority of participants interviewed specified that they view Apolitical as work-related learning. While some participants stated they use Apolitical outside of working hours, most allocate time within their working day to visit the platform.
When asked about the devices they use for work-related learning using Apolitical and other learning platforms, nearly all participants said they do this from their work-issued computers. A number of participants mentioned that they also prefer their work set up with larger and/or multiple screens, which better facilitates reading course content and allows them to simultaneously monitor their work emails.
Most participants seemed to clearly delineate personal vs work-related learning. For personal learning, many expressed a preference for either their personal computers or their tablets over mobile. Those that do use their mobile devices for learning tend to do so to they can access media they can play while multitasking - usually podcasts and occasionally YouTube videos.
Public servants view Apolitical as work-related learning, and as such prefer to access the site from their work-provided computers. When asked about personal learning rather than work-related learning, the majority still preferred to use a computer over a tablet or mobile device. Those that do use mobile for learning typically listen to a podcast or watch a video on their device while multi-tasking. Apolitical’s learning materials are mostly written text and, because of government bandwidth and firewall constraints, video and audio resources don’t often work on government provided devices. None of the features currently offered by Apolitical lend themselves to the typical mobile learning use cases of Apolitical’s target market.
tldr: don’t build a mobile app!
Because members see Apolitical as work-related and prefer using their work-issued computers to access the site, I recommended halting any progress on the mobile app project and taking a mobile-first approach to improving the web app experience instead.
Improving the web app experience (which the analytics data indicates is performing poorly) would have positive knock-on effects on the desktop experience as well. The results would be increased growth, retention and engagement across all devices.
If Apolitical cannot redirect the mobile app funding towards creating a more immersive web app experience, then it needs to think carefully about how to proceed with a mobile app.
Given that Apolitical users primarily use podcasts, forums, newsletters and social media for learning on their phones, Apolitical should build V1 with community features and/or free product highlights if they cannot redirect their funding away from a mobile app.
tldr: funding redirected to improving the web app experience
I presented my findings and my recommendation to cease the mobile app project to Apolitical’s CEO and senior leadership team. I said that because building a mobile app would take significant resources spanning well beyond the initial funding received, proceeding with this project would cost Apolitical substantially more than it would likely gain in the long term. The senior leadership agreed and halted all further progress on the mobile app.
I was later asked to present these findings to the funding body and provide justification/rationale for improving the web app experience on mobile rather than building an app. I’m pleased to say that I was able to persuade them that investing in web app improvements would be a better use of their funds and more efficient way of propelling Apolitical’s mission forward. Apolitical was able to redirect this funding towards general website improvements with a mobile-first design and development approach.