User research on the new Chevening application system

Hello there, I’m Phil Buckley, a Digital Transformation Manager at the Foreign Office, and I wanted to tell you about the unusual and fun challenges faced when testing the new application form for the Chevening Scholarship programme.

How did we test?

Before we started the design of the new system, we were lucky enough to have lots of feedback from users of the current site; and as we have been developing it we have been able to test several times with our current Chevening scholars. We tested on at the ACU headquarters, at the posts in China, and also at Chevening house itself. The Chevening House testing took place during a celebration of 30 years of Chevening scholarships, and scholars from all over the world were kind enough to take a break from partying, have a look at the new system in development and tell us what they think.

What do Chevening scholars look like?

Chevening scholars are an extremely unusual demographic. They are mostly young; for almost all of them, English will not be their first language, and Latin may not be their first alphabet. The form they are completing is in English only, as they have to be able to prove their English language skills.

On the other hand, in order to get a Chevening scholarship you need to be:

• intelligent
• motivated
• ambitious
• have at least one degree already
• have some work experience
• have out-competed hundreds of other people who share these qualities

In sum: Chevening scholars are some of the best people in the world at filling in forms.

User research with incredibly talented users

This brought up some interesting questions. As you would expect, in all our testing sessions Chevening scholars mostly breezed through the new form at top speed; you may not expect that, particularly in our Chevening House test, they were often quite protective of the old form: having made the grade themselves they wanted to protect the standard which a Chevening scholarship implied.

‘The old form was hard to fill in because it used English which I wasn’t brilliant at when applying. However, you don’t want it to be too easy [to fill in the form] because you don’t want people who are not good at English to apply.’ – Chevening scholar from Mongolia.

And filling in forms which other people baulk at is by no means a worthless life skill: to get Chevening live, I had to fill in 4 separate but heavily-overlapping assurance forms in order to persuade 4 separate groups of people that it was OK.

This feedback was interesting and unexpected; but on balance I don’t think that we are in the business of making the form itself difficult to fill in. The scholarship has published ways of judging your application, and being able to find your way through a Byzantine process is not one of them. So we have tried to make the form as friction-free as possible, to give people the biggest chance to show off their abilities and potential.

Future user testing plans

The next time Chevening scholars gather will be at the induction event for the 2014-15 scholars, and we will test the new system again there. I am taking the opportunity to compose a User Research strategy: ‘We test wherever there are both Chevening Scholars and a free bar.’

The research we have done so far has been incredibly useful: even if all the scholars managed to fill in the form, they still gave us feedback and we could still see places where they had to stop and think. We also got some free competitor analysis as many of them had also applied to lesser programmes such as Fulbright or the Erasmus scholarships.

As you will have noted, at the moment we are only testing with the successful candidates: we aren’t catching those who are unable to complete the form. This is a harder challenge and one we are still working on, but we do have a number of clues, from the Chevening Facebook page and from feedback on the new system:

“Very impressive portal I have ever seen before.”

“No place to attach CV.  Maybe it’s not very important”

“The system is working quite fine and okay”

“All in all, the system was very user friendly. Overall, an excellent system.”

Some of the early feedback on the new Chevening application form

We are also researching ways of doing international user testing effectively.

We have plenty to do then, but I’d like to finish by saying a heartfelt thanks to all the Chevening scholars who were kind enough to test with us: it was very useful and – well, we had a great time. If you are applying now, the very best of luck and do let us know what you think of the new system.

Follow Phil at @philbuckley5