CIMA recently collaborated with Kaplan Financial as part of its data driven research series to analyse trends in successful study behaviour.
This was the second phase of our collaboration with Kaplan, having started back in 2015 when the 2015 CIMA syllabus was launched. We’ve looked at a significant amount of data to identify correlations and observe the patterns that emerge, giving us insight into how students learn.
What data did we look at?
We have been able to collect data from every part of the student journey, from registration, through the different stages in the learning process, into the exam room, concluding with an exam result.
Before looking at the findings there are a few things to note with regards to the data:
- The 6,000 plus students selected were all from Kaplan, they had enrolled on either a Classroom, Live Online, recorded (OnDemand), or Distance Learning programme
- The observations are drawn from 2015 to September 2017
- The data only includes those sitting objective test style exams
- Student success in this instance was taken to mean passing the exam at the first attempt.
What did we learn?
In order to add some structure to our research we asked seven questions, the answers are given below. It can be assumed that the results are consistent with our phase one research unless otherwise stated.
1. Does seasonality affect pass rates?
The analysis showed there to be no seasonal peaks or troughs; there isn’t a “lucky month” for taking exams when pass rates are higher. Candidates should sit the exams at a time that fits in with their work and personal commitments.
2. Do exemptions influence pass rates?
The simple answer is no, the data showed that exemptions of 5 or less make little difference to exam success. Although there were instances of students having more than 5 exemptions the data became inconsistent after that point and so no conclusions could be drawn.
3. Does the study method have an impact – such as Classroom, Live Online, recorded (OnDemand) or Distance Learning?
As in phase 1, classroom offered the fastest route. However, this is most likely a result of the structured nature of the course and fixed timetables rather than students learning more efficiently. There are also differences with regard to pass rates and study methods, but it would be wrong to conclude that one is better than the other. For example, some students prefer to study using distance learning and do very well, for others this could prove to be a poor way to learn. As such we concluded that no one study method is better when looking at the data. Students should select the method of study that fits with their preferred way of learning. This is a good example where there may be correlation but we are unaware as to the causation.
4. How long should you take before sitting the exam?
The average pass rate for students sitting the exam within 12 weeks from when the course started fell by 16% compared to those sitting the exam in the following 8 weeks. This is an important result and suggests students should not take increasingly longer periods of time to study. Although this may seem counter intuitive, for example, it would be logical to assume the more time you invest in learning the better prepared, there are many factors at play. It is also worth stating that this is an average over a 12-month period and differs for individual subjects. For this reason, we are suggesting that students sit the exam between 10 and 12 weeks from the start of the course.
We might speculate that the reason we are getting these results is because as time passes students become less motivated and to a certain extent lose their way. But at this stage we cannot say with any degree of certainty.
5. How long after the course should you take the exam?
The data shows a steady decline in pass rates from the end of the course to sitting the exam. Pass rates fell by 6% after the third week and by a further 10% by week six. This is consistent with the findings above and phase one. Taking long periods of time for self-study after the course has ended is reducing the student’s chances of exam success. Although the data includes recorded (OnDemand) and Distance Learning, these findings are more relevant for scheduled courses where there is a clear course end date.
Once again there are differences for individual papers but overall, we would recommend that student sit the exam within 3 weeks from when the course ends.
6. Does rescheduling the exam affect pass rates?
Students who change the exam date from their original booking have a 16% lower pass rate than those who book the exam and don’t reschedule. This is more striking than in our original research where the results were 9% but the conclusion is consistent, book the exam and don’t change it. Once again this might seem strange, if a student doesn’t feel ready to sit the course is it not a sensible strategy to delay? However, the data suggest this is not the case perhaps for some of the reasons already mentioned.
7. Does using MyKaplan affect pass rates?
Although slightly down on our phase one research once again we identified that attempting questions and mock exams on MyKaplan (Kaplans LMS) resulted in higher pass rates. Attempting both knowledge checks (objective tests set after a period of study) and mock exams resulted in students increasing their pass rate by 5%. This is perhaps not surprising, there is already significant evidence that testing which requires the student to retrieve previously learned information increases recall.
It is important not to interpret this as suggesting that attempting questions only increases pass rates by 5%. Many students will be practicing questions from their text books and question banks, the impact of this is not captured in this research because there is no digital evidence. The 5% is in addition to those studying using paper-based resources.
In summary the findings from our research suggest that:
- On average the time from the start of the course to when the student sits the exam should be between 10 and 12 weeks – we recommend the student book the exam when the course starts
- On average the time from the end of the course to when the student sits the exam should be between 1 and 3 weeks. We don’t want students to feel they have to sit the exam on the day after the course - a reasonable period after is all that is required
- To improve the student’s chances of passing, book the exam and don’t reschedule
- Engage with MyKaplan and practice using knowledge checks (objective tests) shortly after studying and sit a mock exam.
As more students engage in online learning, we will be able to gather even more feedback which in turn we will pass onto our students to improve the way they study and their chances of exam success.
You can also watch a recording of the webinar, held 6th December, when Kaplan's Head of Learning, Stuart Pedley-Smith, in association with CIMA, shared these findings.