My First Future Fellows Article

The hectic reserving season has only just ended, so I won't be producing a real post this month. However you can peruse the article I co-authored in the latest issue of Future Fellows. You can view the publication online here: http://casact.org/newsletter/index.cfm?fa=ffmain

I did bring up the issue of exam start times and sleep (my pet project) with the Candidate Liaison Committee. It wasn't met with much enthusiasm; there are concerns about late exam end-times that might require lunch breaks. However I intend to continue promoting later start times, because I really believe it will help candidates.

Speaking of things that will help candidates, I'm learning more about the appeals process through my involvement with the CLC. I hope to post something extensive on that soon. We've discussed having a Future Fellows article on the subject as well. From the feedback I've been getting, it sounds like people really need some clarification on that process. I hope to provide that for you soon. Until then, keep studying!

Update: CAS Releases Exam Responses and the President Addresses Low Pass Ratios

In case you missed it, candidates who scored a 4 or 5 on fall 2011 Exams 6 or 8 can request their ungraded responses from the CAS: http://casact.org/members/index.cfm?fa=viewArticle&articleID=1866&CFID=19255387&CFTOKEN=72268585

If you wish to submit a request, you must do so by March 7th, 2012, so don't delay.

Also Patricia Teufel, CAS President, released the following letter shortly after I published by commentary on the 2011 exams: http://www.casact.org/cms/pdf/Open_Letter_Regarding_Fall_2011_CAS_Exam_Results.pdf

It's a lot more candidate-friendly than the Exam Committee's letter. There's even a way to give her feedback directly. Check it out, when you have a few minutes.

Also I finally received my official letter from the President. I'm genuinely ACAS now. Hurray!

So Close...

...but still not official until I receive my signed letter in the mail. Thanks, Course on Professionalism, for making that crystal clear!

Forgive my poor formatting. I blame Blogger.

2011 Exam Results

If you sat for an upper-level CAS exam in 2011, the probability that you’ll be sitting for that exam again in 2012 is 73.1%. Out of 3,195 exams administered in 2011, only 858 met the Exam Committee’s standards for a minimum qualified candidate. With the exception of the half exams 5A and 5B, every exam’s pass ratio was below the 12-year average (2000-2011). Any way you slice it, these results are frustrating, but are they as surprising as they appear at first glance?

The Exam Committee recently published commentary on the unusually poor 2011 pass ratios, which can be found here. I was somewhat surprised by what the Exam Committee chose to say. In my view, there are two major questions to be answered:

1.       Are 2011 pass ratios really as unusual as they appear? Are they consistent with prior years or are they outliers?

2.       If the 2011 pass ratios are significantly lower than usual, what is driving the change?

The article by the Exam Committee doesn’t really address either of these questions, except insofar as they declare that they consider all of the 2011 exam results “outliers” and discuss grading and pass-mark-setting methods as possible drivers. In other words, they’re on the defensive. Their article is mostly a justification of their grading policy rather than a considered analysis of these seemingly unusual results.

The article strikes me as statistically a bit sketchy, as well. For instance why even bother to address the pass rate on Exam 7 when only nine people sat for it? Compared to a typical year that has around 1,000 exam-takers, it’s hardly a surprise that results from a population of nine appear anomalous. Throughout the article, quantification is generally lacking.

To satisfy my own curiosity, then, I crunched some numbers. I looked at the number of exams taken, number of exams passed, raw pass ratio, and effective pass ratio from 2000 through 2011 for each exam. I mapped new Exam 8 to old Exam 9, new Exam 9 to old Exam 8, new Exam 6-US/CAN to old Exam 7-US/CAN, new Exam 7 to old Exam 6, and new Exam 5 to old Exam 5. I also mapped new Exam 5A (the ratemaking half of Exam 5) to old Exam 5 for comparison purposes. Unless otherwise specified, I included all years 2000 through 2011 in the results discussed below, with the exception of new Exam 7 for which credible data is not available due to the extremely low number of test-takers in 2011.

Eventually I will put all of the metrics I looked into on this site, but for now here are the most interesting highlights:

All full-length exams had pass ratios below the average.

2011 was objectively a below-average year, with the notable exception of the two half-exams, 5A (ratemaking) and 5B (reserving), which, compared to the old Exam 5 and the old Exam 6, respectively, showed extraordinarily high pass rates.

Three of the full-length exam pass ratios are statistical outliers.

The pass ratios for new Exams 8, 6-US, and 5 are the only ones that fall outside of two standard deviations of the mean. Using Chauvenet’s criterion, these three also qualify as statistical outliers. Notably the results for the half-exams 5A and 5B are also statistical outliers, but on the high end.

New Exam 6-US results were the worst.

Excluding Exam 7, the pass ratio for Exam 6-US was the worst, nearly 3 standard deviations below the mean.

Other Considerations

While the general education structure did not radically change from 2000 to 2010, the CAS did make a significant change to their grading policy starting in 2006. From 2000 to 2005, the Exam Committee targeted a specific pass ratio when setting exam pass marks. From 2006 on, however, they revised this policy in an effort to pass all candidates who demonstrated sufficient knowledge of the material and fail all those who did not. One would expect, therefore, to see greater variability in pass ratios after 2005.

In comparing the actual standard deviations for each exam for years 2000-2005 and 2006-2010 (excluding 2011 due to the radical change in exam material and structure), I found that variance did in fact increase in most cases. Old Exams 5 and 8 actually have shown less variation in pass ratio since the grading policy change. Old Exam 9 showed the greatest change, with the standard deviation increasing 120.8%.

Since there is a significant difference in variability after 2005, I applied Chauvenet’s criterion to the years 2006-2011 in order to test whether 2011 would still be considered an outlier. The 2011 results for Exam 6-US remained an outlier as did the results of the full Exam 5. All other results, however, do not qualify as outliers.

Data Analysis Summary

While the results of the 2011 sittings are irregular, once we account for uncredible data (new Exam 7) and the change in grading policy in 2006, the label “outlier” becomes questionable for most exams. 2011 had the lowest number and percentage of passing candidates in the last twelve years, but that in and of itself is not enough to make it a statistical outlier. After all, some year has to be the lowest (and let’s hope 2011 was it.) In my opinion, the only exam that looks like a real outlier is Exam 6-US.

Primary Drivers

Now for the second key question: what factors drove the unusual exam results in 2011? First we need to consider, a priori, what we expected to see in 2011. Three factors stand out in my mind as most relevant: who is taking the exam, how much the exam has changed from prior exams, and how long the exam is.

In the section on Exam 5, the Exam Committee’s article uses the big difference in candidate performance between the full exam and the two half exams to support the thesis that overall performance varies widely between groups of candidates. Personally I don’t see why a belief that candidate quality would vary significantly over time is controversial. Consider for instance that the number of candidates has increased over time, and average quality of an elite group nearly always decreases as group size increases.

We could test this theory If we could assign candidates to “classes.” Unfortunately I don’t have sufficient data to explore this question properly. I did, however, look at the pass ratios by exam by year to see whether performance tended to move in the same direction for all exams in a given year. They do appear to be correlated.

I’ve heard varying opinions as to whether having previously taken a particular exam makes you more or less likely to succeed on subsequent attempts. I’ve never seen data to support one view or the other, but my guess is that a candidate who fails on her first attempt at one exam is likely to fail on her first attempt at another, and that her probability of passing on the second attempt is about the same as the general population.

Finally, while I don’t have data by “attempt number,” we can deduce some useful information about the 2011 exam-takers due to the change in examination structure. Here’s what we can guess:

Exam 5 (full exam) – This is either your first time taking an upper-level exam, or you previously failed old Exam 6 (and never took old Exam 5), or you previously failed both old Exam 6 and old Exam 5. Either way, your performance is likely to be below-average, especially since the repeat-sitting advantage is diminished when the exam changes significantly, as Exam 5 did. In addition this exam is four hours long. Expect poorer than average results.

Exam 5A (ratemaking half) – You previously passed old Exam 6, and either previously failed old Exam 5, or never took it before. Either way, you’ve previously succeeded on an upper-level exam. The exam has changed from the prior, but it’s only two hours long. Expect better than average results.

Exam 5B (reserving half) – You previously passed old Exam 5, but either failed or didn’t sit for old Exam 6. Since 5B is essentially a much easier version of old Exam 6, even failing 6 before would likely be an advantage. It’s also only two hours long. Expect better than average results.

Exam 6-US/CAN – This could be your first upper-level exam, or you failed old Exam 7-US/CAN. The syllabus changed significantly, as it has several times in the past, but otherwise there is no particular reason to consider it more difficult than usual or the candidates any less prepared. The exam is four hours long. Expect average results.

Exam 7 – The nine candidates who sat for this exam had to have previously failed (or didn’t take) old Exam 6 and are either skipping Exam 5B or are attempting to take both Exam 7 and Exam 5B in the same sitting. The latter is much more likely. Tackling six hours of exam (Exam 7 is four hours long) in a single sitting is probably not the best idea. Additionally the syllabus changed significantly. Expect poorer than average results.

Exam 8 – Either this is your first attempt or it’s been 1.5 years or more since you sat for old Exam 9, reducing the advantage of a repeat-sitting. The exam is shorter than in the past at three hours long and contains mostly the same material, which should make it easier than some of the others. [CORRECTION: I was mistaken about the order of the exams under the old structure. It would have been just one year between sittings, not 1.5, and therefore you would expect average results, not poor results.] Expect poorer than average or average results.

Exam 9 – Either this is your first attempt or it’s been just six months since you sat for old Exam 8. This could be an advantage, and in any case isn’t likely to be a detriment. The exam is also shorter now at three hours long and contains roughly the same material. [CORRECTION: I was mistaken about the order of the exams under the old structure. It would have been one year between sittings, six months, and therefore you would expect average results, not better-than-average results.] Expect average or better results.

Expectation versus Reality

Comparing the actual results to our predictions makes some things clear and others more mysterious. The unusually high pass ratios for Exams 5A and 5B make intuitive sense, as do the low pass ratios for Exam 5 and Exam 7. But Exam 6, the biggest outlier with the lowest pass ratio, wasn’t expected to change all that much. Exams 8 and 9 also had lower pass ratios than one might have expected, even if they aren’t clear outliers. What’s going on?

I can only offer three theories:

1. The CAS efforts to reduce travel time by offering lower exams more frequently has resulted in candidates sitting for upper exams with less work experience under their belts. Personally I’ve often wondered how I could possibly learn everything I need to know for an upper-level exam if the material were 100% new to me. More work experience not only gives you more exposure to the concepts that turn up on exams, it also gives you anecdotes to draw upon when answering exam questions. This is a significant, perhaps indispensible, advantage.

2. Candidates have come to rely almost exclusively on secondary materials created by exam prep courses rather than the actual exam papers. Because there are relatively few courses to choose from, many candidates are taking the same course and will therefore have similar strengths and weaknesses. If a popular prep course’s curriculum turns out to be inadequate, more candidates will fail than would if everyone studied independently. This could explain the poor results for Exam 6, as I considered the accounting materials from TIA, a popular prep course, woefully inadequate.

3. The upper exams are getting harder. They’ve been telling us this was coming for a long time. Maybe it’s finally here. The days of straight-forward questions and regurgitation of lists is behind us. Only that doesn’t mean you get to do less memorizing. On the contrary, you have to do everything you did before and then a little more to answer today’s more conceptually challenging, application-based exam questions.

The only other significant change I'm aware of in terms of the exams themselves is the new method used to determine exam length and difficulty. An article appeared in the most recent Future Fellows describing the new practice, which you can find here. In short, recent Fellows who performed extremely well on their exams are being used to testdrive exams. Given the considerable and consistent complaints about exam length these past two sittings, it's possible that these former candidates are harsher judges than the subject matter experts who used to be the primary determiners of exam length.

What to Do about It

If you moved quickly through the lower exams, it will help to recognize that you’re at a disadvantage in terms of work experience. It should also help to remember that the upper exams are not like the lower exams. They require more independent work. I can’t speak for Exams 8 and 9, but for 5 through 7 I found that reading the actual papers – several times – was key. You can’t rely on someone else’s interpretation of the syllabus or the readings, not when the stakes are so high and the exams keep changing. Unfortunately, though, that’s really it. There’s no magic trick that will guarantee you pass every time.

Take advantage of what influence you do have with the Exam Committee, whether it’s through your friendly neighborhood Candidate Representative or exam surveys or other means. Be sure to be professional, however. Even a valid point will be ignored if presented inappropriately. The CAS keeps a close eye on Actuarial Outpost and other forums, so be mindful of that as well.

If you find this conclusion unsatisfying, we’re in perfect agreement. It really amounts to “Well, that’s how it is,” Just keep swimming,” etc. Still, knowledge is power, even when it’s knowledge that you’re powerless, right? Right.

Best of luck to everyone! Here’s to a better 2012.

References:

Data on historical pass ratios by exam: http://casact.org/admissions/examstatsum.pdf

Exam Committee commentary on 2011 pass ratios: http://casact.org/admissions/roth_PassRatio.pdf

Exam Committee commentary on pass ratios for full versus partial exams: http://www.casact.org/cms/index.cfm?fa=viewArticle&articleID=1611

How pass marks are set: http://www.casact.org/admissions/index.cfm?fa=passmarks

2007 article on changes to exams: http://casact.org/admissions/Changes_in_Exams_Article.pdf

Also a note of caution when looking for answers on the CAS web site: I came across a number of articles related to exam policies on the site that were clearly outdated. Pay attention to the date material was published. When it comes to exam questions, I wouldn't give too much credence to anything published before 2006. Note also that the idea of a target pass ratio of 40% and a target pass mark of 70% comes from a wish expressed by the Board at one point, not  from the Exam Committee. It is not the official practice of the Exam Committee to target 40% of exam-takers scoring 70% or higher on each exam.

CAS Website Update

One of my personal missions when joining the Candidate Liaison Committee of the CAS was to campaign for a revamping of the CAS website. Happily I learned during my first meeting with the Committee that a site redesign is already under way. Over the course of 2012, the CAS website will be substantially improved. In fact one of the first major improvements is up today: new and improved exam pages.


Other exam-takers may have experienced the same frustration I've felt with the way exam-related materials are scattered around the Admissions/Exams section of the site. No longer! Now each exam has its own page with links to relevant information and materials for that exam. No more hunting for past exams and pass marks! No more wondering how to purchase that Study Kit! The new layout has essentially everything you need in one location.


I hope everyone who's currently sitting for exams finds this change helpful. As always, I'd love to hear your feedback. Are there any specific changes to the CAS site you'd like to see?

Resolutions to Be a Better Actuary

I’m actually not the New Year’s resolutions-type. Why wait until an arbitrary date to commit yourself to improvement? Instead I make resolutions as they become necessary. This year, however, the two happen to coincide. Here, in no particular order, are my resolutions for my professional life in 2012:

-Obtain Fellowship in the Casualty Actuarial Society. I have filed the necessary paperwork to become an Associate, and I already have credit for Exam 7. Just 9 and 8 left!

-Read one industry news article every working day and one research paper each month (outside of papers for exams).

-Go to sleep earlier and wake up earlier. My office is full of late-risers, but I’ve been working more and more with East Coast folks, not to mention traveling east quite a lot. It’s time to shift my schedule a bit- that, and learn to cope with jet lag.

-Write my blog posts in advance and schedule them to post at 0900 EST on the first Tuesday of every month. Although I barely had to post in 2011, I was late twice, and I’m late again at the start of 2012. Not again!

-Produce two brief research papers, and publish them here.

-Beginning studying earlier. I originally said New Year’s Day for Exam 9, but that didn’t work out. Instead I will assemble my materials today and read my first paper by Saturday. For Exam 8, I will start after the July 4^th weekend.

-Re-read and thoroughly implement David Allen’s Getting Things Done. If you haven’t read this book, I highly recommend it. It’s relevant to far more than just business.

-I’ve said a number of times that I would buy my team at work copies of Getting Things Done if I passed my next exam. This year I will commit to doing this after I pay my taxes.

-File, archive, and otherwise properly keep track of emails. I’m tired of being the person who always has to ask someone else if they have that email or that attachment or some important bit of information. I’ve already filed my email back through September.

-Keep thorough notes on my projects. I’ve often dealt with inheriting projects from people who left the firm. Without adequate documentation, it’s a miserable process.

-Learn Visual Basic. There are a lot of career-improvement things I could commit to, but more and more I find myself thinking it would be really useful if I were fully comfortable with VB code. So this is the year for that project.

-Last, but not least, enjoy my time on the Candidate Liaison Committee. It’s been a great experience so far. I’m wrapping up my first co-authored article now, to appear in the next Future Fellows. If I do obtain Fellowship this year and can no longer serve as a candidate representative, I will try to remain on the committee in another capacity.

What Is the CAS Candidate Liaison Committee?

Last Thursday I attended my first meeting of the Casualty Actuarial Society’s Candidate Liaison Committee. The committee consists of about 20 people who meet four times every year, three times by phone and once in person. This last meeting was the annual face-to-face meet-up in Chicago, IL.

The Candidate Liaison Committee (henceforth the “CLC”) has two purposes: to educate candidates on the path to membership in the CAS, and to represent the views and concerns of candidates to the CAS. I was immediately impressed at how focused and determined the committee members are with respect to both of these purposes. I am one of a significant number of brand new members, however the committee also boasts members with over 15 years of participation on the committee. Considering that many of the members also serve on other CAS committees (and that all of this volunteer work is in addition to their actuarial careers), the level of commitment is admirable.

The CLC is unique among the many, many CAS committees, because it is the only committee that includes people who are not members of the CAS, such as me. Although many of the CLC’s Candidate Representatives are Associates, the designation is not mandatory for this position. The rest of the committee is made up of CAS members and CAS staff. If you’re interested in getting a head start on participating in the CAS, you should definitely consider applying to be a Candidate Representative when positions become available.

As exam-takers know, the main way the CLC communicates with its target audience is through the newsletter Future Fellows. CLC members typically write all of the articles you’ll find in an issue, always with an eye toward what those taking exams need to know about the CAS and the actuarial field in general. Having seen the list of article ideas, I can tell you that Future Fellows will never run out of material; there are many more great ideas than there are pages to hold them. Thus the CLC tries to prioritize articles to provide candidates with the most relevant information possible.

Probably the biggest revelation I experienced at this first meeting was learning how the exam-related committees interrelate. Three committees are involved in the overall exam process: the Education Policy Committee, the Syllabus Committee, and the Exam Committee. The Education Policy Committee is tasked with determining what is required to become an Associate or Fellow of the CAS. For instance this committee would determine which exams are “Associate-level” exams and which are “Fellowship” exams. The Syllabus Committee designs the syllabus for each exam. For instance this committee would select the papers to be included on a particular exam. Finally the Exam Committee is responsible for creating and grading the exams. For instance this committee writes all of the exam questions.

Although each committee has a distinct role, they overlap and connect throughout the exam process. The Education Policy Committee may state what CAS members need to know, but the Syllabus Committee decides exactly which paper or papers will be used to cover that material. At the final step, the Exam Committee decides for a particular exam what questions to ask from that paper, if any.

Although all three committees are part of the exam process, communication between them is not always direct. Each committee has members who serve as liaisons to the other committees. The liaisons attend the meetings of both committees and report back to their committee on what the other committee is doing. To the side of this process is the CLC, with liaisons of its own to all three exam-related committees.

The indirect nature of the communication between these committees is part of the reason why it sometimes seems as if the questions on exams don’t line up with the syllabus, or conflict with the stated intentions of the CAS. Additionally certain things, including exam length, are set by the Board of Directors and are beyond the control of any of the committees. Thankfully the committees are not ignorant of these issues and are taking steps to address them, which leads me to what I found most interesting at our meeting: the addressing of the feedback provided from the exam surveys.

Believe it or not, a lot of people at the CAS are paying serious attention to candidate feedback. I personally read every word of probably close to a thousand responses, partly because many of the comments were highly amusing; all members of the Candidate Liaison, Education Policy, Syllabus, and Exam Committees receive and review the feedback as well. At the CLC meeting, we spent some time discussing the surveys and used the comments to inform our article choices for Future Fellows. The feedback is truly valuable, so thank you to everyone who responded to the survey. I wish I could share some of the more entertaining comments here, however the CAS is very serious about the confidentiality of these responses. It’s certainly important for candidates to be as candid and constructive as possible in their comments.

I was very pleased to discover that many of the concerns I’ve had regarding the exam process and the CAS in general, including many that I had included in my application for the Candidate Representative position, are already being addressed. For instance the CAS website, which (although far superior to the Society of Actuary’s) I’ve always found difficult to navigate, is being overhauled as we speak. One of the key improvements coming soon is a page for each exam that contains all of the information related to that exam, such as the syllabus, past exams, and web notes, all in one place.

Another significant improvement I was relieved to hear about is the ongoing effort to convert more upper exams to the “textbook” format of Exam 5. A textbook for Exam 6 (the regulation and accounting exam) is currently in the works; it is expected to come out next year. There is debate over whether a similar format should be used for Exams 7 through 9, but that question is still being considered. Just keep in mind that it takes time to write a good textbook; changes unfortunately can’t be made overnight.

Going forward the most significant changes to the exams will be the continued move toward textbooks and higher Bloom’s Taxonomy levels. Other changes under consideration include offering Exam 5 twice a year and introducing some type of computer-based format for upper exams, however the CAS has not yet decided whether or not to pursue these options (and if they did, it would be several years before such a significant change could be implemented).

As for what candidates can take away from all this, you should keep in mind that Future Fellows articles are meant to be a resource for you. If you have questions about the exam process, there’s a good chance you can find an answer in the archives, which can be browsed by date or by topic. If you saw the recent survey, you already know that the CLC is considering turning Future Fellows into an online-only publication, which I hope will encourage people to take greater advantage of the archived articles.

If you have a suggestion for the CLC, including articles you’d like to see in Future Fellows, you can submit your comment here. You can also (as always) let me know your thoughts in the comments here. I know first-hand that your feedback is valuable and appreciated, so thanks to everyone who participates.