Blooper Reel

Just like those movie reels where you see funny mistakes or gags that occur when the actors mess things up in filming the movie, the BSDCG got to see a blooper of our own recently.

After receiving his exam results, one of the examinees noticed that the scores listed on the exam results PDF file seemed a bit odd. He contacted us through the website and mentioned that he thought the results were incorrect. Turns out- he was right.


The figure above is from the incorrect exam results PDF that was sent out to our sharp eyed exam candidate. The figure shows, from left to right, the Knowledge Domains on the exam, the candidate score for each Knowledge Domain, and the Group Minimum, Average, and Maximum sub scores for each Domain. The "Group" scores represent the scores for all recent attendees for the exam. It shows the Group Minimum score, and the Average and Maximum scores.

Notice anything unusual? Check the Group Maximum score for "Basic System Administration". It's listed at "17%". Now ask yourself- "Is it really true that the highest score for Basic System Administration could really be just 17%??" especially for an exam that is over one year old?

Turns out- it wasn't. Below are the corrected scores for the same candidate.

As you can see, the Group scores (Minimum, Average, and Maximum) appear closer to what you would expect in the real world. That "17%" for a Group Maximum regarding "Basic System Administration" is really "92%"- reflecting the fact that the group scores are significantly better than what was previously posted.

So.... what went wrong??


Like many system failures, this one was more than just a single mistake. One factor was that we've recently instituted a new exam "form" (separate set of test questions) and our exam scorer decided to roll the results into a new file. However, it's not just one file that has to get updated- there are several files and we missed one.

At the same time, our reviewer who mails out the exam PDFs (uh, that's me) didn't carefully review this set. Actually, I did notice that the scores were strange, but assumed it was the result of the form change.

Once we were alerted to the issue, we rechecked and corrected the procedures we use and reran the exam scoring for everyone who took that set of exams. It turns out that the Pass/Fail status for the exams were not affected- just the individual and Group scores.

For us, this highlights again how important community involvement and feedback is to the success of the exam. In this case, we found the mistake and corrected it immediately. And I'm sure glad we did. It brings the importance of reviews back into focus. You can bet we will all be reviewing all exams in the future much more carefully!

New Video- "BSD Certification Group: A Case Study in Open Source Certification"

Dru Lavigne, Chair of the BSD Certification Group, gives an informative talk about the creation of the Open Source certification system for BSD operating systems. This talk covers the BSD Associate (BSDA) and BSD Professional (BSDP) certifications and describes how they were developed. It presents the value of psychometrics and the process of creating a psychometrically valid exam including

• Job Task Analysis - Determining which skills are frequently used and how important they are
  
• Exam requirements - Determining what should be covered on the exam
  
• SME Question Development - Creating and validating test questions
  
• Beta Period - Creating and distributing a limited version of the new exam
  
• Angoff Session - Determining the final pass/fail score for the exam
  
• Final Launch- Getting the exam out the door
  

The talk also discusses other goals of creating Open Source exams- affordability, local availability, the encouragement of local study groups, commercial training materials, Open Source Testing Centers, and more. Finally, it shows many ways you can help improve BSD Certification and Open Source exams.

The video takes about 35 minutes and is available here, or you can just click on the image below.

Thanks to Patrick McEvoy and the fine folks at Manhattan Neighborhood Network for putting this professional video together, and the Internet Archive website for hosting.

Enjoy!

BSD Professional Job Task Analysis Survey

The first step in getting the BSD Professional exam underway is the Job Task Analysis (JTA) Survey. What is a JTA? It's an online survey where you can help define what skills are important for a BSD systems administrator at the "Professional" level.

The survey presents a collection of tasks organized into Knowledge Domains. Each task includes two rankings- one for Importance and the other for Frequency. These rankings are used in Psychometric Analysis to identify the key skills that should be tested on the exam.

The survey takes about 20 minutes to complete and contains several opportunities for you to express your thoughts on the selection or description of tasks for each Knowledge Domain. Your comments are very important, as they help form the direction and style of the exam.

The survey will be provided in other languages besides English as soon as our translation teams get finished. We hope to have the survey available for 30 days in each language.

As soon as the survey is complete, we'll start compiling the results.

One of the questions asked is about whether the BSD Professional exam should be just a multiple choice exam, just a hands-on lab, or both. We are very interested in your opinion on this issue. There are those who say certifications are useless. But if you can really assess skills, like you can with a hands-on lab exam, this makes the certification verifiable, and provides real value to potential employers.

Let us know what you think. Give the BSDP JTA Survey a go and express yourself.

Update on the BSDA Exam

The BSD Associate Exam is now over a year old! Here are some interesting stats so far:

• 12 Events in all of 2008; 14 events in just the first half of 2009
• Over 1000 people have registered for a BSDCG ID (needed to register for an exam)
• The exam has been held in US, Brazil, Canada, Germany, Japan, France, Denmark, Ukraine, Netherlands, Argentina, and the UK
• So far, 66 people have passed the BSDA exam and received their certificates
  

As of April, 2009, there are 6 exam locations coming up: Canada, Germany, Argentina, UK, Netherlands.

If you know of an upcoming conference near you, let us know (send email to www AT bsdcertification.org) and we'll get in contact with the conference organizers to see if they will provide a time and place for the BSDA exam. We'll bring the pencils, you bring the know-how.

In Other News...

We're in the process of reviewing the exam for the next version. Our psychometrician, Dr. Sandra Dolan, has provided us with a report of the exam results for the first year. BSD subject matter experts, drawn from the BSD community have helped modify some the the questions to increase the accuracy of the exam.

The Winter 2008 DVD is out. This edition includes the QEMU virtualization software (and the AQEMU front end) that enables working with multiple BSD systems at the same time. Additional information on using the software is here [bsdwiki.reedmedia.net]. Purchasing a DVD is an excellent way to prepare for the exam, helps fund the exam development effort, and helps keep the cost of the exam globally affordable (still just $75USD).

Planning for the BSD Professional Exam (BSDP) is underway. We will start with a Job Task Analysis (JTA) survey similar to the JTA we produced for the BSDA exam. The BSDP JTA should be ready in a month or two, and we will announce the survey through all our regular media channels. Stay tuned for the next level.