Brainbench

Like many people, I've been looking for work recently. I'm a contract software architect (developer) by trade, and, for the second time in my career, have been asked to take a Brainbench evaluation in C++. For the second time, I've been terribly unimpressed. Here are some notes about the process; hopefully they'll help others approach the tests successfully.

The Concept

The concept, obviously, is to assign a competence rating of some sort to software developers. Such a rating could be useful when determining who to hire, who to promote, who to train, or who to fire. Such a rating would be of particular value to managers that aren't actually technical and are therefore unable to make such distinctions themselves…and have nobody they trust to make the determination for them. Personally, I believe this is a very small subset of the managers looking to hire technical talent.

Having said that, metrics and superficially objective decision making has a strong following in any bureaucracy. Brainbench appears to be a way to apply this concept to people.

The Implementation

As nearly any professional software developer will tell you, the concept of a high-level language such as C++ is to make life easier. Higher-level languages give you more tools for abstraction and greater ability to segregate and decompose the problem set, thus, at least theoretically, making the job of developing software easier. Languages are supposed to be consistent and relatively easy for a professional to understand; they're not designed to be a puzzle.

While this sounds like a good thing for the world of software developers, it poses a challenge for someone trying to evaluate competence. Basic competence is, well, common. Nearly anyone who has taken a college-level programming course that deals with C++ can answer 95% of the issues that arise when using the language; questions that address simple syntax or structure won't differentiate any but the very most inexperienced candidates. As a result, Brainbench appears to take the only real alternative — they test primarily on the obscure details and what I'll call “tricks.”

So, you might ask: “What is an obscure detail?”

They're the things you essentially never use in the language, either because they're extremely specific in their application or because they're generally considered a bad idea. In other words, things that are uncommon in real-world code.

Examples

A few examples, along with some narrative, follow. These examples are not copied directly from the tests I took, though they are similar in approach to those I saw when I tested.

int + char = ?
If you care, you cast. In other words, if this is a significant issue in your code, you're doing something wrong. You shouldn't assume that the next person to come along will understand type promotion as well as you do. Incidentally, the result is an int.
How do you declare a pointer to a class method?
Seems like a reasonable question, right? Except then you remember the concept behind encapsulation: you really don't want to look at the implementation of any method in a class hierarchy, since that can really mess you up if you're handling an object of the wrong type. In other words, this is just something that's conceptually impure…and likely to lead to problems when you need to maintain the code. If you really, really need to do this, remember that it's not void (*class::method)(int, int), the way you might expect — it's void (class::*method)(int, int). To me, in addition to asking about a questionable practice, it's also a trick syntax question.
What does the “mutable” modifier do?
This is actually a reasonable, but somewhat obscure, question. It's used on two ways; one that strikes me as poor practice, and one that is important but uncommon. The mutable modifier makes a member variable non-const under all circumstances; it forms an exception even when the surrounding object has been declared const. This is, in my opinion, a Bad Thing To Do; it changes the state of an object that is supposed to be stable. On a positive note, this can be very important as an optimizer hint when dealing with hardware registers. In such a circumstance, you're really telling the optimizer to make sure it reads the memory location each time it's accessed, and not to make the assumption that the value hasn't changed. So 50/50 here, in my book; the sample in the test is an example of misleading coding practices, but the modifier does have value in specific circumstances.
What is a null namespace and how is it used?
This is an obscure question. Null namespaces, or namespaces that haven't been given a name, are recent additions to the C++ specification and are unfamiliar to many C++ developers. The functionality wasn't even supported prior to version 4 of g++. Since gcc 3.4.6 is distributed with RedHat Enterprise Linux version 4, this functionality is not widely available to UNIX/Linux developers yet. It's also one of those very strange things that periodically make it into the language; I'm sure there's a good reason that it's there, but I certainly can't think of it. Null namespaces provide another mechanism for variable scoping; any variables declared in the un-named namespace are local to the file where they were declared, as if they were static. They are accessed as if they weren't in a namespace at all.

Strategy

To do as well as possible on a Brainbench test, at least the C++ evaluation, I suggest that you use at least two and preferably three computers. The first must be a Windows box. This machine is the one you'll actually take the test on, since Brainbench won't work with Linux or MacOS X (at the time of this writing, anyway — interesting for a company that also has Linux competence evaluations). The second (and third, if you're able) computer(s) should be dedicated to answering questions that arise during the test; one should be running a web browser so that you can quickly Google for answers, and the other (if available) should be set up with an editor in one window and a command line. This last machine will be used to test simple one-line programs for validity. You should also have a copy of “The C++ Programming Language” (by Bjarne Stroustrop, though I shouldn't have to mention that) on the desk next to you, and perhaps a little bit of scratch paper if you think best that way.

The test — again, as of this writing — is set up as a series of questions presented one at a time. You cannot go back to a previously answered question. You have three minutes to answer each question, which is presented in multiple-choice fashion.

When taking the test, keep in mind that the test results include not only an indication of whether your answer was right or wrong, but also the amount of time you took to answer the question. I suppose there's probably some concept that the speed with which you answer the questions is related to your skill level; I don't know for certain. For myself, I deliberately waited after answering many questions. I'm not an effective test-taker. One of my shortcomings is a tendency to read the questions too fast, thus missing subtleties (and sometimes, not so subtle issues). My advice for myself was to slow down, but that may or may not be an effective strategy for others.

So How Did you Do?

I did fairly but not terribly well. To quote:

Score: 3.66

Proficiency Level: Advanced (3.52 - 4.50)

The candidate has mastered the basic concepts of C++ (Fundamentals of OOD, C++ Class, OOP) and intermediate C++ concepts (C++ Data Types, Basic Structure and Syntax, Class Member Functions). The candidate is likely proficient with more advanced C++ concepts such as:

At the Advanced level, the candidate will be capable of working on projects involving C++ and will be capable of mentoring others on most projects in this area.

According to Brainbench, this put me in the 80th percentile, for whatever that's worth. For the record, I didn't try compiling anything — that was suggested to me later by an interviewer at the company that asked me to take the test.

Conclusion

In my mind, this all adds up to a useless test developed to support a false premise: that people can be “graded” according to their value based on their performance in a specific test. I believe that the Brainbench C++ evaluation does nothing to determine whether a candidate is or is not capable of logical thought or competent software development.

It was also my impression that this test rewards three behaviors that don't benefit anybody in the real world: the ability to memorize a long language specification and remember each exception to the general rules, the ability to quickly enter a test program to determine whether syntax is valid or behavior is as anticipated, and those who are good with Google searches.

To me, it also seems that this test is easily “gamed;” that is to say, it appears to be easy to manipulate the results without breaking one of the few rules that I saw: Don't get help from another person. The rules state that the test is “open book” and reference materials are explicitly permitted. As a result, it seems to be more a test of how quickly you can put together a test program or search for accurate phrases on the Internet than about an understanding of the C++ programming language.

I do, however, believe very strongly that being requested to take this test speaks volumes about the priorities and capabilities of the firm attempting to evaluate your skills through such means. Even if the firm uses such a test as part of a battery of evaluations to determine your skills, I would come away with a very negative impression of their ability to implement a solid hiring process and would question their ability to focus on providing a positive, rewarding work environment.

Search

Static links

RSS Feed RSS Feed

About this site

Our host

Dynamic links

None