Camera Reviews: Understanding the Face-Off

The Essentials

The face-off gives users a comprehensive overview of how well a given camera works in Auto mode. When manual modes are available, in high-end cameras or SLRs for example, these can sometimes be used to obtain better results than those seen in Auto mode. The face-off is therefore not intended to substitute the full review, as this contains valuable information about the camera's real capabilities.

Our test lab has changed twice over the years, both times when we moved offices in 2007 and in 2010. The first test scene (2004) can be recognised by its white background, the second (2007) by its black background and the third (2010) by its fan of coloured cards. Each time we changed labs, new tests were added and the overall conditions were tightened. We therefore don't recommended comparing cameras tested in different labs as the test procedures are so different that they're simply not comparable.

The Test Lab

Our current '2010' test lab is divided into three test areas: the test scene, the Barbie head and the train set, used for testing a camera's video mode.


The lab is a sealed darkroom that keeps out all external light sources, such as sunlight, which could cause the results to vary. It's fitted with two Kaiser proVision light panels and a halogen lamp, the intensity of which can be adjusted, as well as a series of spotlights built into the ceiling. This ensures that in each test and for each camera, the lighting is kept constant in intensity, angle and colour.

As you can see in the picture above, a set of rails on the floor guides a sliding surface on which the main tripod is mounted. This ensures photos are always taken from the same angle.

All cameras are therefore tested under the same conditions and with exactly the same test scenes.

Overall Scene

The overall test scene features a selection of coloured objects, fine details and a grey scale. This test gives us more technical information about a camera than any other. It's lit by the proVision panels (temperature = 5500 K) set at 150 lux, and is then photographed both with and without the flash.


A shot of the full scene allows us to judge the exposure (does it look too light or too dark?), colour reproduction (do the the colour-chart and the fan look saturated or dull?), the white balance (the grey scale and the circuit board shouldn't have a blue or a yellow tinge, bearing in mind that the lighting conditions are close to natural daylight).

We also take pictures of the test scene without the flash at all sensitivity (ISO) settings. These are then included in our reviews rather than in the face-off.

Details

We extract details from the overall scene photo taken at the lowest sensitivity setting to give you an idea of how accurate the lens is at the widest-angle setting and at a focal length equivalent to around 200 mm. We don't have enough room to shoot from any further away as the test lab is only 4.8 metres long. For cameras that can't reach 200 mm, we take a photo with the zoom on its maximum setting.


For each different-sized detail shot (4" x 6" or 8" x 12", etc.), we extract a section from the centre of the overall test scene photo (the bottle) and a section from the side of the scene (the map) to give you an idea of the accuracy of the lens in different parts of the frame.

Since 4" x 6" is the most common size for photo prints, the detail shots resized to 4" x 6" (10 x 15 cm or 11 x 15 cm for 4:3 format) are intended to show what a paper print this size would look like when printed in a resolution of 300 dpi. The whole image is therefore around 2 Megapixels, which means that the 4" x 6" extracts also more or less correspond to what you'd see if you displayed the whole image at full size on a computer screen (as your desktop background image, for example).


The 8" x 12" (20 x 30 cm) details show what an A4 print would look like—this is the largest format consumers are likely to print (A3 and upwards gets pricey). The 100% details show the level of detail actually captured in the image. This determines the maximum feasible print size and the extent to which a shot can be cropped.

Note that your computer screen probably has a resolution of around 100 dpi, whereas prints are made at 300 dpi. Any faults are therefore around three times more visible on the screen than on a paper print. The face-off therefore highlights any weaknesses in a camera that you wouldn't necessarily notice when inspecting a print close-up.

Overall Scene in Poor Lighting

The results of this test are frequently misinterpreted by users. The low-light scene doesn't show how the camera performs at a high sensitivity setting, but instead shows what 'choices' a camera makes to handle the lack of light.


In this test, the scene is lit with a halogen lamp set at 3 lux. In these conditions, some cameras take a long-exposure shot, sometimes lasting several seconds. Other cameras increase the sensitivity setting, while some capture the darkness of the surrounding environment.

This scene also shows how the camera's automatic white balance works under incandescent lighting (approx. 2000 K). Most cameras take a picture that looks yellow (to varying degrees), and which reflects the surrounding conditions. In other words, it's close to what the eye actually sees. However, other cameras take a very neutral-looking picture, which can sometimes give the impression of a much colder light source, while some exaggerate the yellow overtone to highlight the dusky feeling of the scene.


However, the low-light test scene doesn't show how a camera performs at the highest ISO setting (the ISO tests feature in reviews rather than in the face-off). Note that, more often than not, you can get very different results to those seen in the Auto mode by making a few simple manual changes to the ISO sensitivity, white balance, exposure, etc.

Barbie

The Barbie head is perhaps the most emblematic test of the DigitalVersus face-off. We've been using it since the site was launched and it's been a central part of all our different test procedures over the years. It evaluates the camera's face-recognition function, exposure and, above all, its stabilisation system.

To effectively test a stabilisation system, the camera's exposure time needs to be increased by slowing down the shutter speed. However, this can't usually be adjusted manually in compacts. We decided to get round this by changing the ISO sensitivity, which has a direct impact on shutter speed. We can therefore see at which speed Barbie becomes clear and sharp over the whole frame (out of a series of five images). A very good stabilisation system gives good results from 1/3 sec. In an average system, Barbie looks sharp at around 1/10 sec, and in a bad system the shots still look blurred at 1/20 sec.


Due to the lighting in the '2010' test lab, the Barbie shots tell us something else about cameras too. The test is carried out under light from various sources, namely the ceiling-mounted spotlights and the halogen lamp. The angle and colour temperature of the two light sources are slightly different, which is tricky for cameras to handle in Auto mode. In fact, this confuses the camera's light metering systems and, SLRs in particular, often fall into the trap of over-exposing the image and choosing the wrong white balance setting. This can usually be overcome by switching the exposure mode to 'centre' or 'spot'.


Note that our Barbie actually has a pretty deep tan but most cameras lighten her skin tone. Plus, the outline of her lipstick can be more or less visible from camera to camera, but this unfortunately depends on one of the few things that are beyond our control—the ambient humidity.

Macro

The face-off features two different macro shots, both of which are taken as close up as the camera will go (generally in wide-angle for compacts or telephoto for SLRs).

The component on the circuit board measures 3 mm x 3 mm. We move the camera closer and closer to the circuit board up to the point at which it can no longer focus. We then take a photo and take a full-sized extract from the centre of the shot. In general, the bigger and sharper the chip in the centre of the frame is, the better the macro mode.


The picture of the ruler isn't cut out or resized. It therefore shows the size of the smallest object that can fill a frame in a given camera. If the last visible marker on the ruler is 2 cm, an object that's less than 2 cm long would only ever take up part of the frame.

Video

The video test scene is lit with the ceiling-mounted spotlights and comprises a music box, a metronome and an electronic train set. Note that the metronome has a card with black lines attached to its needle. Filming the scene allows us to evaluate the accuracy of the lens (on certain details of the music box), to see how the camera handles capturing movement and to test how it records sound. The general accuracy of the sound can be judged by the the music box and the metallic noise of the train, and the stereo effect can be evaluated from the train passing by the camera then away again.


An image is extracted from the video at the moment the train passes by the camera. This is resized to 1980 x 1080 pixels (apart from in cameras with Full HD video modes), which shows what the frame would look like displayed on a Full HD TV or monitor.

There's a second video frame of the overall test scene, filmed using the same test procedure as for camcorders. This is then reduced to the size of a face-off photo (390 pixels wide). This frame can be used to judge the exposure in video mode, which can sometimes be different to the exposure in photo mode. It also shows whether any distortion in the wide-angle mode is effectively corrected, as some cameras correct distortion in photos but not in videos.

Responsiveness

The responsiveness charts are pretty self-explanatory and aren't often misinterpreted. One important thing to note is that these values can vary considerably from the values quoted by manufacturers, in particular the speed of the autofocus. In fact, we measure the time it takes for a camera to actually activate the shutter-release: if a camera focuses in 0.2 sec, but then takes a further 0.3 sec to actually take the photo, we take the total time of 0.5 sec. We do the same for the start-up time too, as some cameras are quick to display their welcome screen but take two or three seconds longer to be ready to take a photo.

What's important to users is knowing how quickly you can actually take a photo with a given camera, never mind whether the camera is actually busy focusing or doing something else.

Artificial Test Conditions

We're often asked why we only use artificial test scenes shot in a test lab and why we don't include any photos taken in natural conditions, such as outdoor pictures or photos of a real person rather than a doll.


The answer is simple—we've learned from experience that 'natural' conditions vary too much for the shots to be comparable. The original '2004' face-off did include photos taken outdoors, usually of the scene you can see above. However, we decided to stop using these pictures because the lighting, the height of the sun on the horizon (in different seasons), the pollution levels and other uncontrollable variables caused the results to vary wildly, thus randomly favouring one camera over another depending on the conditions on a given day.


Similarly, taking a picture of a real person may seem like a good idea, but (apart from the fact that your model can go on holiday, be off work sick etc.) from experience we've found that a person's complexion can vary from one day to the next depending on their tiredness, general health, what they've eaten and their age. A photo taken after a heavy day's work can therefore give the impression that a camera gives 'harsher' results than may actually be the case.