In-depth understanding of Camera basic knowledge points (2)

3. Introduction to mobile phone cameras

Insert image description here

 

For the camera system on mobile phones, due to size and power consumption limitations, it cannot be like a professional camera. In order to ensure the imaging effect, it can be easily replaced with a larger lens and a larger CCD/CMOS photosensitive device can be added. A more powerful image processing module has been incorporated, so the space left for mobile phones is not very large. However, even so, major mobile phone manufacturers still use limited space and battery life to make the camera system comparable to professional cameras in some fields. Next, let’s briefly introduce this camera system that is small in size but has high power.

As shown in the figure, the camera system of a mobile phone can be divided into two parts, one is the camera module, and the other is the image processor ISP. The camera module is used for photoelectric conversion, and the image processor is as introduced before It is used for image processing. Next, let’s take a look at how the two run on the mobile phone.

 

1. Camera module
Due to size restrictions, mobile phone camera modules are often made very delicate and compact, which mainly include lenses, focus motors, filters and sensors.

 

Lenses in mobile phones generally use a combination of multiple lenses to eliminate chromatic aberration. Lenses in mobile phones are no exception. Most of their materials are a combination of glass and plastic. Plastic lenses have lower costs and are suitable for low-end applications. For the camera system in the product, glass generally has higher imaging quality, but the cost is slightly higher than that of plastic lenses, so it is often used in some mobile phones that pursue imaging quality. At the same time, the lens mainly has the following parameters:

Field of view FOV, this parameter indicates the range of scenes that can be imaged through the lens. Generally, the larger the FOV, the wider the range of scenery that can be seen, but it may cause serious distortion. Post-stage distortion correction algorithms are usually used to correct the scene. Correct distortion caused by large FOV.

 

Focal length F stipulates that the point where all light rays parallel to the main axis of the lens converge is called the focus, and the distance from the focus to the center of the lens is the focal length here. Generally, the larger the focal length, the smaller the FOV of the lens. The shorter the focal length, the larger the FOV.

 

The aperture value f is specified by the ratio of the focal length of the lens to the diameter of the actual aperture. The smaller the value, the greater the amount of light entering. Mobile phone lenses generally use a fixed aperture of f/2.0.

Next is the focus motor, which is mainly used in mobile phones. In order to facilitate the adjustment of the lens, the entire lens is usually integrated into the motor module. The motherboard transmits instructions through the I2C bus to drive the motor. To move and adjust the lens to achieve focus or zoom, here we briefly introduce the voice coil motor.

The voice coil motor is called a voice coil motor in electronics. It is called a voice coil because its implementation principle is similar to that of a speaker. It is inside a permanent magnetic field and changes the DC current of the coil in the motor. It controls the lifting position of the spring leaf, and then drives the lens to move up and down to achieve focusing or zooming. Due to its high sensitivity and high precision, it has become a mainstream focusing component for mobile phones.

On mobile phones, the use of voice coil motors is generally divided into two modes, one is zooming and the other is focusing. The principles and purposes of the two are different.

 

Zoom: The motor adjusts the movement of a certain lens in the lens group, thereby changing the focal length of the entire lens, causing changes in the field of view, thereby achieving the purpose of zooming in and out of the scene. This method is what we often call optical zoom. The advantage of this zoom method is that image details will not be lost during the process of zooming in. However, the disadvantage is also obvious. Due to the limitation of volume, it is unable to perform a wide range of optical zoom. Therefore, mobile phone manufacturers generally use a combination of optical and digital zoom. method to achieve high-range zoom purposes.

Focus: The voice coil motor directly moves the entire lens back and forth to make the image plane of the object coincide with the photosensitive plane of the photoreceptor, thereby obtaining a clear image. This method is the process of focusing. The purpose is to get a clear image.

 

After the light passes through the lens, it will first enter the next component - the filter. This part will further process the light and has two main purposes:

Filtering infrared rays: Since the photoreceptor will sense part of the invisible infrared rays, which will interfere with subsequent image processing, a filter is required to filter out this part of the infrared rays and only allow visible light to pass through.

Correcting light: After the light passes through the lens, it is not all parallel and vertical to the photoreceptor. There are also many light rays that are not direct. Obviously if it is not intercepted, it will cause certain interference to the photoreceptor, so the filter is used The physical polarizing properties of quartz retain direct light and reflect oblique parts to avoid affecting nearby photosensitive points, further improving the imaging effect.

 

After filtering and correction by the filter, the incident light has a certain stability. At this time, it is necessary to perform photoelectric conversion through the core photoreceptor of the camera system.

 

Photoreceptors on mobile phones mainly include CCD and CMOS. However, due to their high cost and large size, CCD is not used much in mobile phones. CMOS has become the mainstream photoreceptor in this field. CMOS on mobile phones still uses a three-layer structure. , microlens/filter/photosensitive layer, the specific definition is as follows:

The microlens layer is mainly used to expand the light-receiving area of a single pixel.

The Bayer mode used by the filter is similar to the RGB mode. It uses RGB color components to measure the gray value of the three channels of each pixel. However, based on the basic rule that the human eye is more sensitive to green, Bayer The mode further emphasizes the green component, thereby defining the green components as Gr and Gb respectively to better express the color and brightness of the image.

The photosensitive layer is used to convert photons into electronic signals, and then convert them into digital signals through the amplification circuit and the analog-to-electrical conversion circuit.

 

The core of the photosensitive layer is a photosensitive diode, and each diode contains an amplifier and a digital-to-analog conversion circuit. Since each photosensitive element has an amplifier, although it speeds up reading to a certain extent, it cannot guarantee that the amplification effect of each amplifier is consistent, so this design will bring possible noise. In addition, since CMOS adds additional hardware circuits next to each diode, it will inevitably reduce the photosensitive area, so this design will affect the overall photosensitive effect. This design is called front-illuminated. In order to solve this problem, CMOS manufacturers have introduced a back-illuminated design. This design places photosensitive pixels and metal electrode transistors on both sides of the photosensitive sheet, which increases the pixel duty cycle, increases light sensing efficiency, increases the number of pixels, and improves the signal-to-noise ratio. The imaging effect is greatly improved.

 

2. Image processor
The implementation process of the image processor on the mobile phone is basically similar to that of the non-mobile phone camera system. For the ISP of the Qualcomm platform, it mainly includes hardware modules such as IFE/BPS/IPE/JPEG. They are respectively responsible for non-pass image processing tasks. Next we will briefly introduce them one by one:

IFE (Image Front End): The data output by the Sensor will first arrive at the IFE. This hardware module will do some color correction, downsampling, and demosaicing statistics for preview and video 3A data processing.

BPS (Bayer processing segment): This hardware module is mainly used for dead pixel removal, phase focusing, demosaicing, downsampling, HDR processing and Bayer's hybrid noise reduction processing of photographed image data.

IPE (Image processing engine): This hardware is mainly composed of NPS and PPS, and is responsible for image processing tasks such as hardware noise reduction (MFNR, MFSR), image cropping, noise reduction, color processing, and detail enhancement.

JPEG: The storage of photo data is performed by this hardware module for jpeg encoding.

Compared with professional cameras, the audience for mobile phone cameras does not know much professional photography knowledge, but this group has a characteristic that is obviously different from the audience for professional cameras, that is, they pay more attention to the portability and playability of the camera. Needless to say about portability, mobile phone cameras as a whole are famous for their compactness, but in terms of playability, major mobile phone manufacturers have also taken great pains and adopted many strategies to expand the playability of cameras, among which multi-shot is a comparison Classic example.

 

Early mobile phone cameras generally had a single rear camera and were popular all over the world. Their functions were relatively simple. Later, with the development of the times and the increasing number of young users, the demand for selfies became stronger, and there were also breakthroughs in technology in this field. . Therefore, mobile phone manufacturers took advantage of the trend and launched a dual-camera mode, adding an additional camera module to the front of the phone mainly for selfies. They also innovatively added a beauty algorithm to the ISP, which greatly improved the selfie image effect. Immediately afterwards, mobile phone manufacturers integrated multiple modules into mobile phones to meet the photography needs of multiple scenes. Next, we will briefly introduce the multi-camera system.

 

Today's mobile phone cameras often use multiple camera modules. There are specialized macro modules for shooting miniature landscapes, wide-angle modules for shooting wide-angle scenes, and dual-camera systems developed to meet specific needs. Due to the rapid development of dual-camera technology, many mature solutions have been produced.

 

As the name suggests, dual-camera technology uses two camera modules to image separately, and processes them through specific algorithms to fuse them into one image to meet specific imaging requirements. Generally speaking, the current dual-camera solution is mainly used to achieve background blur, improve imaging quality under dark light/night scene conditions, and optical zoom. The following is a brief introduction:

a) Background blur (RGB + RGB)
In order to achieve this purpose, two RGB camera modules are mainly used to image the scene at the same time. The triangulation principle is used to calculate the depth of field data of each point. Relying on this series of data, the foreground and background are separated, and then through The blur algorithm targets background blur processing and ultimately creates a blurred background imaging effect. It is worth noting that due to the limitation of the principle of triangulation, the two camera modules need to be calibrated so that the imaging planes of the two are on the same plane and the pixels are aligned.

b) Dark light enhancement (RGB + MONO)

In a darker environment, the shooting effect is often unsatisfactory, so mobile phone manufacturers use an RGB and a black and white camera module (MONO) to improve the dark light imaging effect. The specific principle is that because the black and white camera module There is no Bayer filter, so in dark light conditions, more light can be obtained, thereby preserving more image details. Coupled with the color component of the RGB camera module, this can be better guaranteed. Regarding the imaging quality under dark light, since the imaging of the two camera modules also needs to be fused, a calibration operation is still required so that the two camera modules can maintain pixel alignment.

c) Optical zoom (wide angle + telephoto)

Optical zoom, as introduced before, can achieve the purpose of zooming by adjusting the focal length of a single lens in the focus motor. However, due to volume limitations, it is often impossible to obtain a larger zoom range from a single camera module. Therefore, mobile phone manufacturers have proposed using two camera modules with different focal lengths (wide-angle and telephoto) to jointly achieve the purpose of optical zoom. The principle is to use the wide-angle module to present a wide range of scenes, and to use the telephoto module to see more For distant scenes, module switching and excellent fusion algorithms achieve relatively smooth zoom operations when taking pictures.

From the above introduction, we can see that a camera system is composed of a lens, an aperture shutter, a photoreceptor, and an image processor. In order to improve its imaging quality, filters, focus motors, flashes, etc. are gradually added during the development process. components. At the same time, in order to embed the camera system into the mobile phone, certain hardware cuts are inevitable. For example, the aperture is often abandoned in the adjustable form and adopts a fixed aperture. In addition, due to size and battery life limitations, mainstream photoreceptors on mobile phones mainly CMOS is used, and the focus range of the focus motor is also reduced due to volume limitations. But even if the hardware is subject to considerable limitations, through the continuous development of image processing chips and the continuous optimization of algorithms in recent years, mobile phone camera systems are actually gradually narrowing the gap with professional cameras. I believe that in the near future, the imaging effects of mobile phone cameras will be completely Comparable to professional cameras.