How does AI work in televisions and what can it do?

Many modern televisions ― it's like they're not really TVs anymore. When there is a processor, RAM and a small storage on board, it is more like a computer. However, with the introduction of AI, most major manufacturers made sure that a separate processor designed for this purpose was used to improve the picture on the TV. LG uses Alpha series processors, Samsung has Crystal and Quantum chips, Sony X, and Phillips has P5. Next, we will look at the advantages and disadvantages of the most popular chips, but for now we will return to the question of what exactly and how they improve.

The development of big data and machine learning has opened up new features for manufacturers to automatically improve the image. The processor is able to determine exactly what is happening on the screen, then increase the brightness or contrast of the entire image, locally highlight or darken certain areas of the image, increase the resolution, and so on. And this is only at the most basic level. For example, the flagship LG Alpha 9 chipset is able to analyze environmental conditions, adjusting brightness and contrast to it, and Samsung's Quantum processors are very good at upscaling, up to 8K.

Learn more about how AI improves sound and picture

The power of flagship processors like LG a9 and Philips P5 allows them to analyze and improve the entire image or some of its individual parts in real time. First, the processor determines what is being displayed on the screen right now. For example, a landscape with a sunset over Tokyo, racing cars tearing to the finish line, a pair of fighting rhinos or the shining biceps of Dwayne Johnson. The algorithms were trained by running through hundreds of thousands of images so that they could better understand what they were looking at (animals, sports, landscapes, etc.), and then process the image according to this classification. In general, there is no fiction in this, we have been using similar recognition algorithms in smartphone cameras for several years.

Actually, after that, adjustment algorithms come into play, which choose the optimal indicators for noise reduction, increasing detail, sharpening, adjusting brightness, gamma, etc. Each manufacturer may have different names and differ in details, but in fact it all boils down to several similar processes that they take place in several stages.

Important: like AI in smartphone cameras, algorithms in televisions do not always make the picture better. Firstly, because they are still far from perfect. Secondly, because many people's ideas about the perfect image may differ markedly.

Upscaling

The obscure term upscaling actually hides a rather simple process of artificial image scaling. So that when you buy a 4K TV, you can normally watch your favorite movie or TV series in this very 4K. Otherwise, why do we need such a TV at all? The idea of upscaling is not new, you could meet it in some DVD players from BBK, Sony PlayStation 3 console and numerous spy thrillers in which the hero looks at a photo from a surveillance camera with giant pixels, then says "computer, zoom in" and we immediately see the features of the villain in the photo. The first 4K TVs were created without AI processors and did not shine in this regard either, simply scaling SD videos to the entire area of the matrix with a simple pixel magnification.

Upscaling technology in modern TV allows neural networks not only to stretch the picture, but also to finish drawing the missing pixels, smooth out irregularities, remove unnecessary noise and blurring of objects that are in motion. The end result in some cases looks almost perfect, in others it is tolerable. It depends both on the specific TV and processor, and on the type of content. Therefore, many manufacturers optimize scaling algorithms for different types of content (movies, sports, games) in order to get the best result when viewing this or that content. On our own, we add that during the tests of the next consoles, we compared the primitive 4K resolution in the Xbox Series X with an upscale to 4K in the younger Xbox Series S. And the difference was visible only if you get closer to the TV than a metre. We compared on the Samsung QE-55Q77TA panel, and played Gears 5 and Battlefield V.

TVs with Upscaling up to 4K

Improved brightness and contrast/sharpness

After dealing with 4K, TV manufacturers began to popularize HDR content with an extended dynamic range. The discussion moved from pixel size to statistical/peak brightness and contrast indicators. As a result, each more or less intelligent processor now has several similar algorithms in its asset, which are radically different in their essence and final result. One TV simply tightens the brightness and increases the contrast (and this is a so-so improver), the second does something, but it's unclear what, and the third almost turns ordinary SDR content into HDR, frame-by-frame conjuring over a dozen settings. Accordingly, the effectiveness of such algorithms depends both on the processor itself and on the content that needs to be improved. In some cases, the difference will be cardinal, in others barely noticeable, in others you will want to turn it off as soon as possible.

For example, the flagship Sony X1 Ultimate 8K processor uses an advanced Backlight Master Drive control algorithm with the ability to control individual LED modules. The flagship LG a9 is able to apply the Quad Step four-stage noise reduction algorithm to the image, which corrects errors made at the video encoding stage. And Samsung's Quantum 4K chip analyzes each area of the TV panel scene by scene, adjusting the backlight level in each zone.

TVs with brightness and contrast enhancement technologies

Colour improvement

Okay, the picture is now in 4K, the brightness and contrast are optimal, but something is still missing. Wait, we haven't gotten to the colour rendering improvers yet. This is perhaps the most interesting and at the same time ambiguous point. The so-called "juiciness" of the picture is influenced by many points, including the same brightness and contrast, and manufacturers do not have a consensus on these issues, so there is a lot of marketing in the topic of colour improvement and little specifics about what and how it improves.

For example, Sony's flagship TVs use 14-bit hue gradation even for 8-bit sources, plus there is a Super Bit Mapping 4K HDR feature that minimizes the effect of hue segmentation characteristic of conventional TVs. In fact, algorithms optimize for each colour in real time, which ensures a natural image with smooth transitions between halftones. In LG TVs, you can find a combination of two algorithms: Advanced Colour Enhancer artificially enhances colour reproduction, and True Colour Accuracy corrects it so that everything looks natural. And Samsung's H-series TVs use the Purcolor algorithm, which splits pixels into smaller parts and paints them in the most appropriate colour. Unlike most colour improvers, Purcolor works not only with primary colours (red, green, blue), but also with additional colours (magenta, cyan and yellow).

TVs with colour enhancement technologies

Enhancing the depth of black

LED displays of the past were not particularly able to work with black. It was possible to unscrew the black level slider in plus to highlight details in the dark areas of the image, but then the image seemed unnaturally overexposed. Or, on the contrary, remove it in the negative, so that the black colour looks really deep and black. In this case, the readability of other elements of the image was lost, and dark scenes looked as if they were filmed in a well.

With the advent of AI, QLED and OLED panels, as well as numerous improvers, TV manufacturers have seriously taken up black, realizing how important it is for the perception of the whole picture. It is not so easy to achieve this, so there is no universal recipe yet. Local illumination is most often used, when each LED of the screen is illuminated individually, and unnecessary areas are not highlighted at all, allowing you to leave the black colour black.

TVs with black colour display enhancement technologies

Noise reduction

In some TVs, processors have learned to artificially reduce the noise level. For example, the Sony archive has a giant archive with tens of thousands of reference recordings collected over the years of work on the production of films and TV shows. It is based on these databases that Sony builds its neural networks-improvers. It works like this: the processor "sees" the image on the screen and checks with the database. It's a poodle. It's a car. And this is Cristiano Ronaldo. Then the cleaning takes place in two stages: artificial intelligence checks first with one database, reducing the noise level, then turns to the second database to give the edited image the necessary clarity.

Intelligent sound processing

Artificial intelligence algorithms have learned to work not only with images, but also with sound. There are a lot of implementation options. The most banal is to decompose the stereo signal into 5.1 virtual channels. The more difficult task is to adjust the sound by genre. According to a number of indicators, AI can try to determine the genre of video content (cinema, TV programme, music), and then optimize the sound so that it enhances the impression of the audio sequence. For example, to make the voice of the news anchor louder and clearer. Add "weight" to explosions and special effects during the final battle in the conditional "Avengers".

And these are just flowers. For example, Samsung panels (for example, QE-65Q95TA) use the so-called object-oriented sound (OTS) function. The AI analyzes the movement of objects in the frame and determines which part of the screen the sound comes from, after which, with the help of built-in speakers and soundbar, the sound is focused in the desired area of the screen. Simply put, if a boat is floating from the upper-left corner of the display to the lower-right corner, then the sound of the motor will "float" after it, and not be in static, as in conventional televisions. LG's flagship models like the OLED65C9 use microphones built into the remote control to analyze the acoustic scene of the room where the TV is, and then redistributes the sound stream to create a 360-degree immersion effect.

Smart Home Management

Well, do not forget about smart functions like content recommendations, automatic switching of operating modes or control of "smart" household appliances using a voice assistant. For example, you can use the TV to start a robot vacuum cleaner or turn off the smart stove in the kitchen. Thus, the TV becomes a new chapter in the smart home, replacing the not particularly practical smart speakers in this role. However, the management of smart light bulbs, the Internet of things and smart homes ― this is a topic for a separate conversation.