Computer vision (CV) has come a long way in the last few decades, growing from an exciting research field into what is projected to be a £20 billion+ industry in 2024. From facial recognition and object detection to autonomous vehicles and more, public and private sector organisations around the world are finding new ways of introducing CV systems and innovating their workflows.
CV automates the extraction of information from visual data. Most of the advances in recent years have been driven by new, more sophisticated software and easier access to high-performance computing.
Researchers can now train machine learning models on vast datasets to recognise and understand visual patterns. This enables models to segment, identify, and understand objects present in an image.
For example, developers do not need to write an algorithm from scratch to define what a banana looks like. Instead, they can unleash neural networks on hundreds of thousands, if not millions, of existing images of bananas until the software learns the recurring patterns of shape and colour common to bananas. With this training, modern CV algorithms can identify bananas in new images.
While these algorithms demonstrate impressive results, they require significant compute and long, time-consuming training processes. Plus, they can struggle to differentiate between visually similar objects and can also fail to recognise objects in unfamiliar contexts or when only partially captured.
What if there was a way to shift the burden of performance from the software to the hardware? What if more information was available in the image to start with, beyond just basic colour and shape?
What is spectral imaging?
Spectral imaging refers to analytical techniques that combine both spatial and spectral information into a single image. This means measuring light using more bands or channels than traditional RGB imaging.
A standard camera uses a Bayer filter, grouping light into just three broad channels: red, green, or blue. This mimics how the human eye works, and traditional imaging recreates colour images similar to how we see the world.
But there is no reason to restrict modern technology to human biology. Grouping light into three broad channels throws away most of the data it contains. Just because we can’t see it with our own eyes, doesn’t mean that a computer can’t make good use of this information that otherwise ends up on the cutting room floor. Spectral imaging uses cameras with more than three channels so we can investigate light in greater detail, seeing more than meets the eye.
So, what exactly is there to see?
Materials reflect and absorb light at different wavelengths depending on their chemical properties. Therefore, assuming a broadband light source (a source that contains a wide range of wavelengths, like sunlight) is used, it is possible to detect spectral signatures in reflected light and reveal specific, characteristic information about the materials present.
CV has learned to segment and identify objects algorithmically from RGB images. In contrast, spectral imaging provides data in the image itself to make that work trivial and even deepen the use cases.
Rather than applying compute-intensive algorithms to a 2D array of RGB data, spectral imaging gives developers a greater number of channels (at least 25 – our latest camera has 96) to differentiate between objects in the image more easily.
Think of it as increasing the resolution of the camera, but spectral resolution, not spatial resolution. Rather than covering the visible spectrum of light (400nm – 700nm) with only three channels, spectral imaging adds many more channels to learn more about the light hitting the sensor. Spectral imaging often extends beyond the visible region to identify additional signatures in the near infrared for example.
Incorporating this data into CV offers a range of potential benefits across a number of industries, including:
- Learning about plant health to improve agricultural productivity
- Improving healthcare diagnostics in areas such as skin cancer and burn assessments
- Revealing food quality and ensuring products are safe to be consumed
- Enhancing archaeological data and identifying new sites of interest .
Although spectral imaging is a broad field, it can be divided into two main categories hyperspectral and multispectral imaging.
Hyperspectral vs multispectral imaging
Trials have included testing in rural, urban, and coastal environments using the Living Optics hyperspectral imaging camera to track moving objects outside controlled laboratory conditions. Proving the viability of this technology, opens the door to a range of benefits, including the detection of signatures not previously seen, greater confidence in identifying signatures of interest, and the potential to deploy trustworthy autonomous systems that reduce the exposure of human users and increase the precision and reliability of decision making.
Video-rate tactical hyperspectral image analysis has only recently emerged as a viable sensing approach to overcome the challenges of national security use cases. Therefore, the field is ready for exploration with significant scope for early adopters to reap the potential benefits of hyperspectral imaging technology.
Spectral imaging in practice
Both multispectral and hyperspectral cameras are used to provide greater insights for imaging applications. While they may seem to be in direct competition, the application governs the choice of which technology to use. Each technique can offer advantages depending on the scenario, the user’s pre-existing knowledge, and the required level of data.
With more channels and higher spectral resolution, hyperspectral cameras are better suited to exploratory work. Situations where the user does not know what spectral signatures they are looking for. Hyperspectral cameras also provide greater flexibility with a larger dataset, enabling the user to sort for new materials and investigate the relevant bands needed to distinguish between similar spectral responses.
In contrast, multispectral imaging cameras generally offer a cheaper and faster alternative for identifying disparate signatures in controlled use cases where the user knows what to expect. High spectral resolution over a large range is unnecessary for many applications, and a multispectral system can be optimised to identify specific, known signatures, even across multiple spectral regions.
However, this requires the user to know the spectral bands to record to distinguish between the different materials they are imaging. Again, hyperspectral imaging may be required to perform exploratory preliminary work to better understand the spectral requirements.
Three examples highlighting the difference between multispectral and hyperspectral cameras are discussed below:
- Food inspection – The spectral resolution required to distinguish between different foods and their quality can be lost when utilising a multispectral imaging camera. In specific scenarios, it may be possible to optimise a multispectral system to target a smaller number of known spectral responses related to food quality. This could increase the speed and reduce the cost of the system but would be vulnerable to edge cases and the emergence of new or refined signatures of interest.
- Remote sensing – Multispectral imaging can identify features and patterns and even return data to characterise materials. However, confidently identifying materials and inferring chemical processes requires the higher spectral resolution of a hyperspectral camera.
- National security – This industry has the added complication of new and improving counter-detection systems, with adversaries actively trying to conceal material signatures that reveal information about their activities. For example, a camouflage net that deceives typical CV systems (indistinguishable from its surroundings using RGB data alone) may also deceive a multispectral system by matching the broadband properties of the environment. In these situations, a high-performance hyperspectral camera is likely to be able to distinguish these man-made materials from natural environments.
Living Optics
One of the main benefits of multispectral imaging cameras is their ability to return data quicker. Spectral imaging has struggled to deliver real-time hyperspectral frames until now.
The Living Optics snapshot hyperspectral camera bridges this gap, using advanced compression techniques to return video-rate high spectral resolution data. Get in touch today to learn more about our technology and see how the Living Optics development kit could transform your CV workflows.
