Accident or murder: object recognition helps Shanghai CSI’s determine cause of death
Labworld approached High Tech Software Cluster member Sioux for image analysis technology to make its forensic analysis products more effective. The technology house discovered in a preliminary investigation that the real gains were to be made in a better workflow. With the use of Machine Learning technology bottlenecks were then eliminated, resulting in significant time savings and efficiency improvements.
In the course of 2018 Labworld from Shanghai knocked on Sioux Technologies‘ door in search of specialised image recognition technology. In doing so, they hoped to improve an analysis process used to determine whether a drowning is an accident or murder. Murder by drowning is relatively common in China due to the absence of firearms. In order to clarify the cause of death in drowning victims, forensic experts use electron microscopes to search for diatoms in samples of lung, kidney or liver tissue. These are single-celled algae found everywhere in natural waters. There are as many as ten thousand different species, all different in shape and size. Due to all kinds of influences, all surface waters have their own specific populations. These can therefore serve as a fingerprint of the crime scene. If the algae in the surface water at the supposed site of the drowning do not match the composition of the water in the lungs, then something is wrong.
The important of speaking the language of the end customers
Labworld knew Sioux from Phenom-World, now part of Thermo Fisher Scientific. The Chinese company supplies Thermo Fisher’s table-top electron microscopes along with its self-developed analysis tools to end customers. These tools are aimed at CSI-like investigations into murders, among other things. Labworld saw opportunities for improvement of the product’s image recognition capabilities. Martijn Kabel, innovation manager at Sioux Technologies: “Labworld thought that automatic image recognition would speed things up considerably. That’s what we thought at first.” Sioux started by mapping out the entire workflow. To do this, an employee of their Chinese branch visited forensic experts in Guangzhou. The provincial police station there is recognised as an authority in the field of forensic investigation based on diatoms. Kabel: “It was very important that our colleague spoke the language and at the same time knew what was possible within Sioux”.
The Sioux employee observed experts creating images manually. Then they identified and counted the diatoms. Using this method it took several days to investigate a single drowning case. “Because we mapped the workflow very precisely, we saw that scanning and analysing samples one by one resulted in too much man-machine interaction. Scaling up was almost impossible. More sophisticated image recognition wouldn’t change that much either,” says Kabel. It became clear that the real gain was to be found in reducing the scanning time.
Object recognition and not image recognition turned out to be the key to saving time
The workflow analysis showed that the assessment process would speed up enormously if the forensic expert was helped with an initial rough assessment: whether there are any diatoms in the image at all. Sioux thought this could be done automatically. The computer then had to make a pre-selection of the images on which diatoms might be visible. Experts would then be needed for the final assessment: whether candidate diatoms were indeed diatoms and of what type.
For the automatic preselection of diatoms, object recognition algorithms based on Machine Learning technology from the world of autonomous driving proved to be useful. Rob Knoops, a mathematical engineer involved in the Labworld project: “We put a signal square around candidate diatoms in the microscope image, just like the object recognition algorithm in autonomous driving puts a square around potential traffic signs and people. The next step is to turn these candidates into a higher resolution image. The forensic expert can then assess whether it is indeed a diatom or not. With this rough preselection you will get misses, but they are acceptable”.
In this way, time was saved in this specific workflow by detecting potential hits at the lowest possible resolution. After that, only higher resolution images were created from potential hits. With this method, less high-resolution images are needed, which reducing the time spent on the electron microscope.
An optimal user interface provides even more improvement
Kabel: “During the optimization of the entire analysis process, we put ourselves in the shoes of the end user and made a user experience analysis. Based on this analysis we developed a mockup and discussed it with the customer. On this mockup several samples were shown with an indicator that shows the progress of the automated scanning process. The concept also showed which subareas had already been analyzed and where potential diatoms were present. We asked the people at Labworld: ‘Suppose we set it up this way, does that fit in with what you expect? During the discussion, what works well for them came to the surface. They also came up with new ideas of their own. For example, we fine-tuned the design of the workflow application until we came up with a well-functioning concept”.
Focus on workflow and value more important than technology
Kabel emphasizes that it is important to focus on the workflow and the value stream itself. Technicians often tend to think from technical possibilities, but often these are not the limiting factors. Rather, these are time and budget. “Therefore, focus on what is important to the customer or your product and try to determine within the time and budget available how you can achieve maximum results by smart data collection.”
Knoops also emphasizes the perspective of the end user: “For the forensic expert it is not necessary at all to make a nice picture of a diatom – although this is technically possible and it produces brilliant photos. It’s not even important to recognise diatoms perfectly automatically. Experts mainly want to know that it is a diatom and of what kind. The working method allows forensic experts to save a considerable amount of time, which means that drowning cases can now be dealt with much more quickly”.