Innovation in 3D bin-picking thanks to Future Experience Center

3D bin-picking is a technique whereby a robot knows how to recognize and pick up unsorted parts from a bin. A technique that combines robotics, software, camera and image processing. This is an efficient addition to the manufacturing industry as it means that monotonous tasks no longer need to be carried out by people. However, bin picking does pose some difficulties. For example, a robot’s arm is sometimes unable to reach objects properly, it has difficulty grasping flat objects, and the system has difficulty distinguishing shiny metals or metals smeared with a bit of oil.

Fontys students are working on solutions to those problems at the Factory of the Future Experience Center (formerly Fieldlab Flexible Manufacturing) on the Brainport Industries Campus (BIC) in the Netherlands. They do this together with Affix Engineering from the town of Best, and Yaskawa, which is located on the BIC.

Digital Twin

Students have built a system whereby a sophisticated camera system scans the objects in the bin and creates a 3D image of them. That camera works with a dual laser, so it can also see shiny objects. The software-controlled robotic arm grabs the objects and puts them in a different spot. So that they can be assembled, for example.

The software also helps prevent some practical problems, says Danny de Greef of Affix Engineering. “For example, the software makes sure that positions that were previously unreachable are also covered. The robot is completely incorporated into the software.” The setup is now at a point where the next group of students can start testing whether objects from other companies would stand to benefit from it as well. The software can also create a digital twin of the setup, bin, parts, robot arm and gripper. This way, the setup can first be extensively tested, simulated and optimized before the required investment in hardware has to be made.

This project would never have come about without the collaboration on the BIC grounds, De Greef says. From 2010 to 2019, Affix mainly programmed robots for the automotive sector. De Greef: “In addition to programming, we build the link between the technology that is needed, the hardware and the software that makes the movements possible.” The team at Affix helps companies purchase the hardware and install it at these companies.

Dual laser camera

One of the things De Greef did was go to Silicon Valley to help with the manufacturing process of Elon Musk’s Teslas. “The automotive industry is fantastic, but we also wanted to help other sectors further along with new technologies.” That is why De Greef conducted market research into the needs of companies within the Brainport region. This demonstrated that machine builders would like to do something with robots, but they lacked the knowledge to do that. “Training someone in this is often prohibitively expensive and the right people can’t be found,” says De Greef.

As managing director of Affix, De Greef wants to be able to offer high-end techniques to small and medium-sized enterprises (SMEs). High-end techniques include the collaboration between a robot or cobot, a camera and 3D technology – which is what 3D bin picking actually entails. Since De Greef wanted to have a system up and running quickly, he opted for an existing camera configuration that has already proven itself in practice.

He eventually found that system in Italy, after a global search. The software developed by IT+Robotics from Padua provides the solution to most of the problems. In part because it works with a dual laser camera, this system is less sensitive to reflection. “That’s a problem that crops up with a lot of metal products,” he says.

It was through John Blankendaal, managing director of Brainport Industries, that De Greef came into contact with the Experience Center. A project is underway there as part of the Factory of the Future innovation program. There are other kinds of field labs at BIC. Apart from robotics, there are als field labs for: information exchange in the chain, multi-material 3D printing, industrial metal printing, software, education, logistics and digitalization.

Generic issue

An engineer from Yaskawa, which develops robots among other things, then trained the students on how to use the robotic arm. The company is located practically next door to the Experience Center. Eddie Mennen, managing director of Yaskawa, keeps a close eye on the various innovation projects within the Factory of the Future program. Yaskawa is especially interested in robotics and logistics. The company is also taking part in the Fieldlab Advanced Manufacturing Logistics.

This kind of field lab is important, according to Mennen, because it means that you can work together on innovations that you as a company on your own would not be so quick to pick up. “You complement each other by working together. Not only in terms of knowledge, but also in terms of costs. If everyone contributes something, you also have fewer expenses. Also, you get results faster than if you did it alone. The knowledge you gain is something that you can spread across the market. After all, if there are solutions that are not being promoted, then they will be of no use to us.”

3D bin picking is a generic issue within the manufacturing industry, Mennen continues. “It has not really been solved properly anywhere. This particular solution offers many new opportunities. We are extremely curious to see how they will turn out in practice.”

Fontys University of Applied Sciences, also located on the BIC, had previously bought Yaskawa’s robotic arm. Mennen had approached the Fontys lecturers/researchers Mark Stappers and Randy Kerstjens about it. There are already quite a few systems that can scan, recognize and pick up all kinds of different objects. However, they do not work well with shiny and oil-covered objects as yet, which is what the students had to find a solution for.

Hybrid learning environment

Stappers: “The Experience Center is a source of knowledge for us at Fontys. We pick up valuable information there. But we are also happy when we can provide input.”

The students were given a project assignment. They were supervised by an employee from Affix Engineering, one from Yaskawa, and a teacher from Fontys. “It is a hybrid learning environment, a great interaction between education and business. Such projects allow the students to gain the latest skills and knowledge,” Stappers adds.

They also learn how to deal with uncertainties, Stappers notes. “In the technical world, you also have to contend with problems. These problems are always very diverse. Sometimes the software is the problem, other times the electronics or mechanics aren’t quite right. Sometimes it is a combination of things. A project in the field lab simulates the real world. Through the experiences that students gain from that project, they learn to be open to those uncertainties and to look for a solution.”

The preliminary exploratory work on the setup has been completed and the students wrapped up their assignment earlier this month. A 3D bin-picking system that can grasp shiny objects is now in place. Mennen: “The foundation is there. Follow-up research is still needed, however. We also want to do this with Fontys again. Other grippers still need to be built and the speed of the process needs to be looked at.”