Linking computer vision with robotics is a tricky problem, but the benefits in places like factories, fields and the ocean floor are worth the trouble. Brent Balinski spoke to Professor Peter Corke of Queensland University of Technology about giving robots the power of sight.
A richer view of the world
Lidar is the sensing modality of choice for many examples of sophisticated, well-known robots, such as the Atlas and the Google self-driving car. Laser rangefinders, rather than vision, have dominated the world of robotics since the 1990s, when they became cheap enough to be considered.
Though computer vision had grown up with robotics, the computing muscle required and frustration sometimes involved in making things work has seen vision, well, overlooked. The Queensland University of Technology-led Australian Centre of Robotic Vision wants to remedy this.
Though lidar is useful in outputting metric detail about an environment, it presents nothing in the way of the colour, texture or semantic information contained in a scene.
Professor Peter Corke, who heads the ACRV and whose professional motivation is to create robots that can see, calls a point cloud an “impoverished view” of the world. “And that’s really a cutting-edge problem today and has been, I guess, for decades.”
Australia possesses some skilled system integrator companies, largely thanks to its (now disappearing) automotive sector. However, In terms of manufacturing robots, there isn’t much sense in taking on well-established giants like Kuka, Fanuc and ABB, believes Professor Corke.
“We are now making real progress now in being able to solve that problem. “The question is ‘why is this useful for robots?’ It’s useful for robots because we get, in a single snapshot, information about what’s in a whole, big area of the world that our picture encompasses.” Professor Corke, an IEEE Fellow, has been at QUT since 2010.
His career has included starting the CSIRO’s Autonomous Systems Laboratory, and he began in the field of manufacturing robotics, including work with high-speed computer vision techniques.
At the time (the mid-1980s) there were plenty of Australian robotics and machine tool companies, building equipment to service auto manufacturers and other heavy industry. However, today’s situation is rather different.
Current projects
According to the ACRV, robotic sight will lead to outcomes including safer, better industrial robots, able to anticipate what nearby human workers will do next and acting as a smart co-worker.
Current projects illustrating the value of vision include agricultural applications and one dealing with a problem threatening the health of the Great Barrier Reef. The AgBot II is able to see and extirpate weeds, either spraying them with chemicals or digging them out of the ground if this is an option.
According to those behind the project, it can save 60 per cent of the amount of pesticide usually used.
“So it’s moving along relatively slowly, looking at the ground beneath it, and sees all of the plants, classifies them as this type of weed or that type of weed; it knows a lot of weeds now,” explained Professor Corke.
The other agricultural application is one that can properly identify and collect capsicums. It uses a more geometric approach, looking at range images with colour images, considering shape, colour and texture and discerning ripe and capsicums from unripe capsicums from leaves on the plant.
The COTSBot, currently in sea trials, has been shown “tens of thousands” of examples of “starfish” and “not starfish” pictures, and has a deep inference system to make judgments.
A lidar would not work underwater, with vision and ultrasonics the only two existing sensing modalities up to the job. It is able to detect what is a crown of thorns starfish – a pest that has destroyed vast tracts of the Great Barrier Reef – and autonomously inject it with a bile salt developed by the Australian Institute for Marine Science.
“It’s a nice example, I think, of a robot that performs a useful task, and the task is critically based on its ability to see the world, to recognise objects and to estimate the three-dimensional shape of the world using stereo vision,” explained Professor Corke.
A QUT team using computer vision methods last month qualified for the Amazon Picking Challenge (see below).
Looking to the future
The ACRV was officially launched last year, and officially began its work in 2014 with a seven-year Centre of Excellence term. The goal is to reunite computer vision and robotics, following a kind of disconnectedness blamed on the rise of affordable lidar and the difficulty of the problem mentioned above.
According to the ACRV, it’s time to have another look at a tough problem for a few reasons. Moore’s Law continues to allow images from cameras be processed quicker and quicker. Image processing algorithms are getting better.
Also, cameras are lightweight as well as dirt cheap, both in price (about a buck each in bulk) and in energy consumption, compared to lidar. As the quest for computer vision continues, where can the Australian robotics community make a difference? Professor Corke suggests that the country’s efforts are best spent in field robotics, where our major strengths lie. “They do have a lot of scope, because it plays to significant economic strengths in Australia, which are around primary industries, mineral extraction, agriculture,” he explained. “And that’s where I think there’s a lot of need for robots, in those sectors.”
As populations shift away from rural areas, there are workforce vacuums to be filled. The difficulty in attracting labour to jobs such as fruit picking is well known. In areas like this, robots able to discern the right things to pick and work around the clock would be of enormous value. “So in some ways labour is being removed from remote areas and being concentrated in the cities, and someone’s going to have to do that labour, if we want to have those industries,” he added.
Australia possesses some skilled system integrator companies, largely thanks to its (now disappearing) automotive sector. However, In terms of manufacturing robots, there isn’t much sense in taking on well-established giants like Kuka, Fanuc and ABB, believes Professor Corke.
The local industry looks very different to when Professor Corke finished his PhD in Mechanical and Manufacturing Engineering, and the lack of critical mass is a big disincentive.
“I think we’re going to struggle to be able to make factory robots as cheaply and effectively as a number of big overseas corporations do,” he said. “I think it would be kind of fruitless for an Australian company to try and compete with those guys head-on.”
Picking winners: a sidebar on robotics competition
When a problem comes along, a competition can be a wise response. Stumping up a healthy cash prize is a way to spur on innovation: think of the Darpa Grand Challenge and its impact in kickstarting autonomous vehicles. According to Peter Diamandis, founder of the X-Prize, prizes are more and more useful because there’s more and more computing ability available to more and more people. “Today one of you… have more capabilities than a government had 20 years ago or a large corporation had 20 years ago to access computational power,” Diamandis said.
Amazon has made huge investments and efforts in automating their fulfillment centres, but the next step – finding and picking the right item out of the Kiva robots, moving shelves from the warehouse to the worker – is too tricky to automate – so far.
Much of the walking and searching has been eliminated, but, for Amazon, “commercially viable automated picking in unstructured environments still remains a difficult challenge.”
Their response was to launch the Amazon Picking Challenge at The International Conference on Robotics and Automation last year. (Proving the popularity of contests to solve tough automation problems, the Airbus Shopfloor Challenge will take place at ICRA this year. It will look at ways to make a dent in a backlog of some 7,000 planes via the use of light robotics.)
A QUT team has qualified for this year’s APC in Leipzig, Germany and will take on the challenge of correctly, autonomously picking the correct item from each bin. “We’re using a mixture of sensing modalities: 3D cameras, colour cameras, shape, texture, colour, in order to find that object – it has some similarities to the capsicum picking problem, actually,” explained Professor Corke.
“The other challenge is if the object you’re interested in is, say, at the back of the shelf, the object in front of it we can see, we can reach in and grab it, but in extracting it we can’t pull any of the things out onto the floor.” Each of the fairly deep shelves has a mixture of products, which helps Amazon optimise their scheduling process.
Challenges include picking the right item, not dropping it (and recovering if this happens), and selecting the right effector end. Earlier this year, Amazon introduced the opposite to picking: a restocking challenge. “It’s a very complex challenge, and very critical to Amazon’s business model,” said Professor Corke. “And as seems to be the case a lot lately, when you’ve got a very hard problem, you offer a prize and hope that smart people will solve your problem for you.”