Composing An Automation Symphony

Dmitry Lukovkin, Zyfra Robotics,

Finland, answers the question, ‘how do you make a dump truck autonomous?’, and explores what the future might hold for automation in opencast mining.

To turn a conventional dump truck into an autonomous one, two things are needed. First, to enable electronic control of the truck’s actuating mechanisms – such as front (steering) wheels, transmission, brake system, retarder, etc. – control of the truck by sending the appropriate electronic signals and receiving feedback, rather than by mechanical actions on the steering wheel or pedals, must be made possible.

Secondly, it is necessary to ‘teach’ it to determine its position in space and perceive the surrounding environment. To do this, it needs a set of sensors: lidars, radars, digital video cameras, IMU sensors, communication equipment, and a high-precision satellite navigation system – for example: GPS, GLONASS, or Galileo.

This being said, ‘brains’ are also needed on the truck to operate it all. High-performance on-board computers, network, and communication equipment need to be installed on the truck, as well as a reliable power source for all this equipment. Since the design of the dump truck does not

provide power for the sensors or additional electronic units, a significant part of the whole complex is providing power. A dump truck is not a device with a reliable 220 V power supply. Therefore, the essential part of making a dump truck autonomous is not just the work of software developers, but also engineers who will develop a whole range of hardware solutions – from brackets, housings and cabinets with environmental protection, to the power system and even on-board computing units of the required configuration.

Why is making a dump truck autonomous less than half of the problem?

It is important to understand that, unlike passenger cars, automating a standalone dump truck at a mine site makes no sense. A quarry or a mining site is an orchestra, and it needs a conductor to control it as a whole.

To make an autonomous operations zone productive and efficient, consideration not only has to be put into the trucks, but also the loading equipment and auxiliary vehicles. Loaders will have to be equipped with a high-precision navigation system and a touchscreen panel to allow loading tool operators to interact with autonomous trucks. In some cases, loaders could be used in a tele-operations mode to further enhance safety. Auxiliary vehicles that routinely work in the autonomous operations zone side-by-side with autonomous haul trucks should be also equipped with high-precision navigation and communication systems.

Who rules them all, and how?

When an orchestra is tuned, ready and waiting in its pit, how is its performance directed? A conductor of course.

Obviously, when it comes to conducting an automated dump truck symphony, there should be software that fills this role. However, as the whole area is still very new, there is no unified name for this software. In the closed-stack, vertically integrated implementation of an automated haulage system (AHS), it is a part of, or an add-on to, the vendor’s fleet management system (FMS). In the ‘Open Autonomy’ initiative by Wenco International Mining Systems, it was called an ‘autonomy controller’.1 Sometimes it is referred to as AHS, though the ‘AHS’ term generally refers to the wider scope. Zyfra Robotics uses the internal term ‘Robot of Robots’.

So, what should this piece of software do exactly? Unfortunately, this is also not clearly defined yet.

In Zyfra Robotics’ vision, it comes down to the following main functions: n Tracking of the location and the state of all autonomous and connected vehicles, as well as provision of information about nearby vehicles to the autonomous trucks. n Planning of actions performed by autonomous trucks.

As a striking difference from passenger cars, haul truck have many things to perform, in addition to driving by the trajectory itself. n Calculation and management of routes and trajectories, in addition to speed limits. n Calculation of dynamic manoeuvers (cloud could also be implemented onboard). n Management of loading and unloading operations, including facilitating the interaction between loading tool operators and haul trucks.

On top of this, Zyfra utilises an autonomous operations management software with the following functions: n Autonomous operations zone configuration. n ‘Bird-eye view’ and human interface for AOZ operator, providing a visual representation of the connected vehicles on the map, and the ability to assign vehicles to different tasks and routes, allowing them to drive and send them to maintenance. n Data analysis and reporting, especially in terms of reports that are specific for autonomous operations. n Implementation of APIs to high-level mine-wide FMS and other information systems.

Martin Politick, Director of System Engineering and Architecture at Wenco, splits Open Autonomy services the following ways: n Spot – manages loading and unloading zones and queues. n Path – path planning. n Map service provides map engine and localisation functions in the same manner for each of the other components. n Dispatching service provides the FMS with the means to tell each truck where it should go and what to do. n Global traffic – traffic management service in terms of handling intersection priorities, segment permissions, and general speed limits.

In this concept, the AHS component provides path and map services to other components of the system. ISO is working on the FMS to autonomous haulage interface standard, ISO 23725; unfortunately it is not published yet.

All this is to say that the design and development of an autonomy controller represents a tremendous challenge.

Developing an autonomy controller

From the architectural point of view, an autonomy controller should combine two approaches: to be event-driven to efficiently react to constantly changing and unpredictable environment changes and changes in vehicles’ states; but also to be able to execute fixed sequences of actions, representing tasks of each dump truck and other vehicles. It should be capable of ‘mapping’ the pre-defined and rigid task execution actions sequence to a very fluid spatiotemporal space representing the state of an autonomous operations zone. Moreover, though at this level the system should not adhere to real-time constraints, it is still critical to limit latencies and ensure that the correct order of sequential events is maintained.

There are no ready-made best practices, architecture guidelines, or frameworks for the development of such services. That being said, and keeping in mind that autonomous hauls trucks are already in operation,

the problem is being solved in different ways by different vendors.

One of the approaches is to use open source software frameworks and libraries as building bricks, though it will require significant efforts to develop an autonomy controller that will comply with functional and non-functional requirements.

Considering that this is the autonomy domain that is being discussed, robot operating system (ROS) and ROS 2 frameworks could look like a natural match for the job. For the sake of the design of the autonomy controller, an ROS provides a communication infrastructure for the distributed set of worker processes, called nodes. Different nodes are responsible for different functions and can exchange messages via a publish/subscribe paradigm, thus naturally enabling an event-driven approach. ROS 2 provides further improvements, being more suitable for real-time, or near real-time, applications.

Another interesting take on the problem was proposed by GAIA with their GAIA Platform. The GAIA Platform’s goal was to accelerate autonomous development, including the development of software for mobile robots and cars. The GAIA Platform offers a breakthrough solution to many questions by using the rule-based, declarative approach to software design. GAIA’s developers defined sets of rules (rulesets) in Declarative C++, that would be invoked in the case of changes in a specific GAIA database field. According to GAIA, their approach allowed developers to write 10x less code to implement the same functionality, in comparison with more traditional approaches. Unfortunately, it seems that GAIA have wound down commercial and development activities, however the GAIA Platform has been made available as an open source.2

Looking to the future

What does the future hold for autonomous vehicles in mining and how could the software and artificial intelligence (AI) community facilitate further development?

There are two twists in how robots in the industry will evolve in the coming years. First, not all machines are easy to make autonomous. For example, in the case of loaders and bulldozers, the bucket operation is poorly amenable to be controlled by deterministic algorithms. That is to say, a person works with them intuitively, but a machine cannot do that yet. Thus, for the time being at least, the industry is left hoping for a breakthrough in simulation and machine learning. A lot of recent developments in AI, such as zero and few-shot learning, reinforcement learning, and generative-adversarial networks hardly found their way to the application in autonomous mining vehicles.

The second breakthrough has to do with the fact that the unit of autonomy in mining is the pit, not the vehicle itself. Right now, robotic dump trucks are like the first cars, which were like carriages, only without horses. Accordingly, quarries and mines are now being designed for manually operated vehicles. This should change, as well as the dump truck designs themselves. For example, the cab should be removed because it is not needed for an autonomous vehicle. Perhaps the concept of reversing will disappear: robotic dump trucks will not care which way they go.

At this point, it will be the right time to change the configuration of opencast mines: reduce safety berms, reduce haul road width, and remove edge kerbs and other elements that are only necessary for people to work.

The challenge for a software and AI development community here is to radically improve the speed of the delivery of new functionality to the vehicles and systems in mines and quarries, without sacrificing safety or productivity. Software and AI developers working in the mining industry should adopt the best engineering practices – such as continuous integration and continuous delivery and machine learning operations – and adapt them to the requirements and realities of the mining industry.

Conclusion

Only mutual efforts of the mining industry, software and AI engineering communities, shaped as collaborations on different aspects of autonomous systems for mining and open interfaces between them and other information systems, as well as software engineering practice, will allow unlocking the full potential of autonomous technologies in mining. As of now, the industry has barely scratched the surface.

Reference:

1. ‘'Open Autonomy' initiative’, Wenco International Mining Systems, www.wencomine.com/vision/open-autonomy-vision 2. ‘The Gaia Platform’, GitHub, www.github.com/gaia-platform/

GaiaPlatform