Research · Peer-reviewed · Case study
Multi-Agent Robotic Search Team
Reinforcement learning for robot search teams that must cope with unknown areas and failing members
This is research, not a client project. It comes from our founder's work in AI and robotics and was published in the Mehran University Research Journal of Engineering & Technology (2021).
Search and rescue puts people at risk, and robots fail too.
Disasters such as fires and building collapses are dangerous for rescue workers and dogs. Robots can search in their place, but most robots used in the field today are tele-operated.
Autonomous teams face three hard problems: the area is unknown, some areas are hazardous, and individual robots can run low on fuel, lose their connection or be destroyed.
The authors found that earlier models handled some of these (uncertainty, coordination, learning, fault detection) but not all together. The question behind this research: can a team learn good search behaviour and keep working when a member fails?
A leader, its members, and a learned policy.
The search area is split into partitions, each divided into blocks. A leader agent assigns members to partitions and manages their modes. Each partition is unknown, friendly or hostile, and that status is only discovered by searching.
Both levels are modelled as Markov Decision Processes. The leader tracks agent status, goal status, partition assignment, partition status and standby agents. Members move block to block (left, right, straight, back, stay) while tracking fuel, faults and hazards.
Q-learning is then applied on top of the MDP to build a Q lookup table, so each member can act on its own with less dependence on the leader. A member that stops responding is treated as faulty and put on standby, and the leader keeps trying to reconnect or sends another agent.
What the work covers.
Q-learning against a plain MDP policy.
Selected simulation cases, as described in Section 4.2 of the paper.
| Scenario | MDP policy | Q-learning policy |
|---|---|---|
| Redundant agent (leader, Cases C–E) | Swaps agent positions; no agent moves to standby | Moves the redundant agent to standby |
| Hostile blocks 3 and 5 (member, Fig. 10) | Block 4 → 6 → 7, avoiding block 5 | Block 3 → 4 in a single move, reaching safety |
| Hostile block 5, agent in block 1 (member, Fig. 11) | Moves to block 7 | Moves one block right to block 2 |
| Companions present, all blocks hostile (member, Fig. 13) | Moves to partition 1, then a second step to block 6 | Moves straight to block 7 of partition 1 |
In the paper's qualitative comparison (Table 1), the proposed model adds learning and fault detection over a prior MDP model.
In the cases above, the learned policy reached safe states in fewer moves and used standby for redundant agents, where the plain MDP policy only moved agents around.
What these results do and don't show.
All results come from simulations on small grids (for example three-by-three partitions), with policies computed in MATLAB 2013a. The paper reports no field tests or real-robot deployment.
The comparison is by scenario and by qualitative properties. The paper does not report a single accuracy or speed-up figure, so this page does not either.
Coordinating several autonomous agents, handling the one that fails and learning from feedback are the same problems we meet when building multi-agent AI systems for businesses.
To see how we apply that today, read the RTM Kinetics case study, or meet our founder.
Read the research.
- Reinforcement Learning Based Hierarchical Multi-Agent Robotic Search Team in Uncertain EnvironmentS. Hamid, A. Nasir, Y. Saleem. Mehran University Research Journal of Engineering and Technology, vol. 40, no. 3, pp. 645–662, July 2021 (open access, CC BY 4.0). DOI 10.22581/muet1982.2103.17
Have an idea worth building?