Instance Segmentation for Infrastructure and Obstacle Detection in Automatic Train Operation Systems

Abstract

The paper considers the application of instance segmentation methods for solving problems of detecting railway infrastructure objects and obstacles in automatic train operation (ATO) systems. Particular attention is paid to the possibility of using the You Only Look Once version 11 (YOLOv11) deep neural network architecture in the context of increasing automation levels to Grade of Automation 3 and Grade of Automation 4, where reliable operation of perception systems in real time is critical. One of the major contributions in this paper includes developing a specialized dataset on the perception of railways with 20,000 images, each with pixel-level annotation on 46 object classes. This paper has evaluated 25 different YOLOv11 models, which vary in terms of depth and input image resolutions. All configuration models were trained on a specially collected dataset of images obtained from rolling stock video cameras in real operating conditions. Performance evaluation included such metrics as segmentation accuracy, inference speed, and computational complexity. The results demonstrate that YOLOv11 provides a flexible choice of configurations, allowing the model to be adapted to specific technical conditions and requirements. This confirms the feasibility of using YOLOv11 in real ATO systems to improve the reliability of perception, support autonomous navigation tasks, and timely detection of critical objects along the train route.

Received: 24 April 2025 | Revised: 29 December 2025 | Accepted: 2 April 2026

Conflicts of Interest

The author declares that he has no conflicts of interest to this work.

Data Availability Statement

Data available on request from the corresponding author upon reasonable request.

Author Contribution Statement

Vladimir A. Fedorov: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Resources, Data curation, Writing – original draft, Writing – review & editing, Visualization, Supervision, Project administration.