Abstract. Segmentation of medical images is one of the key tasks of biomedical informatics, especially in the context of cancer diagnosis. The task of precise segmentation of cell nuclei is particularly relevant and necessary to identify morphological changes in tissues. However, the main challenges and problems in this area remain the variability of image quality, limited amounts of available data, and the need to ensure the high accuracy and stability of deep learning models. This study analysed the impact of ten data augmentation techniques, including the addition of random noise of varying intensity, on the cell nucleus segmentation model’s performance. The experiment also specialises in techniques using artificial noise addition, simulating real-world conditions such as lighting variations, artefacts, and defects during sample preparation. Two types of noise were used to add: additive Gaussian noise and uniform random noise with an intensity from minus 0.05 to 0.05, which demonstrated a significant effect on the generalizing ability of the model, improving its resistance to heterogeneous data. The architecture used was based on a modified UNet model with the introduction of the CBAM module, which focuses the model’s attention on significant areas of the image. The module has been added to the decoding part of the model architecture. For the experiments, two reference datasets CryoNuSeg and MoNuSeg were combined with preprocessing, including patching and image format conversion. Applied augmentation techniques included horizontal and vertical reflection, random rotation, contrast change, elastic deformation, and noise addition. In addition, augmentation combinations were investigated during the experiments. The study results showed that the techniques of horizontal and vertical reflection augmentation and random rotation achieved minimal losses, while the accuracy of the models exceeded ninety per cent. The contrast change technique demonstrated the most balanced performance, providing an average intersection ratio over the union of 0.928 and an accuracy of 92.2 per cent. The data from the study results emphasize the importance of using artificial noise addition to increase the model’s resistance to artefacts and data variability, as well as the use of combined augmentation techniques. In the future, combined augmentation techniques may become the basis for the development of adaptive algorithms capable of effectively working with heterogeneous biomedical images
Keywords. Histological images, cancer, nuclei segmentation, convolutional neural networks, attention module, augmentation.