Kaggle Top 1% · Deep Learning

Sartorius Neuronal Cells Segmentation

A robust deep learning pipeline for segmenting neuronal cells in microscopy images — Cascade Mask R-CNN X152 with semi-supervised pseudo-labelling and cascade IoU fusion, reaching top 1% on Kaggle.

Xinyue Wang
Inference result showcase.
Inference result showcase. Instance masks on light-microscopy neuronal cells.

Why

Neurological disorders have caused increasing death and disability across the globe, which increases the demand for treatment and rehabilitation. As a key step in drug development, neuronal cell segmentation in light microscopy is vital to obtain a disorder’s response to treatment — but manual instance segmentation is inefficient, time-intensive and labor-costing.

The pipeline

We trained a robust baseline on Cascade Mask R-CNN X152 with large-scale training, multi-scale jittering and copy-paste augmentation. Post-processing tailored to the dataset — mask screening, overlap removal, small-instance removal — cut false positives. We then ran inference on the unlabeled LiveCell set to obtain pseudo-labels, screened them by prediction confidence, folded them into a larger training set and repeated the cycle. Finally, to avoid the negative effect of naively fusing instance segmentation models, we devised a cascade IoU screening method to fuse predictions from six models across folds and seeds. Test-time augmentation (flip and resize) was used throughout inference.

Pipeline workflow.
Pipeline workflow. Baseline, post-processing, semi-supervised rounds and cascade IoU fusion.