Débora E. C. Oliveira, Caio L. Demario, Paulo A.V. Miranda
Image Segmentation by Relaxed Deep Extreme Cut with Connected Extreme Points,
IAPR International Conference on Discrete Geometry and Mathematical Morphology (DGMM).
May 2021, Sweden, accepted, to appear.

Abstract

In this work, we propose a hybrid method for image segmentation based on the selection of four extreme points (leftmost, rightmost, top and bottom pixels at the object boundary), combining Deep Extreme Cut, a connectivity constraint for the extreme points, a marker-based color classifier from automatically estimated markers and a final relaxation procedure with the boundary polarity constraint, which is related to the extension of Random Walks to directed graphs as proposed by Singaraju et al. Its second constituent element presents theoretical contributions on how to optimally convert the 4 point boundary-based selection into connected region-based markers for image segmentation. The proposed method is able to correct imperfections from Deep Extreme Cut, leading to considerably improved results, in public datasets of natural images, with minimal user intervention (only four mouse clicks).
Figure 1 shows the segmentation of an insect by the proposed method. The extreme points with 4 clicks (Figure 1a). The probability map for the object produced by the CNN of Deep Extreme Cut [1] (Figure 1b). The optimal connection for the extreme points (Figure 1c). The final segmentation using the resulting markers (Figure 1d).

(a) extreme points(b) probability map by Deep Extreme Cut [1]
(c) optimally connected extreme points(d) final segmentation
Figure 1: The segmentation of an insect by the proposed method.

The ground truth databases of public images and the source code used to evaluate our method are available. We used four datasets of natural images:

  1. GrabCut dataset of 50 images from [2].
  2. NatImg21 dataset of 64 natural images of various objects, which is available here.
  3. BirdsInsects dataset of 120 images, possessing objects with thin and elongated parts, which is available here.
  4. GeoStar dataset from [3], which is composed of 151 images.

The pre-computed results by Deep Extreme Cut [1] and the extreme points, automatically computed from the ground truth images, for all datasets, as used in our experiments, are available here.

Our source code is available here. The code was implemented in C/C++ language, compiled with gcc 9.3.0, and tested on a Linux operating system (Ubuntu 20.04.1 LTS 64-bit), running on an Intel® Core™ i5-10210U CPU @ 1.60GHz × 8 machine. The ImageMagick command-line tools are required to convert images from different formats.

Below are some sample images of the NatImg21 dataset:

(a) Image A(b) Image B(c) Image C
(d) Ground truth of image A(e) Ground truth of image B(f) Ground truth of image C
Figure 2: Some sample images from NatImg21 dataset.

[1]Maninis, K.K., Caelles, S., Pont-Tuset, J., Gool, L.V.:
Deep extreme cut: From extreme points to object segmentation.
In: In IEEE Conf. on Computer Vision and Pattern Recognition, pp. 616-625 (2018)
[2]Rother, C., Kolmogorov, V., Blake, A.:
"Grabcut": Interactive foreground extraction using iterated graph cuts.
ACM Trans. on Graphics, 23(3), 309-314 (2004)
[3]Gulshan, V., Rother, C., Criminisi, A., Blake, A., Zisserman, A.:
Geodesic star convexity for interactive image segmentation.
In: Proc. of Computer Vision and Pattern Recognition, pp. 3129-3136 (2010)