A Computer Vision System for the Automatic Classiﬁcation of Five Varieties of Tree Leaf Images

: A computer vision system for automatic recognition and classiﬁcation of ﬁve varieties of plant leaves under controlled laboratory imaging conditions, comprising: 1– Cydonia oblonga (quince), 2– Eucalyptus camaldulensis dehn (river red gum), 3– Malus pumila (apple), 4– Pistacia atlantica (mt. Atlas mastic tree) and 5– Prunus armeniaca (apricot), is proposed. 516 tree leaves images were taken and 285 features computed from each object including shape features, color features, texture features based on the gray level co-occurrence matrix, texture descriptors based on histogram and moment invariants. Seven discriminant features were selected and input for classiﬁcation purposes using three classiﬁers: hybrid artiﬁcial neural network–ant bee colony (ANN–ABC), hybrid artiﬁcial neural network–biogeography based optimization (ANN–BBO) and Fisher linear discriminant analysis (LDA). Mean correct classiﬁcation rates (CCR), resulted in 94.04%, 89.23%, and 93.99%, for hybrid ANN–ABC; hybrid ANN–BBO; and LDA classiﬁers, respectively. Best classiﬁer mean area under curve (AUC), mean sensitivity, and mean speciﬁcity, were computed for the ﬁve tree varieties under study, resulting in: 1– Cydonia oblonga (quince) 0.991 (ANN–ABC), 95.89% (ANN–ABC), 95.91% (ANN–ABC); 2– Eucalyptus camaldulensis dehn (river red gum) 1.00 (LDA), 100% (LDA), 100% (LDA); 3– Malus pumila (apple) 0.996 (LDA), 96.63% (LDA), 94.99% (LDA); 4– Pistacia atlantica (mt. Atlas mastic tree) 0.979 (LDA), 91.71% (LDA), 82.57% (LDA); and 5– Prunus armeniaca (apricot) 0.994 (LDA), 88.67% (LDA), 94.65% (LDA), respectively.


Introduction
In the past decades, the use of herbicides, pesticides, and other chemical substances has continuously been increased. Unfortunately, overuse of these materials cause surface water pollution, environmental pollution, and animal and human toxicity [1] (Liu and O'Connell, 2002). For this reason, scientists proposed the use of precision agriculture. One of the advantages of precision agriculture is the use of chemical substances only over the area of interest, called site-specific spray. The first step in site-specific spray operations is the proper recognition of the area of interest. Image processing is applied in this field to automatically identify plant species. In the past, there have been various researchers working on the recognition of different plants and trees.
Singh and Bhamrah [2] stated that the recognition of different plants, especially medicine plants is challenging. Eight species of different plants including Amrood, Jaman, Kathal, Nimbu, Palak, Pippal, Sfaida and Toot, were used in this study. From each specimen, ten images were taken. The proposed The aim of this study is to design an intelligent computer vision system to classify five species tree leaves, including 1-Cydonia oblonga (quince), 2-Eucalyptus camaldulensis dehn (river red gum), 3-Malus pumila (apple), 4-Pistacia atlantica (mt. Atlas mastic tree) and 5-Prunus armeniaca (apricot). Figure 1 depicts a flowchart of the computer vision system for plant tree leaves automatic classification, here proposed.

Materials and Methods
As it can be seen from Figure 1, the proposed computer vision system consists of five steps: the first step includes imaging, the second step includes segmentation and pre-processing, the third step includes feature extraction (including texture descriptors based on the histogram, texture features based on the gray level co-occurrence matrix, shape features, moment invariants, and color features), fourth step includes discriminant feature selection (based on two methods: hybrid artificial neural network-differential evolution (ANN-DE) and hybrid artificial neural network-genetic algorithm (ANN-GA)), and fifth step comprises classification based on three different classifiers, including hybrid artificial neural network-ant bee colony (ANN-ABC), hybrid artificial neural network-biogeography based optimization (ANN-BBO) and Fisher's linear discriminant analysis (LDA). Each of the five steps in the here proposed computer vision system, depicted in Figure 1 flowchart, will be further explained with more detail in the next paper sections.

Database Used in Computer Vision
As already mentioned, in this study five different types of leaves were investigated, including 1-Cydonia oblonga (quince), see Appendix A-Eucalyptus camaldulensis dehn (river red gum), see Appendix A-Malus pumila (apple), see Appendix A-Pistacia atlantica (mt. Atlas mastic tree), see Appendix A-Prunus armeniaca (apricot), see Appendix A. Table 1 shows the class number, English common tree name, scientific tree name and the number of samples in each class. Figure 2 shows one sample of each leaf type. All images were taken in Kermanshah, Iran (longitude: 7.03 • E; latitude: 4.22 • N). High-quality images were taken with a color GigE industrial camera (ImagingSource, model DFK-23GM021, 1/3 inch Aptina CMOS MT9M021 sensor, 1.2 MP 1280 × 960 spatial resolution, Germany), mounting an appropriate Computar CBC Group lens (model H0514-MP2, f = 5 mm F1.4, 1/2 inch type megapixel cameras, Japan). The camera was fixed at 10 cm above ground level, and all images were taken under white light of 327 lux lighting conditions.

Image Pre-Processing and Segmentation
The segmentation stage is an important one in image processing, since in case of wrong segmentation, either the background is considered as an object (in this study, any tree leaf) or object is considered as background. To reach a proper segmentation, six standard color spaces (RGB, HSV, HSI, YIQ, CMY, and YCbCr) were taken into account to find the best color space for segmentation purposes. Results have shown that the best color space for image segmentation was YCbCr color space and the best channels for image thresholding were luminance Y and chrominance Cr channels. Equation (1) was used to set the value of the threshold used to segment objects from their background in tree leaf images: The implications of this equation are as follows: each pixel in YCbCr color space that has Y components smaller or equal to 100, or Cr components larger or equal to 15, is considered as background, otherwise, a pixel is considered as foreground (tree leaf object). In order to extract shape features from segmented images, binary images are needed. To extract shape features with high accuracy, some morphological operations are needed, since usually, some "noisy" pixels can exist in segmented images. For this reason, the Matlab imclose function was used, (Gonzalez et al. [10]). Figure 3 summarizes pre-processing operations, shown over three sample tree leaf images.

Feature Extraction
It is well-known that in order to properly classify different tree leaf types, feature extraction is needed. Features of various types and in different fields include texture descriptors based on the histogram, texture features based on the gray level co-occurrence matrix, shape features, moment invariants and color features (others may exist that are not used here). Above mentioned feature types were extracted from each leave. Indeed, the first 285 features of each leaf image were extracted and then effective (high discriminant) features were selected among them by hybrid ANN-DE and hybrid ANN-GA optimization approaches.

Texture Descriptors Based on the Histogram
Texture features that were extracted based on histogram include homogeneity, entropy, smoothness, third moment, average and standard deviation.

Texture Features Based on the Gray Level Co-Occurrence Matrix (GLCM)
It is well-known that neighbor angles have a big impact on the estimation of the values of each feature based on the GLCM. Table 2 shows texture features extracted based on the gray level co-occurrence matrix. These features were extracted for four different angles: 0 • , 45 • , 90 • , and 135 • . Thus, in total, 27 × 4 = 108 texture features were extracted from each object.

Shape Features
In this study, 29 shape features were extracted. Table 3 lists the shape features that were used here.

Moment Invariant Features
Moment invariant features have the advantage that are insensitive to translation, reflection dilation, and rotation. In this study, 10 moment-invariant features were extracted: first-order moment invariant, second-order moment invariant, third-order moment invariant, fourth-order moment invariant, fifth-order moment invariant, sixth-order moment invariant, seventh order moment invariant, difference of first and seventh order moment invariants, difference of second and sixth-order moment invariants and difference of third and fifth-order moment invariants.

Discriminant Feature Selection
As mentioned before, 285 features were extracted from each leaf image object. The use of all features as input to classifiers is not wise, given the problem of overfitting and poor generalization to the test set. The selection of discriminant groups of features is a well-known good practice. To do so, two methods based on artificial intelligence were used to select effective features (results to be shown later on): ANN-GA.

Optimal Neural Network Classifier Parameters
Classification is the final step in designing a computer or machine vision system. A highperformance classifier guarantees the accuracy of the computer version system. In this study, three classifiers including hybrid ANN-ABC, hybrid ANN-BBO and Fisher's linear discriminant analysis (LDA), were used in classifying. In order to have statistically valid results, 100 uniform random simulations were repeated. All data were divided into two disjoint groups, training, and validation set data (60% input samples) and test set data (40% of input samples) in each of the 100 averaged simulations, following a uniform random distribution with probability 0.6 and 0.4 to belong to train/validation and test sets, respectively. It is worth mention that a multilayer perceptron (MLP) ANN was used. An MLP neural network has five adjustable parameters: number of neural network layers, number of neurons in each layer, nonlinear transfer function, back-propagation network training function, and back-propagation weight/bias learning function. ABC and BBO algorithms were used to select optimal MLP parameter values. Table 5 shows the optimum values of the MLP parameters which were determined with ABC and BBO algorithms.

Results and Discussion
To evaluate the three different classifiers' performance (hybrid ANN-ABC, hybrid ANN-BBO and LDA), various criteria are computed, including confusion matrix, sensitivity, specificity and accuracy, receiver operating characteristic (ROC) and area under the ROC curve (AUC). Before the evaluation of the classifiers' performance, selected features based on ANN-DE and ANN-GA approaches are shown next.

Feature Selection Based on ANN-DE and ANN-GA Approaches
As mentioned in previous sections, two ANN-DE and ANN-GA approaches were used to select effective (discriminant) features. Table 6 shows the seven selected features by ANN-DE and ANN-GA. As one can see, ANN-DE method selected all features among shape and texture features based on the gray level co-occurrence matrix, while the ANN-GA method selects discriminant features among shape, texture based on the gray level co-occurrence matrix and color features. Figures 4 and 5 show boxplot graphs for the value of each extracted feature by ANN-DE and ANN-GA methods, respectively. Figure 4 (ANN-DE) shows that quince, red gum, mastic tree, and apricot classes can be separated based on only three features, inverse difference moment normalized related to 0 degrees (IDMN_0), homogeneity related to 0 degree (homogeneity_0) and mean related to 45 degrees (mean_45), since those features have the lowest overlapping boxplots among classes. The same can be said, for obvious reasons, about Figure 5c (ANN-GA) regarding homogeneity related to 0 degrees (homogeneity_0) feature. Figure 6 shows the validation step, where we plot mean square error (MSE) as a function of iteration number while learning for hybrid ANN-ABC, hybrid ANN-BBO and LDA classifiers, based on the selected features obtained with ANN-DE and ANN-GA methods listed above. Figure 6a Table 6).

Classification Based on Hybrid ANN-ABC, Hybrid ANN-BBO and LDA
In this study, three classifiers were used: hybrid ANN-ABC, hybrid ANN-BBO and LDA. To achieve statistical valid results, we averaged 100 simulations with uniform random train and test sets; the probability of an input sample to belong to the train set was 0.6 and 0.4 to belong to the test set. Table 7 shows confusion matrix classification results, using ANN-ABC classifier for 100 iterations, over the test set. The lowest classifier error value is that of red gum class with only 17 misclassified samples among 4703 total test samples, resulting in accuracy above 99.6% for this class. The highest classifier error value is given in the mastic tree class, where 453 out of 3542 samples were misclassified, resulting in a misclassification rate of 12.79%. Figure 2 shows that the shape of 2-Eucalyptus camaldulensis dehn samples (river red gum) are very different from other class samples, so samples related to this class are classified with high accuracy. On the other hand, as a possible explanation to previous facts, Figure 2 shows how river red gum tree leaves are quite different in shape to other four tree leaves species, which at the same time have quite similar shapes among them, thus resulting in most misclassification errors among these four classes. In summary, the ANN-ABC classifier classified the 20,700 test samples with an accuracy above 94%, which has to be judged as very good for our tree leaf image database.  Table 8 shows confusion matrix classification results using hybrid ANN-BBO classifier for 100 iterations averaged, over the test set. As can be seen, 2230 samples from 20,700 samples were misclassified by this classifier (10.77%). Again, the lowest ANN-BBO classifier accuracy is for mastic tree (82.20%) and the highest classifier accuracy is again for river red gum tree (94.60%), similar to the hybrid ANN-ABC classifier, and considered as promising results, despite on average the ANN-BBO classifier performance is behind its partners.  Table 9 shows classification results presented as a confusion matrix for the LDA classifier for 100 uniform random train and test set iterations averaged, over the test set. The interesting thing here is that the LDA classifier classified all samples in the river red gum class correctly. So it can be concluded that for easy class data, the LDA classifier has higher accuracy. On the other hand, the LDA classifier has less accuracy than the hybrid ANN-ABC classifier in quince and apricot classes. Finally, one can see how LDA classifier classified all test data with an overall accuracy of 93.99%, that is high accuracy very close to the 94.04% overall accuracy for the ANN-ABC classifier, and both clearly over the 89.23% overall accuracy for the ANN-BBO classifier, which performed the poorest of the three.

Classifiers Performance: Evaluation
Two common methods to evaluate classifiers performance over the test set are: 1.
Based on the sensitivity, specificity, and accuracy values (confusion matrix).

2.
Based on receiver operating characteristic (ROC) and area under the ROC curve (AUC).
We evaluate classifiers using both methods next.

Classification Performance Based on the Sensitivity, Specificity, and Accuracy Values (Confusion Matrix)
With sensitivity, specificity and accuracy criteria, one can evaluate a classifier system based on correct and wrong classified samples, detailing between which classes errors are taken place if needed. Sensitivity, specificity, and accuracy are defined next in Equations (2)-(4): where true-positive fraction (TP) is equal to the number of samples in each class that are correctly classified, true-negative fraction (TN) is equal to the number of samples on main diagonal in confusion matrix (square matrix that has dimensions equal to number of classes, cf. Tables 7-9) minus the number of samples that are correctly classified in the intended class, false-negative fraction (FN) is defined as the sum of horizontal samples in the investigated class minus number of samples that are correctly classified in the intended class, and false-positive fraction (FP) is defined as the sum of vertical samples of the investigated class minus the number of samples that are correctly classified in the intended class (Wisaeng [17]). In short, a sensitivity value of 100% in a certain class means that all samples of this class were classified correctly; a specificity value of 100% in a certain class means that no samples from other classes were misclassified in that given class; and an accuracy value of 100% means that neither no samples from other classes were misclassified in a certain class nor any samples from that certain class were misclassified in other classes. Table 10 shows classification performance criteria including specificity, sensitivity and accuracy results for 100 uniform random iterations using hybrid ANN-ABC, hybrid ANN-BBO and LDA classifiers. By looking carefully at Table 10, one realizes that the LDA classifier is in general terms superior to both ANN-ABC (slightly) and ANN-BBO (clearly), that red river gum tree class is consistently easier to classify among all three classifiers, and that opposite situation happens with mt. Atlas mastic tree class. In precise terms, the following results are reached: best classifier, mean sensitivity, and mean specificity, were computed for the five trees under study, resulting in 1-Cydonia oblonga (quince) 95.89% (ANN-ABC), 95.91% (ANN-ABC); 2-Eucalyptus camaldulensis dehn (river red gum) 100% (LDA), 100% (LDA); 3-Malus pumila (apple) 96.63% (LDA), 94.99% (LDA); 4-Pistacia atlantica (mt. Atlas mastic tree), 91.71% (LDA), 82.57% (LDA); and 5-Prunus armeniaca (apricot) 88.67% (LDA), 94.65% (LDA), respectively, all over the test set.  Figure 7 shows ROC graphs for the different classes and different classifiers considered. In short, when plotting ROC curves, we are plotting FPR versus TPR, on the specificity-sensitivity plane. The higher the AUC values is the higher the classifier accuracy, while having an AUC = 1/2 value is totally useless and simply equivalent to a random toss of a coin. As one can see, the ROC graphs related to hybrid ANN-ABC and LDA classifiers are near the upright end, so this implies that these two classifiers have good accuracy. At the same time, in Table 11 we show the mean and standard deviation of AUC values for the five different classes and three classifiers under consideration, over 100 repeated uniform random iterations. Figure 8 shows AUC boxplot graphs for 100 iterations based on ANN-ABC, ANN-BBO and LDA classifiers. As one can see, these three classifiers classified red river gum class samples with extremely high accuracy, since boxplot graphs are very compact. Also, LDA if often ahead of its partners, closely followed by ANN-ABC, in terms of AUC values (please note that vertical axis AUC ranges are different in each figure). In precise terms, mean AUC values were computed for the five trees under study, resulting in 1-Cydonia oblonga (quince) 0.991 (ANN-ABC); 2-Eucalyptus camaldulensis dehn (river red gum) 1.00 (LDA); 3-Malus pumila (apple) 0.996 (LDA); 4-Pistacia atlantica (mt. Atlas mastic tree) 0.979 (LDA); and 5-Prunus armeniaca (apricot) 0.994 (LDA), respectively, all over the test set.   Finally, to compare the CCR in this study with others, Singh and Bhamrah [2], Kadir [3], Mursalin et al., [5] and Ehsanirad [4], were used. It should be noted that since studies do not share the same leave image database, no direct comparison is possible. Singh and Bhamrah [2] classified 8 species of different plants with artificial neural network classifier and Kadir [3] used a Bayes classifier to classify some leaves related to several plants. Mursalin et al., [5] used some shape and color features for plant classification purposes. Ehsanirad [4] used only some texture features for the classification of different plants. Table 12 shows the accuracy or CCR of the different classification methods for comparison purposes. As one can see, the proposed method has a higher number of samples and higher accuracy than the other methods being compared with.  [3] 100 88 Mursalin et al. [5] 400 98.24 Ehsanirad [4] 65 78

Classification Performance Based on ROC and AUC
One possible reason for the superiority of the proposed method might be the use of different properties. As can be seen in Section 2.3, feature extraction, in this study several features comprising five different feature types were extracted. That can be of help in finding effective discriminative features that can properly separate different plant type leaves. While on the contrary, other researchers often used only a couple of feature types, such as color and texture features. A second possible reason for higher accuracy might refer to the structure of the neural network since the study here proposed the optimal structure of the ANN is adjusted by both ABC and BBO algorithms.

Conclusions
This study aims to design an imaging computer vision system to classify five types of tree leaf images. Relevant results can be summarized as follows to conclude:

1.
Tree leaves images had the lowest level of noise (noise pixels) after segmentation in YCbCr color space, in comparison with RGB, YIQ, CMY, HSV and HSI color spaces.

2.
Two channels in YCbCr color space were needed to image thresholding (a luminance and a chrominance channels).

3.
Among the seven discriminant features selected, four features were texture features based on the gray level co-occurrence matrix and three features were shape features.

4.
We believe that since samples in 2-Eucalyptus camaldulensis dehn (river red gum, class 2) have clear different shapes, shape features were selected as effective features, reaching almost no classification errors inside 2-Eucalyptus camaldulensis dehn (river red gum) class.

5.
Whenever features in different classes do not present overlapping, the statistical LDA method usually has a superior performance.

Conflicts of Interest:
The authors declare no conflict of interests.