Damaged roads cause many problems in transportation. Holes and cracks on the roads are hazardous to the drivers. The road damage can be manually identified by a transportation expert. However, this process was inefficient. Therefore, it is important to identify the damages on roads by using machine learning. In this study, a classification system based on the Siamese Convolutional Neural Network (SCNN) is developed to classify road images. The road images in our dataset are divided into two classes, i.e., hole and crack damage. The input images were converted to grayscale images. Then, we implemented an image segmentation method and the Canny edge detection to the grayscale images. Finally, we apply the SCNN to classify the images. Experimental results show that our approach can reach an accuracy of 84,38%.