Automated Essay Scoring (AES) presents a significant challenge in the field of education, especially when dealing with students’ handwritten final answers that require visual-based evaluation. This research proposes a deep learning approach using a Convolutional Recurrent Neural Network (CRNN) to assess the similarity between students’ handwritten final answers and reference answers. The dataset used consists of 12 questions, each with 30 handwritten responses, which were preprocessed through normalization and noise reduction techniques to ensure consistency. The CRNN model was trained using well-tuned hyperparameters—batch size of 32, a learning rate of 0.002, and the Adam optimizer—which yielded the best performance. Exceeding other baseline models including the standard CRNN (87.50%), 1D CNN (81.12%), and ResNet (62%), the model attained an accuracy of 91.28% and a loss of 1.148. Moreover, several train-test split strategies were investigated; the 40:60 split produced the best average accuracy of 91.28%, hence suggesting this arrangement as the most suitable for evaluation. Although the overall results are very good, the proposed model still suffers from shortcomings in some situations, the most prominent being in question 5a, where the accuracy drops to 77.48%. Additional investigations found that this is due to the imbalance of the dataset, compromising the model’s generalization capacity across answer classes. This implies that although the CRNN-based model has proven to be quite successful for AES of handwritten images, future research should focus on improving the balance of the dataset and increasing the number of samples per question to further improve reliability and accuracy