The traceability of palm oil seed origins plays a vital role in ensuring transparency, legality, and sustainability across the palm oil supply chain. Recent advances in deep learning have created new opportunities to improve classification systems by leveraging both visual and contextual data. This study proposes a deep learning-based model for classifying the origin of palm oil seeds by integrating thermal imagery with agricultural data. Two convolutional neural network (CNN) architectures, ResNet50 and MobileNet, were evaluated under three experimental setups: using only thermal images, combining thermal images with agricultural features (socio-economic, soil, and spectral fruit characteristics), and applying hyperparameter tuning to the best-performing model. The results show that ResNet50 consistently outperformed MobileNet, particularly in multimodal configurations. The highest performance was achieved using ResNet50 with the Adam optimizer, a learning rate of 0.001, and a batch size of 16, resulting in training accuracy of 99.75%, validation accuracy of 99.92%, and test accuracy of 100.00%. Evaluation metrics confirmed the model’s robustness with precision, recall, and F1-score all reaching 100.00%. This research highlights the significant potential of combining thermal imagery and agricultural data in CNN-based models for accurate and reliable classification of palm oil seed origins. The approach can support traceability systems in the palm oil industry, offering a scalable and data-driven solution for ensuring supply chain integrity and sustainability.