Abstract Expensive dairy products like ghee are at high risk of hazardous adulteration with cheaper fats, and traditional testing approaches such as FTIR device and conventional machine learning are too slow, complex, and lack the interpretability to be useful for practical food safety. To address this, we propose a new deep learning framework which is robust and highly interpretable.