Fine-grained food image retrieval is vital for applications such as dietary monitoring and personalized nutrition. While hashing-based methods are popular for their storage and computational efficiency, existing approaches often fail to exploit category-specific spectral-spatial features and tend to preserve redundant representations, thereby limiting their discriminative ability. To address these issues, we propose SAHNet, a hierarchical network with a multi-scale backbone and two key modules: Spectral-Spatial Information Mining (SSIM) for dual-branch spectral/spatial feature extraction, and Selective Feature Filtering (SFF) for enhancing critical cues while suppressing noise. SAHNet generates compact, discriminative hash codes and achieves state-of-the-art performance on four benchmark fine-grained food datasets.