🐟 Benchmarking Deep Learning Models for Fish Classification with Explainability and Model Transparency 🔎

Aquaculture is the fastest-growing food production sector of this century. Although artificial intelligence has driven numerous advancements in this field, no studies have focused on the risks associated with fish escapes from aquaculture farms. These events can cause environmental issues, disrupt wild populations, compromise the health of other species and negatively impact both aquaculture and traditional fisheries. This work introduces an innovative approach to determining the origin of two economically significant Mediterranean fish species, Sparus aurata and Dicentrarchus labrax, using a CLIP- based multimodal framework combined with a lightweight Linear Probe classifier. By integrating expert-crafted textual descriptions with visual embeddings, the proposed system achieves high classification accuracy, outperforming traditional CNN and ViT models, which require extensive pretraining. Additionally, the methodology is validated with various explainability techniques, including GRAD-ECLIP, expert-driven feature importance anal- ysis and manual feature manipulation, ensuring a transparent and interpretable system for aquaculture professionals. Eval- uations conducted on wild, farmed and escaped fish and the validation done demonstrate the model’s strong performance and its potential to improve traceability and management in aquaculture.

The proposed classification framework is based on the CLIP model, leveraging both textual descriptions and visual embeddings to distinguish between wild and farmed fish.
The architecture of our approach follows these main components:

1. Vision Encoder (ViT-B/16): The image of a fish is processed through a Vision Transformer encoder, generating a high-dimensional feature representation.
2. Text Encoder (GPT Transformer): Expert-curated textual descriptions are encoded into a multimodal space using CLIP’s text encoder.
3. Feature Aggregation: Multiple textual embeddings (T₁, T₂, ..., Tₙ) are combined to refine the classification decision. Additionally, weighted feature vectors, WF (Wild Features) and CF (Cultivated Features), are computed as the average of image embeddings extracted from a training set.
4. Cosine Similarity Computation: The embeddings from the image, text encoders, and weighted features are compared using cosine similarity to determine the alignment between visual and textual features.
5. Classification Methods: The concatenated embedding (e.g., I1T2) is passed through different classifiers including Linear Probe (LP), k-Nearest Neighbors (KNN), Random Forest (RF), and Support Vector Machine (SVM).

Main feature prompts and visual descriptions for S. aurata and D. labrax:

🔎 Explanation:
Among CNNs, InceptionV3 performs best, especially on S. aurata. However, ViT-B/32 surpasses all CNNs with outstanding accuracy across both species. On the multimodal side, CLIP stands out, significantly outperforming other vision-language models. This confirms that integrating image and text improves fish classification performance, especially when paired with robust visual encoders like ViTs.

🔎 Explanation:
When used with CLIP, ViTs consistently outperform ResNets. Notably, ViT-B/16 and ViT-B/32 achieve balanced performance across species. ViT-L/14 excels in D. labrax detection but is slightly weaker for S. aurata. Among ResNets, RN50 is the most balanced, while RN50x4 shows instability between classes highlighting the robustness of transformer-based architectures.

🔎 Explanation:
Across all ViT variants, SVC and Linear Probe (LP) consistently outperform simpler classifiers like KNN and Random Forest (RF).

• For ViT-B/16, both SVC and LP reach 100% accuracy on S. aurata and 93.5% overall.
• ViT-B/32 follows a similar trend, with LP achieving 92.0% overall.
• The strongest results come from ViT-L/14 + SVC, reaching 95.0% total accuracy with high precision for both species.

These results confirm that ViT embeddings are highly expressive and benefit most from classifiers capable of leveraging complex feature spaces.

🔍 Feature relevance:

For D. labrax, the standard and average-based T-SNE projections consistently align textual prompts with specimen classification: wild features cluster on the left, and farmed features on the right. For S. aurata, the trend is reversed—farmed on the left, wild on the right but both projections maintain internal consistency.

Regarding feature importance, gray and orange stand out as the most dominant (related to Color), followed by brown and blue (Lateral Line). These traits are especially consistent in S. aurata, aligning with expert knowledge that prioritizes Color and Lateral Line as key identifiers.

In D. labrax, the model emphasizes morphological traits, brown, green, gray, pink, blue, and orange, over pigmentation (red and purple), suggesting it relies more on body shape, fin positioning, and lateral line for classification. This reinforces the model’s focus on structural features over color when identifying this species.

🔍 Image Analisys:

The images above illustrate the behavior of our model using S. aurata as an example. When using the Mouth Prompt (right), the model concentrates its attention on the fish’s mouth region, reinforcing its importance in classification. In contrast, the Color Prompt (left) leads the model to shift focus toward pigmentation related areas, showing its ability to adapt feature selection based on textual input.

This pattern is consistently observed in D. labrax as well, where the model initially detects general fish contours before refining attention based on the specific prompt. The fact that the model dynamically adjusts focus while consistently extracting meaningful features in both S. aurata and D. labrax confirms the effectiveness of both the model and the prompt design.

These saliency maps strongly support that the classification is based on biologically relevant traits rather than random correlations, reinforcing the validity and interpretability of our approach.

This work presents a novel CLIP-based multimodal framework for identifying the origin of two key Mediterranean fish species: Sparus aurata and Dicentrarchus labrax. By combining expert-defined textual prompts with visual embeddings, the model achieves high classification accuracy, surpassing traditional CNN and ViT-based methods.

Coupling CLIP embeddings with a lightweight linear probe balances performance and interpretability, while explainability tools such as GRAD-ECLIP, feature analysis and prompt manipulation confirm alignment with biological expertise.

The system proves effective even in complex real-world scenarios, such as escaped fish with mixed traits, where it helps flag labeling inconsistencies and supports traceability in aquaculture.

Looking ahead, the framework will be adapted for real-time use on embedded devices and mobile applications, enabling easy deployment in ports, fish farms and fieldwork. The integration of in situ imagery will further test its robustness in uncontrolled conditions.

Overall, this study highlights the potential of vision-language models to improve transparency and reliability in morphology-based classification, advancing the application of AI in sustainable aquaculture.

The study was funded by the project “GLObal change Resilience in Aquaculture-TOOls for Long-term Sustainability (GLORiA-TOOLS),” supported by the Biodiversity Foundation of the Spanish Ministry for the Ecological Transition and Demographic Challenge through the Pleamar Program and co-financed by the European Maritime, Fisheries and Aquaculture Fund (EMFAF).

├── images/              # Visuals and figures used in the README
├── CNNs/                # Trained model checkpoints for CNN-based architectures
├── ViTs/                # Trained model checkpoints for Vision Transformers
├── CLIP/                # Trained model checkpoints for CLIP-based models
└── README.md            # This file