WeedSense: Multi-Task Learning for Weed Segmentation, Height Estimation, and Growth Stage Classification

Method Summary

Dual-path UIB Encoder: Combines a Detail Branch for fine-grained spatial features with a Semantic Branch using Universal Inverted Bottleneck blocks for efficient context extraction.
Multi-Task Bifurcated Decoder: Enables joint learning of semantic segmentation, height regression, and growth stage classification in a unified framework.
Temporal Growth Decoder: Leverages transformer-based feature fusion with multi-head self-attention to capture complex relationships between visual features and growth attributes.
Comprehensive Dataset: 120,341 annotated images across 16 weed species, 11-week growth cycle with pixel-level masks, height measurements, and temporal labels.

Parameters

30.50M

GFLOPs

16.73

Speed

160 FPS

Architecture

WeedSense Architecture: The proposed framework consists of three main modules: A Dual-path UIB Encoder with parallel Detail and Semantic branches to extract multi-scale features; an Aggregation Layer that fuses these features through attention-guided operations; and a Multi-Task Bifurcated Decoder that simultaneously predicts semantic segmentation masks (17 classes including background), plant height (regression), and growth stage (11 classes).

Abstract

Weed management represents a critical challenge in agriculture, significantly impacting crop yields and requiring substantial resources for control. Effective weed monitoring and analysis strategies are crucial for implementing sustainable agricultural practices and site-specific management approaches. We introduce WeedSense, a novel multi-task learning architecture for comprehensive weed analysis that jointly performs semantic segmentation, height estimation, and growth stage classification. We present a unique dataset capturing 16 weed species over an 11-week growth cycle with pixel-level annotations, height measurements, and temporal labels. WeedSense leverages a dual-path encoder incorporating Universal Inverted Bottleneck blocks and a Multi-Task Bifurcated Decoder with transformer-based feature fusion to generate multi-scale features and enable simultaneous prediction across multiple tasks. WeedSense outperforms other state-of-the-art models on our comprehensive evaluation. On our multi-task dataset, WeedSense achieves mIoU of 89.78% for segmentation, 1.67cm MAE for height estimation, and 99.99% accuracy for growth stage classification while maintaining real-time inference at 160 FPS. Our multitask approach achieves 3× faster inference than sequential single-task execution and uses 32.4% fewer parameters.

Results

Segmentation

mIoU: 89.78% mF1: 94.54%

Height Estimation

MAE: 1.67 cm RMSE: 2.32 cm R²: 0.9941

Growth Stage Classification

Accuracy: 99.99% F1: 99.99%

Height Estimation Performance

Height estimation performance across the full measurement range (0.2-155 cm). Predictions cluster tightly along the perfect prediction line (y=x) with tolerance bands showing ±1cm, ±2cm, and ±5cm accuracy zones. The error distribution shows symmetric behavior around zero with no systematic bias.

Attention Activation Maps

Visualization of attention activation maps from the aggregation layer. Each task displays distinct activation patterns: segmentation shows uniform boundary-focused activation, height estimation concentrates on plant extremities, and growth stage classification focuses on stem and mature leaf regions containing temporal growth indicators.

Qualitative Comparisons

Visual comparison of weed segmentation across different methods. WeedSense provides more accurate boundary delineation and better precision for fine-grained plant structures compared to baseline methods.

Dataset

We present a novel multi-task temporal dataset of 16 weed species growth patterns for semantic segmentation, height regression, and growth stage classification. The dataset spans 11 weeks from sprouting through flowering, providing comprehensive coverage of the primary growth cycle for weed species commonly found in Midwestern cropping systems of the USA.

Overview

Total frames	120,341
Weed species	16
Growth duration	11 weeks
Total videos	349
Frame resolution	720 × 960 pixels
Annotation types	Segmentation masks, Height, Growth stage

Data Split

Training	80%
Validation	10%
Test	10%
Height range	0.2 - 155 cm
Growth stages	Week 1 - Week 11
Classes	17 (16 species + background)

Growth Progression Over 11 Weeks

Temporal progression of three representative weed species (SETFA, AMARE, ERICA). The figure demonstrates the diverse growth patterns and morphological changes captured in our dataset, from sprouting (Week 1) through flowering (Week 11).

Weed Species Distribution

Distribution of the 16 weed species in our dataset. Species are categorized by growth rates: fast-growing (>10 cm/week), medium-growing (5-10 cm/week), and slow-growing (<5 cm/week). The dataset shows significant variation in maximum height (17.3-155.0 cm) and growth patterns (1.70-14.06 cm/week).

Key Features

Comprehensive Annotations: Pixel-level segmentation masks generated using SAM2-Hiera-L with manual verification and correction.
Temporal Coverage: Weekly imaging from Week 1 (BBCH stage 11, first true leaf) through Week 11 (BBCH stage 60, initial flower appearance).
Height Measurements: 325 manual height measurements ranging from 0.2 cm to 155 cm with substantial intra-species variability.
Diverse Growth Patterns: Species exhibit growth rates from 1.70 cm/week (ERICA) to 14.06 cm/week (SORHA).
Controlled Environment: Data collected in greenhouse with 1000W High Pressure Sodium grow lights at optimal temperatures (30-32°C).

BibTeX

@article{sarker2025weedsense,
  title={WeedSense: Multi-Task Learning for Weed Segmentation, Height Estimation, and Growth Stage Classification},
  author={Sarker, Toqi Tahamid and Ahmed, Khaled R and Islam, Taminul and Rankrape, Cristiana Bernardi and Gage, Karla},
  journal={arXiv preprint arXiv:2508.14486},
  year={2025}
}