SignDEG

Abstract

What a hand expresses lies largely in its articulated state: which fingers are selected, how they bend, how the palm is oriented. Encoders that learn from appearance alone see this state only indirectly, because appearance similarity also follows viewpoint and the person in the frame. Hand reconstruction models estimate the articulated state explicitly, but running one at deployment adds a second model to every video. SignDEG uses that geometry only as a pre-training target.

Frozen DINOv3 features of tracked hands and faces feed per-stream temporal Transformers trained by temporal self-distillation. Geometric latent prediction (GLP) supervises an intermediate block with WiLoR latents, MANO articulation and 3D joints, including at masked frames. Geometric relation transfer (GRT) matches cross-video frame-similarity distributions at the final block. Both objectives are learned jointly with self-distillation; WiLoR and the geometry heads are then discarded, so the encoder runs on video alone.

We evaluate on American Sign Language. Under the SHuBERT public-data evaluation protocol, SignDEG reaches 17.9/24.5 BLEU-4 on How2Sign/OpenASL, compared with 16.2/23.2 for SHuBERT. Rank-1 adaptation improves ASL Citizen and Sem-Lex recognition by 0.02 Recall@1 each, and phonological classification improves across 16 attributes. Removing GRT while retaining GLP reduces How2Sign BLEU-4 by 1.4 and ASL Citizen Recall@1 by 0.034.

Method

A YOLOv8n detector and ByteTrack follow the left hand, right hand and face of each video, and a frozen DINOv3 ViT-B/16 maps every crop to its 768-d class token once. Each stream has its own 10.8M-parameter temporal Transformer (6 blocks, width 384), trained by self-distillation between a student and its EMA teacher. The two hand streams additionally receive the geometric objectives.

Temporal self-distillation

Each sample draws two long (global) and several short (local) temporal crops. The teacher sees the global crops; the student sees all crops, with a random subset of frames in its global crops replaced by a learned mask embedding. The objective combines a crop-level DINO cross-entropy, its frame-level iBOT counterpart at masked frames, and a KoLeo regulariser; teacher distributions are normalised by Sinkhorn–Knopp iterations.

Geometric targets and reliability

For every available hand crop, a frozen WiLoR supplies the geometric latent u (the mean of its 16 MANO pose-query tokens after backbone block 23, standardised), the 15 local finger-joint rotations θ in 6D, and the wrist-relative 3D joints J. Per-frame regressors fail on blurred and interacting hands, and such failures appear as abrupt jumps in θ; each frame is therefore weighted by the detector score, tracker state, WiLoR validity and a temporal-smoothness term.

Results

All results follow the SHuBERT public-data evaluation suite: the ≈984-hour YouTube-ASL source split with OpenASL evaluation clips replaced by non-overlapping YouTube-SL-25 videos, and SHuBERT's target splits, task heads, metrics and downstream schedules. SignDEG entries are means over three seeds; baselines are published results.

Sign language translation

Isolated sign recognition & fingerspelling

What the geometric objectives contribute

The controls below hold the image backbone, crops, encoder capacity, 984-hour source split, 450K updates and downstream schedules fixed, and change only the pre-training objective.

A frozen retrieval diagnostic uses human handshape labels instead of WiLoR: ASL Citizen queries retrieve training clips while the query signer and the query lexical sign are excluded.

Phonological feature classification

Sixteen classifiers for ASL-Lex 2.0 categories test whether the representation preserves sub-lexical structure. On Sem-Lex, Handshape improves from 62.93 to 71.16, Selected Fingers from 79.53 to 89.94, and Spread from 79.42 to 90.99. These labels are independent of WiLoR.

Feature clusters

Code

The method code — articulator streams and feature extraction, pre-training with GLP and GRT, and the translation, recognition and fingerspelling models — is on GitHub under the MIT licence. Configurations follow the paper.

git clone https://github.com/junyi2005/signDEG.git
cd signDEG && pip install -r requirements.txt

# synthetic smoke test, no data needed
python train.py --config configs/signdeg_hand.yaml --synthetic-smoke --max-steps 20

# pre-training of one hand stream on a FeatureStore (see README)
torchrun --nproc_per_node 4 train.py --config configs/signdeg_hand.yaml --stream rh \
    --store /path/to/store_source --output-root runs/ssl

Datasets and third-party models (DINOv3, WiLoR and MANO, ByT5-Base, the YOLOv8n detectors) are not redistributed; obtain each from its official release under its own terms.

Citation

@article{hu2026signdeg,
  title   = {{DINO} Eats Geometry: Self-Supervised Sign Representations that Absorb
             Articulated Hand Geometry},
  author  = {Hu, Junyi and He, Zhewen and Huang, Haomian and Li, Zhenhua and
             Li, Zhifei and Yuan, Shuaihang and Fang, Yi},
  journal = {arXiv preprint arXiv:2609.06296},
  year    = {2026}
}