Undergraduate, School of Mathematical Sciences, Fudan University
Research Assistant, New York University
Research
I work on multimodal alignment. Across quite different problems, the same question keeps returning: what is the right correspondence between a continuous visual stream and the discrete structure a downstream model actually needs — a sentence, or an action? Two threads follow from it.
Gloss-free translation, where the decoder is normally left to discover a latent cross-modal permutation on its own — and the large-scale, structure-preserving data needed to make any of it hold up outside near-frontal studio conditions.
Learned, context-dependent safety criteria for robots acting from raw egocentric RGB-D. Privileged geometry supervises training offline; the deployed policy builds no map and needs no per-scene tuning.
Publications
Gloss-free translation asks the decoder to learn translation and silently discover a latent cross-modal permutation at once — fragile, because sign languages do not follow spoken word order. VTaMo makes the alignment explicit at three granularities: entropy-regularized optimal transport with a learnable null token for local frame↔token correspondence, a learnable orthogonal transform over a memory queue for global calibration, and position-aligned contrastive learning over reordered features. A text-only recovery model restores fluent sentences from the decoded pseudo-gloss.
Existing SLT benchmarks are shot near-frontal, in studios, with a handful of signers, so state-of-the-art models break under viewpoint, background and signer shift. SignNet-1M is ~1M augmented clips across ASL, DGS and CSL from seven source corpora, varied along three structure-preserving axes — 3DGS novel-view rendering, diffusion-based scene editing, and cross-reenactment identity substitution — each tagged with its factor axis and severity. The paired Orig / Zero-shot / Trained protocol separates the coverage blind spot of existing benchmarks from the training value of augmented data; training on SignNet-1M recovers up to +14.71 BLEU-4.
A fixed clearance margin is wrong in both directions: set it high and the robot detours in open space, set it low and it cuts corners in clutter. A diffusion policy proposes K = 16 candidate trajectories from egocentric RGB-D, and a learned critic carrying a trajectory-dependent clearance budget scores them. ESDF geometry is privileged information used only offline to supervise the teacher — the deployed selector runs from RGB-D alone, with no map building. 0.783 SR / 0.611 SPL on HM3D against NavDP's 0.710 / 0.529, and 8/10 on the hardest real-world scene with a Unitree G1 trained purely in simulation.