Towards More Flexible and Efficient Non-Autoregressive Language Models
A joint IESL talk on learnable insertion-order dynamics and relayed latent computation for non-autoregressive language models.
Paper
Speakers: Dhruvesh Patel , Benjamin Rozonoyer
Insertion Based Sequence Generation with Learnable Order Dynamics
Learned Relay Representations for Forward-Thinking Discrete Diffusion Models
A joint talk on making non-autoregressive language models more flexible and efficient, covering two recent IESL group papers:
- Dhruvesh Patel presents Insertion Based Sequence Generation with Learnable Order Dynamics — LoFlexMDM, an insertion-based masked diffusion model that learns data-dependent insertion and unmasking rates via a generalization of discrete flow matching to variable-length sequences.
- Benjamin Rozonoyer presents Learned Relay Representations for Forward-Thinking Discrete Diffusion Models — Relay, a differentiable per-token channel that propagates latent information across denoising steps (trained with truncated BPTT) to make masked diffusion models “forward-thinking.”
Discussion points:
- How does learning the target generation order improve insertion-based diffusion without giving up tractable training?
- What information is worth relaying across denoising steps, and how does truncated BPTT keep it trainable at scale?
- Where do learnable-order and relayed-computation ideas complement each other for non-autoregressive language models?