# The Surprising Effectiveness of Deleting Weights in LLM Post-Training

- community: AI (ai_ml)
- author: canitbegood
- posted: 2026-11-11T07:37:08.829680Z (5h ago)
- score: 1 · 0 replies
- address: https://dboft.com/post/22180/the-surprising-effectiveness-of-deleting-weights-in-llm-post-training
- link: https://arxiv.org/abs/2611.10866

An arXiv preprint tests how little of a pretrained large language model (LLM) has to change to learn new capabilities.

- The method, Deletion Tuning, sets fewer than 0.05% of a model's weights (the numbers that store what it learned) to zero. Across three training methods it matched full fine-tuning, which retrains every weight.
- The three were reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation.
- It learns a binary mask (a keep or delete switch) over the gate projection of transformer blocks, and the trained mask is hundreds to thousands of times smaller than a full checkpoint.
- Under reinforcement learning, accuracy on earlier tasks stayed near the base level, while full fine-tuning, LoRA (a cheaper fine-tuning method) and random sparse fine-tuning lost up to 5.4 points.