Yash Maheshwari
Researcher in small language models · Mountain View, California
I study small language models, and especially what they can learn when data, memory and compute are scarce. My papers ask why small recurrent models lose grammar late in pretraining, and what compressing a model below one bit per weight does to its robustness once it runs on a device. I train nearly all of these models from scratch on a MacBook using MLX. At Stanford, I work on benchmarks for AI agents in hospital administration with the Shah Lab, and on AI reading and handwriting tutors with the Lemons Lab.
Positions
-
2026 to present
Research Intern, Shah Lab, Stanford Medicine
With Prof. Nigam Shah. Benchmarks for AI agents doing healthcare administrative work, such as prior authorizations and denial appeals.
-
2025 to present
Research Intern, Lemons Lab, Stanford Graduate School of Education
With Prof. Chris Lemons. Kai, an AI reading tutor used by 1,200+ students, and PAWS, a handwriting tutor for kindergarteners.
-
2025 to present
Independent Researcher, small language models
Self-directed work on pretraining under tight data budgets and sub-1-bit compression, trained locally with MLX. Led to first-author papers at EMNLP 2026 and NeurIPS 2026 workshops.
-
Class of 2027
Student, Mountain View High School
Publications
2026
-
EMNLP 2026 · BabyLM WorkshopAcceptedArchival
Halved CLM Exposure Mitigates Late-Training Degradation in Small Recurrent Language Models
BabyLM Workshop at EMNLP 2026
Halving CLM exposure mitigates late-training BLiMP degradation in small RWKV-7 models; auxiliary objectives are not required for the observed stability, while learning-rate trajectory remains coupled to update count.
Small recurrent language models can degrade late in training under standard causal language modeling (CLM). On the BabyLM Strict-Small corpus, vanilla RWKV-7 peaks near 69% BLiMP early, then loses 2.57 to 3.85 pp by step 18k across all three seeds we ran. Halving how often CLM updates the weights mitigates this decline, via two mechanisms: alternating a backward MLM objective every other step (BiRWKV-7), or skipping every other gradient step (vanilla-skip). Vanilla-skip carries no auxiliary objectives yet beats BiRWKV-7 on peak BLiMP in every seed, so the auxiliaries are not necessary for stability. What halving changes is coupled: it reduces the CLM update count and doubles the learning-rate decay in update space. A constant-rate control still degrades at the two higher rates we tested, so a low terminal rate is not necessary for the decline; but at fixed update count, flooring the rate earlier recovers most of it. The update-space rate trajectory therefore also modulates the decline; it stays coupled to update count, and our design does not separate the two. On the official BabyLM 2026 pipeline our submission scores 68.70% BLiMP, above the GPT-2 baseline (65.23%, 98.4M) at about 28% as many parameters.
-
NeurIPS 2026 · ODI WorkshopAcceptedPoster
Below One Bit: Training Route and Budget Shape Robustness Under On-Device Storage Constraints
NeurIPS 2026 Workshop on On-Device Intelligence: Foundation Models under Real-World Constraints (ODI)
Under a fixed in-format token budget, how a sub-1-bit model was trained into its format changes how much it degrades on corrupted input, especially on CPython, but the contrast is strongly budget-dependent, so bit width alone does not describe a compressed on-device model.
Sub-1-bit weight quantization fits language models under stringent on-device storage budgets and is judged almost entirely on clean benchmarks. We ask what the route to those bits costs under corrupted input. Byte-level models trained from scratch across three corpora, four scales (0.86M to 10.03M parameters), 0.41 to 4.19 effective bits per weight, and six seeds per cell compare two routes to the same format at matched in-format token budgets: native quantization-aware training (QAT) and calibrated post-training quantization plus in-format fine-tuning (PTQ+FT). Under nine synthetic character-corruption families, the native route degrades less on CPython at every tested scale, width, and seed, and again on an independent Python corpus at 4.87M; on natural language the pooled advantage, normalized by each domain's own degradation scale, is 8 to 20 times smaller at comparable widths and not generally seed-verified. The contrast tracks the recipe, not the format: quadrupling the native token budget reverses the sign, and matching total lifetime tokens leaves no seed-verified difference at either code scale tested. On an Apple M5 Max the packed checkpoint is 8.8 to 22.9 times smaller than 16 bits per parameter, growing with scale, but with no fused sub-1-bit kernel, unpacking to execute costs 6.0 to 16.1 times keeping the weights resident at batch 1. Bit width alone does not describe a compressed on-device model; route and token budget belong on the spec sheet next to it.
-
Journal of Emerging InvestigatorsAcceptedIn press
Training and comparison of fine-tuned small language models as math tutoring assistants
Journal of Emerging Investigators (JEI)
Across nine fine-tuned Llama math tutors (1B, 3B and 8B parameters, 1 to 5 training epochs), parameter count mattered more than training time; the 8B model trained for five epochs gave a correct first step 93.65% of the time.
Recent breakthroughs in artificial intelligence (AI) have revolutionized not only computer science but also automation, enabling machines to perform tasks only humans could once do. AI tutoring assistants can support math education by providing step-by-step guidance, but their effectiveness depends substantially on model size and training duration. However, large language models require a very significant number of resources, making them impractical for schools. This study evaluates Llama 3.2 configurations with 1 billion (1B), 3 billion (3B), and 8 billion (8B) parameters, trained for 1, 3, or 5 epochs, as tutoring tools for elementary, middle, and high school math. We hypothesized that increasing a Llama 3.2 model’s parameter count would improve step-accuracy more effectively than increasing the number of epochs during fine-tuning. Using Google Colab and Unsloth’s open-source code, we fine-tuned the models on 538 math queries. We assessed performance on 126 test prompts based on their ability to avoid revealing answers, provide accurate step-by-step hints and guidance, and achieve high human-rated quality. The 8B model at 5 epochs performed best (93.65% accuracy, 0.946 rating), while the 1B model at 1 epoch lagged in performance (55.56% accuracy, 0.700 rating). These results suggest that small, fine-tuned language models can provide a practical balance between educational quality and efficiency while mitigating cost, making them viable as AI tutors for classrooms.
