Projects

Analyzing Corrupted Segmentation Maps in ControlNet: Robustness and Reliability-Aware Conditioning

Diffusion models like ControlNet let you steer image generation with structured inputs such as segmentation maps, but they quietly assume those inputs are clean and accurate. Real annotations rarely are! I studied what happens when they aren't, and whether the model can be taught to notice.

I corrupted ADE20K segmentation maps with three kinds of realistic annotation noise (boundary perturbation, region dropout, and label flipping) across four intensity levels, then measured how generation quality degraded using CLIP similarity and a boundary alignment score. Semantic corruption turned out to be the real threat: label flipping cut CLIP similarity by up to 32%, while the diffusion backbone shrugged off geometric boundary noise almost entirely. I also found a structural alignment paradox, where the baseline's alignment score rises under noise because the model is faithfully reproducing corrupted structure.

To address this, I introduced a lightweight, non-learned reliability estimator that scores conditioning quality from edge density and fragmentation, then uses that score to adaptively scale ControlNet's conditioning strength at inference time. It improved CLIP similarity by 18.0% under 50% label-flip noise with no retraining or architectural changes, showing that even simple input-quality awareness can make conditional generation meaningfully more robust.

I conducted this research as my final project for the Intro to Computer Vision course at MIT (6.s058) Spring 2026.

Node Perturbation for Gradient-Free Fine-Tuning of Large Language Models I'm currently working as an undergraduate researcher in Prof. Ila Fiete's lab at MIT, where I'm exploring gradient-free optimization of LLMs with mechanisms inspired by the brain! Typical fine-tuning techniques include supervised learning and reinforcement learning with human feedback (RLHF). These methods are effective but time intensive, especially with their reliance on back propagation. RL also often correlates with reward-hacking and can be unstable. Not to mention that the reliance on back propagation and partial derivates translates poorly to the processes that occur in the brain! My research explores how we can emulate the brain's own learning mechanisms to fine-tune LLMs using node perturbation (NP), a gradient-free alternative to backpropagation. I've run ablation studies across MLP depth and width and batch size to identify the regimes where NP outperforms backpropagation, and extended NP to LLM-scale training (0.5B+ parameters) by deriving a normalization scheme for the delta loss update, which resolved the instability that had previously made NP infeasible at that scale. This work is heading toward a paper comparing policy gradient methods (like RLHF/GRPO) to evolution strategies, including NP, as fine-tuning approaches for LLMs.

Computationally Emulating the Evolution of Eyesight using Reinforcement Learning I worked as an AI/ML undergraduate researcher at the Media Lab (Camera Culture group). I expanded an existing reinforcement learning (RL) framework in Python, PyTorch, and MuJoCo by training agents across diverse environments (multi-eye perception, adversarial setups) to test robustness and adaptability in navigation tasks. I also discovered an emergent dragonfly prediction strategy, where agents anticipated goal locations before arrival, demonstrating RL's ability to uncover novel optimization behaviors. I engineered a hybrid approach combining supervised models (CNNs, MLPs) with RL by passing distance-to-goal predictions as inputs, improving learning efficiency and enabling multisensory input modeling. I finally validated the approach through trajectory mapping and performance benchmarking, achieving competitive results with baseline RL while reducing training complexity