# Study evaluates stability of LLM-generated proofs during program refactoring

Published: 2026-10-29 · Updated: 2026-10-29 · Category: Research

Source page: https://www.sipply.dev/story/study-evaluates-stability-of-llm-generated-proofs-during-5680ded2 · 中文: https://www.sipply.dev/zh/story/study-evaluates-stability-of-llm-generated-proofs-during-5680ded2

## Summary

A recent study introduced RefactorProof, a tool designed to assess the stability of proofs generated by large language models (LLMs) when subjected to program refactoring. The research highlights that specialized provers tend to maintain higher rates of unchanged-proof survival compared to reference proofs. However, both types of proofs demonstrate a level of brittleness when faced with structural changes in the code, indicating that while LLMs can generate verifiable code, their outputs may not always withstand modifications effectively.

## Key takeaways

- RefactorProof tool assesses LLM-generated proof stability during refactoring.
- Specialized provers show better performance than reference proofs in stability.
- Both proof types are vulnerable to changes, highlighting a need for robust verification.

## Why it matters

This research underscores the challenges of using LLMs for reliable code generation, especially in dynamic programming environments where refactoring is common.

- **Product managers:** Product managers should be aware of the limitations of LLM-generated proofs to ensure that products relying on such technology are robust against code changes.
- **Developers:** Developers can leverage insights from this study to better understand the reliability of LLM-generated code and the importance of testing after refactoring.
- **Students and job seekers:** Students and job-seekers should recognize the evolving landscape of AI in software development, particularly the need for skills in verification and testing of AI-generated outputs.

## Sources

- [How Stable Are LLM-Generated Proofs Under Program Refactoring? A Stress Test for Verifiable Code Generation](https://arxiv.org/abs/2610.36328) (arXiv cs.AI, 2026-10-29)

## 中文

### 研究评估程序重构过程中大型语言模型生成的证明的稳定性

最近的一项研究介绍了RefactorProof，这是一种旨在评估大型语言模型（LLMs）在程序重构时生成的证明稳定性的工具。研究强调，专用证明者在保持未改变证明的存活率方面通常优于参考证明。然而，两种类型的证明在面对代码结构变化时都表现出一定的脆弱性，这表明尽管LLMs可以生成可验证的代码，但其输出可能并不总能有效抵御修改。

**要点：**

- RefactorProof工具评估LLM生成的证明在重构过程中的稳定性。
- 专用证明者在稳定性方面表现优于参考证明。
- 两种证明类型都易受变化影响，强调了对强健验证的需求。

**为什么重要：** 这项研究强调了在动态编程环境中使用LLMs进行可靠代码生成的挑战，尤其是在重构常见的情况下。

---
Sipply: Sip AI News Simply. https://www.sipply.dev