Troubleshooting Validation of LLM-Modernized Scientific Software with Differential Fault Injection

Symptoms of Validation Failure

When an LLM rewrites legacy Fortran for a production scientific code, the new implementation may pass standard unit tests yet diverge from the original under fault conditions. Common symptoms include final energy values that differ by more than the expected numerical tolerance after identical perturbations, convergence failures that appear only when the code runs with reduced floating‑point precision, and intermittent deadlocks in parallel regions that were not present in the original. These signs indicate that the modernization has not preserved the original code’s response to faults.

The problem arises because traditional testing focuses on nominal execution, where no faults are injected. Scientific software often relies on subtle error propagation and iterative convergence mechanisms that are sensitive to precision loss. Without targeted fault injection, small differences in error handling can grow and alter the final outcome.

Root Causes in LLM‑Generated Code

The transformed code often introduces subtle changes in error propagation. Possible causes are:

Such differences become visible only when the software is stressed with controlled faults.

Diagnostic Procedure with Differential Fault Injection

A systematic validation can be performed with a differential fault‑injection harness. The steps are:

This procedure isolates the converted integral kernels and reveals whether they react to faults in the same way. For a detailed harness implementation, see the Gaionix documentation.

Evidence from the GAMESS Case Study

The method was tested on the GAMESS quantum‑chemistry package. The research team instrumented the self‑consistent‑field driver at twelve sites and conducted a large‑scale fault‑injection campaign involving many runs. The cost of absorbing transient faults was consistent with a contraction‑based model, with predicted and measured slopes in close agreement. Persistent perturbations reduced the final‑energy error as precision increased.

The campaign uncovered phase‑dependent parallel deadlocks and false convergence when the code ran with reduced precision. The original and modernized kernels agreed in all paired injections, and a measurement‑guided synchronization change composed with the modernization, matching in all tested pairs.

Remediation and Synchronization Fixes

When discrepancies appear, the first action is to align the synchronization points in the parallel driver. The GAMESS study shows that a small change guided by fault‑injection measurements restores agreement. After the fix, repeated injections confirm that the modernized kernel behaves like the original.

Limitations and Ongoing Monitoring

Differential fault injection requires access to the original source code and a representative set of fault patterns. It may not capture all possible software states, so continuous monitoring in production is advisable. The technique is most effective when integrated into the continuous integration pipeline.