Efficient Constant Optimization for Symbolic Regression with GPU-Accelerated Tree-Based Genetic Programming

Efficient Constant Optimization for Symbolic Regression with GPU-Accelerated Tree-Based Genetic Programming is a paper submitted on September 3, 2026 (arXiv:2609.03352) that introduces a GPU-resident, batched Levenberg–Marquardt solver to address the high computational cost of constant optimization in tree-based genetic programming.

The study highlights that while constant optimization refines numerical coefficients, its per-generation cost has led modern GPU-accelerated frameworks like EvoGP to omit it or restrict it to lightweight forms, limiting their ability to recover governing equations with inner nonlinear constants. To resolve this, the authors developed a solver that uses reverse-mode automatic differentiation to assemble per-tree Jacobians in a single backward sweep, ensuring the dominant per-iteration cost is independent of the number of constants per tree.

Key performance metrics and findings include: Throughput: On early-generation populations, the solver sustains up to $5.1 \times 10^5$ trees per second on an NVIDIA A100. Speedup: It delivers roughly $9.9\times$ the throughput of the CPU-based Operon library running on a 64-core EPYC 7763, while matching fp64-reference quality. Effectiveness: Integrated into EvoGP, the solver enables end-to-end search to recover governing equations on 10 of 18 constructed problems requiring inner-constant fitting, compared to 0 for stock EvoGP. Efficiency: Applying constant optimization every fifth generation is statistically indistinguishable from applying it every generation, allowing for significant cost amortization.

This material addresses a practical bottleneck in GPU-accelerated symbolic regression: the cost of constant optimization in tree-based genetic programming. In symbolic regression, candidate models are often symbolic expressions with tunable numerical coefficients, and refining those constants can substantially improve both fit and interpretability. However, because each tree may require its own nonlinear least-squares solve, per-generation constant optimization has traditionally been expensive, especially in large populations. As a result, many modern GPU-oriented frameworks either omit constant optimization entirely or restrict it to simple, low-cost forms, sacrificing model quality for throughput.

The paper’s central contribution is a GPU-resident, batched Levenberg–Marquardt solver designed to optimize the constants of a heterogeneous population of tree structures efficiently. Rather than treating each expression as an isolated small-scale optimization problem on the CPU, the approach keeps the optimization loop on the GPU and batches many constant-fitting problems together. This is important because tree-based populations are structurally diverse: expressions differ in depth, operator composition, and number of free constants. The method is therefore positioned to handle that heterogeneity while still exploiting GPU parallelism and memory locality, turning what is normally a collection of many small, latency-bound solves into a more scalable batched computation.

The broader significance is that this work helps close an efficiency gap between classical symbolic regression and modern large-scale GPU genetic programming. By making constant optimization feasible at every generation, rather than only as a post-hoc refinement step, the approach can improve the search signal, produce more accurate candidate expressions, and reduce the need for manual or heuristic coefficient handling. For technically oriented readers, the contribution matters because it re-enables a standard but important modeling capability in high-throughput evolutionary search, potentially improving the quality of discovered symbolic models in scientific discovery, data-driven modeling, and interpretable machine learning while preserving the performance advantages of GPU acceleration.

The paper addresses a practical bottleneck in GPU-accelerated tree-based genetic programming for symbolic regression: constant optimization. In this setting, each candidate expression is a tree whose structure is evolved by genetic operators, while its numerical leaves are refined by local optimization to better fit the target data. Although this step can substantially improve expression quality, it is expensive when applied to every individual in every generation. As a result, many modern high-throughput GPU frameworks either skip constant optimization entirely or use only simplified, lightweight variants that do not fully exploit the available search space.

The main contribution is a GPU-resident, batched Levenberg–Marquardt solver designed to optimize constants across a structurally heterogeneous population of symbolic expressions. Rather than treating each expression as an isolated, small-scale optimization problem on the CPU or as a separate GPU kernel launch, the method batches the required linear algebra and keeps the relevant state on the GPU. This allows the solver to handle variable expression sizes and structures while amortizing memory-access and kernel-launch overhead. The practical effect is to make per-generation constant tuning cheap enough to be integrated into large-scale evolutionary search without negating the throughput advantages of GPU acceleration.

This matters because it removes one of the key tradeoffs in modern symbolic regression systems: either run very fast but structurally biased search, or run more expensive local refinement that may improve accuracy but limits population size or generation count. By making constant optimization scalable on GPUs, the work enables more complete fitness evaluation, tighter coupling between symbolic structure search and numerical parameter fitting, and potentially better final expressions for scientific discovery, program synthesis, and data-driven model learning tasks.