Multi-objective process parameter optimization with reinforcement learning

Follow-up journal paper (Applied Sciences, 2025) ↗

Master’s thesis in Data Science, submitted to the Aachen Institute for Advanced Study in Computational Engineering Science (AICES), RWTH Aachen University, April 2023. Supervised by Prof. Benjamin Berkels and Prof. Robert Schmitt; industrial data provided by IconPro GmbH.

The problem

Defective products are expensive: parts rejected at the end of a production line are often scrapped entirely. Whether a part turns out well depends on the process parameters. Some can be controlled (pressure, temperature, speed, applied force), while others cannot (ambient humidity, material batch). Most processes are judged on several quality parameters at once, and these often conflict: a setting that improves one can make another worse. Today, adjusting the controllable parameters is usually left to experienced production engineers.

Most earlier work optimizes parameters for one specific process. The goal of this thesis was a general framework that works for any production dataset. When a predictive model flags that the current settings will produce poor quality, the framework suggests new controllable parameters in real time, leaves the uncontrollable ones untouched, and improves all quality targets together.

The thesis asked two questions:

1. Can a reinforcement learning (RL) model find controllable parameters that improve multiple objectives simultaneously?
2. Can it find Pareto-optimal solutions, where no objective can be improved without making another worse?

The framework also had to meet practical requirements:

• handle continuous and categorical parameters and objectives
• respect parameter bounds and keep uncontrollable parameters fixed
• support priorities between objectives
• keep suggestions close to the current settings, since large jumps are not acceptable in a running line
• avoid unstable regions, where a tiny change flips the predicted outcome

Approach

Predictive models as the environment. Experimenting on a real production line is expensive and risky. Instead, one neural network per quality parameter is trained on historical data and serves as a fast stand-in for the process. The RL agent proposes values for the controllable parameters, and the environment combines them with the fixed, uncontrollable ones to form the next state. It then scores that state with the predictive models.

A reward that encodes the constraints. Each objective $k$ gets its own reward in $[0, 1]$, evaluated at the new state $s_{t+1}$:

$$r_k = O_k^{\,C} \cdot (1 - d) \cdot S_k,$$ $$S_k(s) = \frac{1}{1 + \bigl(\lVert \nabla P_k(s) \rVert / K\bigr)^2}.$$

• $O_k$ measures how close the predicted quality is to its target. For continuous objectives this is $\max(1 - |P_k(s) - y^\ast|, 0)$; for categorical ones it is the probability of the desired class.
• $d$ is the Gower distance from the original settings $s_0$. It handles mixed continuous and categorical parameters and penalizes large changes.
• $S_k$ is a stability term. It penalizes points where the predictive model’s gradient is steep, because predictions there are unreliable.

The agent: MO-MPO with hybrid actions. After reviewing multi-objective RL methods, I built on Multi-Objective Maximum a Posteriori Policy Optimization (MO-MPO), using DeepMind’s Acme library. Alternatives such as thresholded lexicographic ordering, Pareto Q-learning and Pareto-DQN either optimize objectives one after another, rely on lookup tables that don’t scale, or struggle with large action spaces. MO-MPO learns a separate Q-function per objective and combines them in distribution space rather than adding up rewards. As a result, priorities are set per objective as KL-divergence bounds $\epsilon_k$ and are invariant to the scale of each reward. I extended the actor network to output a Gaussian over the continuous parameters and a softmax over each categorical one, so the agent can act in a hybrid action space.

Training and inference. The task has no natural end state, so each episode runs for a fixed number of steps. Episodes start from representatives of k-means clusters in the training data, so the agent learns to optimize from many different operating points. At inference time, the trained actor runs for 200 steps from the given settings, and the best-scoring set of parameters is returned.

Checking Pareto optimality. A solution is not Pareto-optimal if there is a direction $z$ that improves every objective at once, that is, $z \cdot (-\nabla F_i(x)) > 0$ for all objectives $i$. I turned this into a feasibility problem solved with linear programming (SciPy’s linprog). If the problem is infeasible, no such direction exists and the solution is Pareto-optimal for the continuous parameters.

Experiments and findings

Both case studies used real production data. In each, the model was asked to fix the 100 worst (“faulty”) samples in the test set.

Case study 1: sandpaper production

• Setup: 58 input parameters, 29 of them controllable (for example voltage, temperature, roller speed). The objectives are 10 grain-density measurements across the sheet, each with a target value of 100.
• Results: mean reward 0.70–0.84 per objective, or about 0.90 without the stability term. Mean distance from the original settings 0.19. 6 of 100 solutions were Pareto-optimal.

Case study 2: pinion gear forming

• Setup: 4 input parameters, and only one of them, the applied force, is controllable. The objectives are the radius and thickness of the gear, which conflict with each other.
• Results: mean reward 0.89 and 0.90. Mean distance 0.13. 81 of 100 solutions were Pareto-optimal.

What I found

• All objectives improved together. In both case studies, the suggested parameters moved every quality measure toward its target while staying close to the original settings and in stable regions of the predictive models (mean stability 0.75–0.88 for sandpaper, 0.98–0.99 for pinions).
• Priorities work as intended. Raising the priority of one sandpaper objective from 0.1 to 0.9 improved that objective, at the expense of another. This is exactly the trade-off the priorities are meant to control.
• The suggestions are realistic. As a sanity check, I took known-good samples, randomly perturbed their controllable parameters, and let the model repair them. The repaired parameters landed close to the original good ones (Gower distance 0.30 for sandpaper and 0.13 for pinions), and the predicted quality improved over both the perturbed and the original settings.
• Pareto optimality depends on problem size. With one controllable parameter and two objectives, 81% of solutions were Pareto-optimal. With 29 parameters and 10 objectives, only 6% were.
• Cost. Training took about 50 hours for the sandpaper case (1,500 episodes) and about 5 hours for the pinion case (500 episodes). Optimizing one pinion sample took about 12 seconds.

Limitations and next steps

• The framework is only as good as its predictive models, and it needs gradient-based models (here, neural networks) for the stability term and the Pareto check.
• Rare, extreme faults are under-represented in the training data and are improved less.
• MO-MPO learns a single policy, so new priorities require retraining. A natural extension is a preference-conditioned, multi-policy agent. Early stopping and model-based RL are other promising directions.

This work continued after the thesis and led to a journal paper, published in Applied Sciences (2025). It is listed on my publications page.