Appearance
Evaluation: KPIs
Six numbers say whether the tuner works. Each was fixed before its first result, each is computed from evidence files rather than claims, and each can fail. Two of six are met at the latest run.
KPIs
| KPI | what it measures | target | latest | ||
|---|---|---|---|---|---|
| K1 | beats factory | share of the 33 simulated joints where the tuned controller beats factory gains on both stop p90 and tracking RMS, on motion no stage saw, in a run its own gates admit | 100 % | 100 % | met |
| K2 | keeps the analytic start | joints whose tuned result ends > 10 % worse than the analytic start it began from (mean of the stop and tracking ratios) | 0 | 1 | not met |
| K3 | inside the stop budget | two simulated test joints built from measured real-joint twins: held-out stop p90 of the best integrator-free controller the hardware path can send, against a 3.6 mrad budget | ≤ 3.6 mrad | 7.0 mrad | not met |
| K4 | holds converge | the same two joints: worst share of holds whose error ends, and stays, within 1 mrad | ≥ 80 % | 6 % | not met |
| K5 | the simulation is the joint | the same two joints: |sim / real − 1| of held-out stop p90 under the same controller, simulated vs the real joints | ≤ 0.5 | 0.19 | met |
| K6 | reaches the oracle | worst regret, tuned cost / best-possible cost − 1, over 35 joints (oracle) | ≤ 0.15 | 2.03 | not met |
What the misses mean:
- K2: the one joint is the lever. Its analytic start saturates 2.3 % of held-out rows against the 2 % gate, so verify refuses it and the tuner falls back to the factory gains plus feed-forward. The start's raw numbers are better than what was delivered.
- K3, K4: a geared joint with stiction parks anywhere in its friction band (about τc / kp) under PD alone. With a small integrator the same joints reach 1.8 and 0.8 mrad (K3 met) and 47 % convergence (K4 still short), but the integrator is out by design. Within the 2× gain guard, no integrator-free setting tried gets there.
- K6: 12 of 35 joints land within 15 % of the oracle. The misses are the lever (2.03), the conveyor (1.16), 17 of the 20 finger joints (0.31–0.69), the three UR5e wrists (0.22–0.28) and one gantry axis (0.33). On the lever, the best admissible controller sits right at the saturation gate, and the search's coarse grid steps past it.
K1 is computed on eased arm tapes (each joint's tape slowed to what it can follow). On the original, uneased arm tapes it is 97 %.
Scoring
Every trace is scored in one set of terms (score.score(traces) -> Score), whether it came from a simulation or a real joint:
| term | definition |
|---|---|
| stop p50 / p90 / max | a hold is ≥ 0.25 s of zero commanded velocity at one target, followed by more motion. Its stop error is the mean of q − q_target over the hold's last 50 ms. p50, p90 and max are over the magnitudes; stops counts them |
| tracking RMS | RMS of q − q_target over every row of the tape (prelude dropped); also per motion class in by_class |
| ringing | per hold, RMS of the linearly detrended error over its settled second half (an oscillating hold, not an offset or a slow creep); the worst hold |
| crossings | per hold, how often the error crossed the target with hysteresis (a deadband of 5 % of the move, at least 0.02 mrad); the most in any hold |
| chatter | at holds, the RMS of the row-to-row change in requested torque over √2 (the kd buzz); the median run |
| saturation | share of rows with requested torque ≥ 98 % of the cap |
| overshoot | median excursion past the target as a fraction of the move (reported; not a gate) |
| complete | the run ended normally, has rows, and every position, velocity, torque, target and timestamp is finite |
The gates built on these terms are in the pipeline.
python
from pidtuna import Envelope, Gains, Servo, tapes
from pidtuna.plants.servo_sim import ServoSim, Truth
from pidtuna.score import score
servo = Servo(name="elbow", gains=Gains(kp=8.0, kd=0.4), torque_limit_nm=6.0,
envelope=Envelope(travel_rad=0.4, max_speed_rad_s=2.0, max_torque_nm=6.0, origin_rad=0.0))
joint = ServoSim(servo, Truth(inertia_kg_m2=0.03, viscous_nm_s_rad=0.05, coulomb_nm=0.25, gravity_sin_nm=1.2))
traces = [joint.run(tape, servo.as_found) for tape in tapes.evaluation(servo, seed=0)]
s = score(traces)
print(f"stop p90 {s.stop_p90_mrad:.1f} mrad over {s.stops} stops, tracking {s.tracking_rms_mrad:.1f} mrad")
print(f"ringing {s.ringing_mrad:.3f} mrad, crossings {s.crossings}, saturation {100 * s.saturation:.1f} %")
print({name: round(rms, 1) for name, rms in s.by_class.items()})text
stop p90 49.0 mrad over 16 stops, tracking 35.2 mrad
ringing 0.000 mrad, crossings 0, saturation 0.0 %
{'cycle': 31.5, 'jog': 19.6, 'pickplace': 38.7, 'sweep': 50.0}Held-out tapes
Evaluation tapes come in four motion classes, each drawn from a random seed and kept inside the servo's envelope:
- pickplace: four large moves at 60–100 % of top speed into 0.6–1 s holds.
- cycle: the same out-and-back move three times.
- jog: six small moves (5–15 % of travel) at modest speed, where the friction band dominates.
- sweep: one slow move across most of the travel.
The search scores candidates on seed 0. tune's verify step proves the pick on seed 1. The results below use seed 2, motion no tuner stage has seen. The arm check plays its own pick-and-place sets (seeds 101–107), and the multi-joint results use seed 202. Every tape is eased until the identified model can follow it on 80 % of the torque cap.
Nine robot kinds
Three controllers per robot, all on the same unseen motion. Factory is the conservative gains a vendor or a first bring-up would set. Analytic is the start from the description alone. Tuned is the full pipeline seeded with the analytic start, run on a hidden true plant the description gets partly wrong. Medians over each robot's joints:
| robot | motors | what the model does not know | stop p90 factory → analytic → tuned | tracking RMS | binding bound |
|---|---|---|---|---|---|
| lever (geared servo) | 1 | 15 % more inertia than described; 4.1 N·m of gravity | 125 → 2.04 → 3.10 mrad | 80.4 → 1.73 → 2.06 | linear range |
| conveyor | 1 | 15 kg of parcels; belt stretch | 113 → 2.55 → 0.284 mm | 90.5 → 1.32 → 0.187 | structural mode |
| UR5e arm | 6 | a 2 kg payload | 43.2 → 0.443 → 0.091 mrad | 39.6 → 0.478 → 0.200 | gain margin, linear range, quantization |
| Cartesian gantry | 3 | a 1 kg payload (belt X/Y, ball-screw Z) | 12.6 → 0.057 → 0.020 mm | 15.0 → 0.093 → 0.056 | linear range |
| five-bar arm | 2 | a 0.5 kg payload; closed chain, 2 driven + 2 passive joints | 85.4 → 1.13 → 0.276 mrad | 70.8 → 0.736 → 0.360 | linear range |
| hand, 2 motors | 2 | true friction and rotor inertia | 364 → 1.52 → 0.484 mrad | 257 → 1.41 → 0.781 | gain margin |
| hand, 3 motors | 3 | the same | 403 → 1.58 → 0.513 mrad | 299 → 1.40 → 0.806 | gain margin |
| hand, 6 motors | 6 | the same | 404 → 3.00 → 0.773 mrad | 289 → 2.80 → 0.791 | gain margin |
| hand, 9 motors | 9 | the same | 403 → 1.09 → 0.945 mrad | 274 → 1.25 → 0.767 | gain margin |
Tool-point RMS error from forward kinematics: UR5e 38.5 → 0.61 → 0.33 mm, gantry 32.3 → 0.17 → 0.14 mm.
These are simulations with an opened guard
The factory gains here are deliberately soft, and a finger's analytic start sits about 700× above them, so these runs open the gain guard to 0.001–1000×. On a real joint the guard stays at 0.25–2× and a session moves gains at most 2× per run.
Every robot replays in MuJoCo on the examples page.
Real joints
Before the analytic start existed, the tuner ran from factory gains, inside the 0.25–2× guard, on three real test joints: two geared servos and one direct drive. Held-out stop p90 went 48.5 → 13.6, 38.1 → 4.58 and 15.5 → 6.32 mrad; tracking RMS 38.1 → 10.6, 35.8 → 5.96 and 16.4 → 6.93 mrad. Those picks remain proposals. Everything since has been simulated, which is the main limit on these numbers.