Solve system of overdetermined simultaneous equations

Question

I have a large number (tens not hundreds) of parameters: a₁, a₂, a₃ ... , a_n and b₁, b₂, b₃ ... , b_n which I want to find using linear algebra.

And a system of simultaneous equations:

a₁b₂ = 4
a₂b₁ = 2
...
a₄b₇ = 3
a₇b₄ = 0
...

Note:

these formulas always take the form a_nb_m = x
x is always a positive integer (or zero)
the equations always come in pairs: a_nb_m = x & a_mb_n = y
we never get a_nb_n = x equations (e.g. a and b values with the same index in the same equation)
each parameter (e.g. a₁ or b₆) will occur in multiple equations
not ever combination of a_nb_m will necessarily be present in the system of equations
To constrain the system, I'm happy to assume the mean of the b values is 1
There will be enough equations to over constrain the parameters, I'm looking for the optimal solution, not an exact solution.

Is it possible to pose this problem in a matrix form such that I can solve it using matrix inversion and a QR transform? I'm currently using scipy.optimize.minimize() which is very slow.

If the formulas took the form a_n + b_m = x this would be relatively simple to solve using something like this, but the parameters are multiplied, not added :-(.

Part of the problem I'm having while trying to find a mathematical solution is I'm not sure what to call this sort of problem. Is there a name for problems like this?

@Bananach That was my initial thought, too. But what about $a_7 b_4 = 0$? — RobPratt, Apr 27 '21 at 20:43
@RobPratt, very good point I thought logs would solve this but I don't know how to work around the zero case. — Samuel Colvin, Apr 27 '21 at 20:49
Replace zeros by small positive numbers, or try solving it assuming that any subset of your variables is zero and the rest isn't — Bananach, Apr 27 '21 at 22:27
That's what I thought, I guess I could even add a small number to all values. — Samuel Colvin, Apr 28 '21 at 10:03

vb628 · Answer 1 · 2021-04-27T20:25:15.080

1

It seems you are trying to find a non-linear least squares solution to something that looks like

$$ Y_{k\ell} - a_k \: b_\ell = 0\text{ .} $$

We seek to minimize the squared residuals between the data and the model you use to fit the data. For you it would look something like this.

$$R = \sum_{k\neq\ell} \big(Y_{k\ell} - a_k \: b_\ell\big)^2 $$

Minimization will happen when the gradient $\nabla R$ with respect to all parameters $a$ and $b$ are equal to zero. Taking derivatives to find equations

$$\begin{align} \dfrac{\partial R}{\partial a_j} &= 0\\\dfrac{\partial R}{\partial b_j} &= 0 \end{align}$$

you should now be able to solve a least-squares system using Levenberg-Marquardt or whatever algorithm you wish.

edited Apr 27 '21 at 20:25

answered Apr 27 '21 at 19:21

vb628

1,784
5
12

Thanks for the suggestion. I think the best solution is that suggested by @Bananach above - take logs of the equations: log(a1*b12 == log(a1) * log(b2) with that it can be solve like this. – Samuel Colvin Apr 27 '21 at 20:45
2

Yeah that should be valid. There are some edge cases however where taking the log of your equations and using linear least squares is far from the actual least squares solution. You should be fine here though. If curious, take a look at this. – vb628 Apr 27 '21 at 22:36
1

I second the least squares approach, and scipy has an API dedicated to that. – jadelord May 03 '21 at 08:32

Solve system of overdetermined simultaneous equations

1 Answers1