Can this be converted to a L1 Regularization problem?

Question

I have the following optimization problem $$\min_{x\in \mathrm{R}^N}||Ax-b||_2^2 \\ s.t. ~~x^Te=1 \\ x\geq 0$$ where $e$ is the all-ones vector. Is there a way to connect this to optimization problems of form $$\min_{x\in \mathrm{R}^N}||Ax-b||^2+\lambda ||x||_1$$ In fact, if you can suggest other ways to solve the original problem itself, I would be more than happy. My issue is that $A$ is roughly of dimensions $140000 \times 100000$. Thus, even though it is a convex problem, the scale is huge. Thus I believe I need to depend on some heuristic method.

Answer 1

Your feasible set is known as "the probability simplex", because discrete probability distributions have the same constraints that

$\sum_{i=1}^{n} p_{i}=1$

$p \geq 0$.

There have been many papers that deal with convex optimization problems on the probability simplex. These are often solved by general purpose QP solvers using active set methods or interior point methods. However, given the very large size of your problems and depending on the sparsity structure of $A$, those approaches are probably not appropriate.

Depending on your accuracy requirements, a first order method that might be appropriate for your problem is projected gradient descent. This requires one projection onto the probability simplex per iteration (this can be done in $O(n\log n)$ time) and a line search that requires one matrix-vector multiply per inner iteration of the line search.

See this recent ArXiv paper for an $O(n \log n)$ algorithm for the projection operation:

Weiran Wang, Miguel Á. Carreira-Perpiñán. 2013. Projection onto the probability simplex: An efficient algorithm with a simple proof, and an application. https://arxiv.org/abs/1309.1541

I'd be willing to attempt to solve your problem using this approach if you'd be interested in collaborating.

Answer 2

Ya your problem is superficially connected to the variational l1 problem.

Indeed your constraint can be rewritten as $C = \{x \in \mathbb R^N | \|x\|_1 \le 1\} \cap \mathbb R^N_+$, and if your forget the $\mathbb R^N_+$ part, all you have is an $\ell_1$-norm constrained least-squares problem, and has an equivalent variational form which is precisely your second problem.

Now, as regards, your original problem, since computing the euclidean projection on the constraint set $C$ (a simplex) is exact and "easy", you can unroll an accelerated projected scheme like FISTA, on it. See my answer to an equivalent question here.