Abstract
The augmented Lagrangian method (also called as method of multipliers) is an important and powerful optimization method for lots of smooth or nonsmooth variational problems in modern signal processing, imaging and optimal control. However, one usually needs to solve a coupled and nonlinear system of equations, which is very challenging. In this paper, we propose several semismooth Newton methods to solve arising nonlinear subproblems for image restoration in finite dimensional spaces, which leads to several highly efficient and competitive algorithms for imaging processing. With the analysis of the metric subregularities of the corresponding functions, we give both the global convergence and local linear convergence rate for the proposed augmented Lagrangian methods with semismooth Newton solvers.
Similar content being viewed by others
Data Availability
Enquiries about data availability should be directed to the authors.
References
Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, New York (2011)
Bertsekas, D.P.: Constrained Optimization and Lagrange Multiplier Methods. Academic Press, Paris (1982)
Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging and Vis. 40(1), 120–145 (2011)
Clarke, F.H.: Optimization and Nonsmooth Analysis, Vol. 5, Classics in Applied Mathematics, SIAM, Philadelphia (1990)
Clason, C.: Nonsmooth Analysis and Optimization, Lecture notes, arXiv:1708.04180, (2018)
Cui, Y., Sun, D., Toh, K.: On the R-superlinear convergence of the KKT residues generated by the augmented Lagrangian method for convex composite conic programming. Math. Program., Ser. A, 178, 38–415 (2019). https://doi.org/10.1007/s10107-018-1300-6
De Luca, T., Facchinei, F., Kanzow, C.: A semismooth equation approach to the solution of nonlinear complementarity problems. Math. Program. 75, 407–439 (1996)
Dontchev, A.L., Rockafellar, R.T.: Functions and Solution Mappings: A View from Variational Analysis, 2nd edn. Springer Science+Business Media, New York (2014)
Facchinei, F., Pang, J.: Finite-Dimensional Variational Inequalities and Complementarity Problems, vol. I. Springer-Verlag, New York Inc (2003)
Fortin, M., Glowinski, R. (eds.): Augmented Lagrangian Methods: Applications to the Solution of Boundary Value Problems. North-Holland, Amsterdam (1983)
Glowinski, R., Osher, S., Yin, W. (eds.), Splitting Methods in Communication, Imaging, Science, and Engineering, Springer (2016)
Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1968)
Hintermüller, M., Kunisch, K.: Total bounded variation regularization as a bilaterally constrained optimization problem. SIAM J. Appl. Math 64(4), 1311–1333 (2004)
Hintermüller, M., Papafitsoros, K., Rautenberg, C.N., Sun, H.: Dualization and automatic distributed parameter selection of total generalized variation via bilevel optimization. Numer. Funct. Anal. Optim. 40(8), 887–932 (2022). https://doi.org/10.1080/01630563.2022.2069812
Hintermüller, M., Stadler, G.: A infeasible primal-dual algorithm for total bounded variaton-based inf-convolution-type image restoration. SIAM J. Sci. Comput. 28(1), 1–23 (2006)
Hoffman, A.J.: On approximate solutions of systems of linear inequalities. J. Research Nat. Bur. Standards 49, 263–265 (1952)
Klatte, D., Kummer, B.: Constrained minima and Lipschitzian penalties in metric spaces. SIAM J. Optim. 13(2), 619–633 (2002)
Klatte, D., Kummer, B.: Nonsmooth Equations in Optimization. Regularity, Calculus, Methods and Applications, Series Nonconvex Optimization and Its Applications, Vol. 60, Springer, Boston, MA (2002)
Ito, K., Kunisch, K.: Lagrange Multiplier Approach to Variational Problems and Applications, Advances in design and control 15, Philadelphia, SIAM (2008)
Ito, K., Kunisch, K.: An active set strategy based on the augmented Lagrangian formulation for image restoration. RAIRO, Math. Mod. and Num. Analysis 33(1), 1–21 (1999)
Leventhal, D.: Metric subregularity and the proximal point method. J. Math. Anal. Appl. 360(2009), 681–688 (2009)
Mifflin, R.: Semismooth and semiconvex functions in constrained optimization. SIAM J. Control Optim. 15(6), 959–972 (1977)
Li, X., Sun, D., Toh, C.: A highly efficient semismooth Newton augmented Lagrangian method for solving lasso problems. SIAM J. Optim. 28(1), 433–458 (2018)
Luque, F.J.: Asymptotic convergence analysis of the proximal point algorithm. SIAM J. Control Optim. 22(2), 277–293 (1984)
Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization, pp. 283–298. Academic Press, New York (1968)
Reed, M., Simon, B.: Methods of Modern Mathematical Physics: Functional Analysis, Methods of Modern Math. Phys. 1, Academic Press, New York (1980)
Robinson, S.M.: Some continuity properties of polyhedral multifunctions. In: König, H., Korte, B., Ritter, K. (eds.) Mathematical Programming at Oberwolfach, Math. Program. Stud., Springer, Berlin, Heidelberg, pp. 206–214 (1981)
Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14(5), 877–898 (1976)
Rockafellar, R.T.: Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1(2), 97–116 (1976)
Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D. 60(1–4), 259–268 (1992)
Scholtes, S.: Introduction to Piecewise Differentiable Equations, Springer Briefs in Optimization, Springer, New York (2012)
Stadler, G.: Semismooth Newton and augmented Lagrangian methods for a simplified friction problem. SIAM J. Optim. 15(1), 39–62 (2004)
Stadler, G.: Infinite-Dimensional Semi-Smooth Newton and Augmented Lagrangian Methods for Friction and Contact Problems in Elasticity, PhD thesis, University of Graz (2004)
Sun, D., Han, J.: Newton and quasi-Newton methods for a class of nonsmooth equations and related problems. SIAM J. Optim. 7, 463–480 (1997)
Tomioka, R., Sugiyama, M.: Dual-augmented Lagrangian method for efficient sparse reconstruction. IEEE Signal Process. Lett. 16(12), 1067–1070 (2009)
Ulbrich, M.: Semismooth Newton Methods for Variational Inequalities and Constrained Optimization Problems in Function Spaces, MOS-SIAM Series on Optimization (2011)
Van der Vorst, H.A.: Iterative Krylov Methods for Large Linear Systems. Cambridge University Press, Cambridge (2003)
Ye, J., Yuan, X., Zeng, S., Zhang, J.: Variational analysis perspective on linear convergence of some first order methods for nonsmooth convex optimization problems. Set-Valued Var. Anal. 29, 803–837 (2021). https://doi.org/10.1007/s11228-021-00591-3
Yu, P., Li, G., Pong, T.K.: Deducing Kurdyka-Lojasiewicz exponent via inf-projection. Found. Comput. Math. (2021). https://doi.org/10.1007/s10208-021-09528-6
Zhang, F. (ed.): The Schur Complement and Its Applications, Numerical Methods and Algorithms 4, Springer US (2005)
Zhang, Y., Zhang, N., Sun, D., Toh, K.: An efficient Hessian based algorithm for solving large-scale sparse group Lasso problems. Mathematical Programming A 179, 223–263 (2020). https://doi.org/10.1007/s10107-018-1329-6
Zhao, X., Sun, D., Toh, K.: A Newton-CG augmented Lagrangian method for semidefinite programming. SIAM J. Optim. 20(4), 1737–1765 (2010)
Acknowledgements
The author acknowledges the constructive comments from the anonymous referee, which greatly improve the paper. The author also acknowledges the support of Beijing Natural Science Foundation No. Z210001 and NSF of China under Grant No. 11701563. The work originated during the author’s visit to Prof. Defeng Sun of the Hong Kong Polytechnic University in October 2018. The author is very grateful to Prof. Defeng Sun for introducing the framework on semismooth Newton based ALM developed by him and his collaborators and for his suggestions on the self-adjointness of the corresponding operators for Newton updates where CG can be employed. The author is also very grateful to Prof. Kim-Chuan Toh, Dr. Chao Ding, Dr. Xudong Li and Dr. Xinyuan Zhao for the discussion on the semismooth Newton based ALM. The author is also very grateful to Prof. Michael Hintermüller for the discussion on the primal-dual semismooth Newton method during the author’s visit to Weierstrass Institute for Applied Analysis and Stochastics (WIAS) supported by Alexander von Humboldt Foundation during 2017.
Funding
The authors have not disclosed any funding.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The authors have not disclosed any competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Sun, H. An Investigation on Semismooth Newton based Augmented Lagrangian Method for Image Restoration. J Sci Comput 92, 82 (2022). https://doi.org/10.1007/s10915-022-01907-7
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10915-022-01907-7
Keywords
- Augmented Lagrangian method
- Semismooth Newton method
- Local linear convergence rate
- Metric subregularity