A Convexity-dependent Two-Phase Training Algorithm for Deep Neural Networks

Journal
Proceedings of the 17th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management
Type
conference paper
Date Issued
2025-10-27
Author(s)
;
;
;
;
DOI
Abstract
The key task of machine learning is to minimize the loss function that measures the model fit to the training data. The numerical methods to do this efficiently depend on the properties of the loss function. The most decisive among these properties is the convexity or non-convexity of the loss function. The fact that the loss function can have, and frequently has, non-convex regions has led to a widespread commitment to non-convex methods such as Adam. However, a local minimum implies that, in some environment around it, the function is convex. In this environment, second-order minimizing methods such as the Conjugate Gradient (CG) give a guaranteed superlinear convergence. We propose a novel framework grounded in the hypothesis that loss functions in real-world tasks swap from initial non-convexity to convexity towards the optimum - a property we leverage to design an innovative two-phase optimization algorithm. The presented algorithm detects the swap point by observing the gradien t norm dependence on the loss. In these regions, non-convex (Adam) and convex (CG) algorithms are used, respectively. Computing experiments confirm the hypothesis that this simple convexity structure is frequent enough to be practically exploited to substantially improve convergence and accuracy.
Language
English
HSG Classification
contribution to scientific community
Refereed
Yes
Publisher
SciTePress
Volume
1
Start page
78
End page
86
Pages
8
Event Title
International Conference on Knowledge Discovery and Information Retrieval
Event Location
Marbella, Spain
Event Date
23-24 October 2025
Official URL
URL
Contact Email Address
bernhard.bermeitinger@unisg.ch
File(s)
Thumbnail Image

open.access

Name

2025-10-23 - KDIR2025 - Conjugate Gradient Two Phase Training.pdf

Description
Slides for the paper presentation at the conference. Held by Götz-Henrik Wiegand
Size

2.87 MB

Format

Adobe PDF

Checksum (MD5)

8f1e7e8f4b80541512d0ac09532fd435

Thumbnail Image

restricted

Name

136961.pdf

Type

Main Article

Description
Published document
Size

587.23 KB

Format

Adobe PDF

Checksum (MD5)

79f1daab58e1468c42f5af0278b1069a