A NEW MODIFIED HS ALGORITHM WITH STRONG POWELL-WOLFE LINE SEARCH FOR UNCONSTRAINED OPTIMIZATION

Optimization is now considered a branch of computational science. This ethos seeks to answer the question «what is best?» by looking at problems where the quality of any answer can be expressed nume rically. One of the most well-known methods for solving nonlinear, unrestric-ted optimization problems is the conjugate gradient (CG) method. The Hestenes and Stiefel (HS-CG) formula is one of the century’s oldest and most effective formulas. When using an exact line search, the HS method achieves global convergence; however, this is not guaranteed when using an inexact line search (ILS). Furthermore, the HS method does not always satisfy the descent property. The goal of this work is to create a new (modified) formula by reformulating the classic parameter HS-CG and adding a new term to the classic HS-CG formula. It is critical that the proposed method generates sufficient descent property (SDP) search direction with Wolfe-Powell line (sWPLS) search at every iteration, and that global convergence property (GCP) for general non-convex functions can be guaranteed. Using the inexact sWPLS, the modified HS-CG (mHS-CG) method has SDP property regardless of line search type and guarantees GCP. When using an sWPLS, the modified formula has the advantage of keeping the modified scalar non-negative sWPLS. This paper is significant in that it quantifies how much better the new modification of the HS performance is when compared to standard HS methods. As a result, numerical experiments between the mHSCG method using the sWPL search and the standard HS optimization problem show that the CG method with the mHSCG conjugate parameter is more robust and effective than the CG method without the mHSCG parameter


Introduction
The CG method is one of the most popular methods for solving nonlinear unconstrained optimization problems. This method class has a wide range of applications, particularly in the design, construction, and maintenance of any engineering system. Engineers must make numerous technological and managerial decisions at various stages. The ultimate goal of all such decisions is to either minimize the amount of effort required or maximize the desired benefit. Because the effort required or the benefit desired in any practical situation can be expressed as a function of a decision variable, optimization can be defined as the process of determining the conditions under which the maximum or minimum value of a function can be found. The HSCG formula is widely regarded as one of the most effective methods devised during the twentieth century. If the descent condition is not met, the HS parameter may fail to satisfy the CGP of the CG method with the sWPLS. This method class has a wide range of applications, most notably in engineering.

Literature review and problem statement
In 1952, HS proposed a CG algorithm for solving symmetric positive definite systems of equations [1], while in 1964, FR [2] presented a nonlinear conjugate gradient method for solving unconstrained optimization problems based on the linear conjugate gradient method. Many researchers have ex-pressed an interest in developing the HS formula to improve algorithm performance while ensuring that the proposed method meets the SDP and CGP. As a result, [3] developed a new formula to ensure that the conjugation condition is met. To demonstrate that the proposed formula is very efficient with any line search, [4] proposed a modified three-term HS-CG formula to show that the proposed formula is extremely efficient with any line search. Based on Armijo-type line search, [5] developed a new formula for HS [6] proposed another formula, different parameters have different iterative effects [7] suggested a new modification to HS. To improve the functionality of this algorithm, the researchers proposed the development of the HS formula [8], [9] proposed a hybrid of the HS formula and the PRP formula to improve the direction's performance.
Because of their low memory requirements and strong local and global convergence properties, CG methods are widely used. Our goal is to minimize a function of n-variables function by solving the optimization problem min : ,  → is a smooth and non-linear function. The CG method is iterative, the iterative scheme is defined by: x x s j j j The step length μ j >0 yields by some line search, and the direction d j is generated by: where the gradient is defined by g f x j j = ∇ ( ) , the scalar β j ∈ is determined by various CG methods. The stopping criterion for the CG line search is often adopted on some version of the Wolfe equations [10]. The global convergence of the mHS method is proven under the sWPLS: where 0<σ 1 <0.5 and σ 1 <σ 2 <1 [11] proved that the CG method is globally convergent when they generalized, the absolute value in (4) is replaced by: where σ 4 0 ≥ , 0<σ 1 <σ 4 <1, 0<σ 3 <1 and σ σ 3 4 1 + ≤ . The special case, σ 3 = σ 4 = σ 2 corresponding to sWPLS [12]. These conditions guaranteed that: where y j = g j+1 -g j and s j = x j+1 -x j . By using the special case of (5) for the above equation [13], we get: Furthermore, we have: It is critical to use the inexact line search (ILS) to ensure the global convergence of the nonlinear CG method [13].

The aim and objectives of the study
The aim of this study is to propose a method to generate an SD direction with sWPLS at each iteration, which plays an important role to guranatee the GCP for general nonconvex functions.
To achieve this aim, the following objectives are accomplished: -to make sure that the proposed mHS formula is not negative; -to ensure that the direction obtained by mHS is satisfying (7).

Materials and methods
In this paper, all codes are written in FORTRAN 77 double precision and compiled Visual (Fortran 6.6) (default compiler settings). Table 1 lists the names of the test functions that we used. We conduct numerical experiments on some nonlinear unconstrained test functions. These functions are discussed in CUTE [14], and their details are provided in the Appendix, see Andrei [15].
HS proposed a CG algorithm for solving symmetric positive definite equation systems [1], which are defined as follows: We present an alternative to the mHSCG method. The generated directions are always descending, which is one of the method's appealing features. We proposed the following new mHS: .
Using (5), (10) and (11) If the restart criteria (9) are achieved, set d g 7. Set j = j+1 and go to 3. Now we have the following theorem, which shows that the mHS method, when combined with the sWPLS, can guarantee SDP satisfaction.
Consider the following theorem (1) to clarify the direction d j obtained by mHS and the importance of satisfying (7) in the analysis of GCP [17]. We make the following basic assumptions about the objective function for our subsequent analysis.
2) f(x) is continuously differentiable in a specific neighborhood N of Λ, and its gradient is Lipschitz continuous, that is, there exists a constant L>0, s. t.
Proof. Firstly, for j = 0, the proof is trivial, namely d g g d g By using another side of (9), we have g g g , so: Now, put (6), (7) & (15) in (14), we will get: In this section, we want to demonstrate the GCP of mHS under certain assumptions (A). The following lemmas are required, which are commonly used to prove GCP and are provided by Zoutendijk [10].
Theorem (2). Assume Assumption (A) is correct. Assume any iteration method (1) and (2), and μ j obtained by the sWPLS (3), (4). If We are aware of this: g s g s g s g s y s g s y s . Now from (6), (9), (12), and (15): We employ the above-mentioned relationships: Proof. To begin, note that d j ≠ 0, because otherwise (7) would imply g 0 = 0. As a result, v j is well defined. Let us now define: We now have v j+1 = v j = 1, Since α j+ ≥ 1 0, the triangle inequality and (19) provide us with: As a result of the definition of w (j+1) , it follows that: As a result of (20) and (21) When the mHS algorithm generates a smaller step-size, the next search direction will automatically approach d j = -g j . Furthermore, small step-size is not continuously genetized. This is primarily due to the following property: β j mHS . Because the step-size is small enough, mHS tends to zero. Property (*) was introduced for the first time by Gillebert and Nocedal [17].

1. Non-negative formula
It has been demonstrated that the new formula remains non-negative, which is critical in ensuring the method's regression, and it turns out that 0 ≤ ≤ β β j mHS j HS .

2. Global convergence
The proposed method has demonstrated that it achieves the GCP by utilizing the sWPLS, with the help of which the method achieves SDP, and this resulted in the method achieving its desired goal, as demonstrated in theory 3.
The comparison of the proposed mHS (modified Hestenes & Stiefel) and HS (classical Hestenes & Stiefel) algorithms by using (40) well-known test functions is done in Table 1. Our comparisons are based on the number of iterations (Ni); the number of restarts (Nr), the number of test functions and gradient evaluations (Nfg); finally, the total time (in seconds) required to complete the evaluation process. These algorithms implement the sWPLS with σ 1 = 0.01 and σ 2 = 0.1, and the initial step size μ j is computed from: Each function is tested 10 times to gradually increase the number of variables: n = 1000, 2000, …, 10000. The stopping criterion used in this algorithm is g j+ − ≤ 1 6 10 , denoting the Euclidean norm. The unconstrained optimization functions with the given initial points. Table 2 shows that the percentage performance of the mHS method is slightly better than that of the classical HS methods. We see that the mHS method gave (Ni 30.4 %), (Nr 7.7 %), (Nfg 30.6 %) and finally (Time 8.5 %) compared with the classical HS method. This behavior could be explained by making a small change to the HS method so that the generated direction always satisfies the sufficient descent condition, whereas search directions generated by classical HS methods do not guarantee to satisfy the descent property for some problems.  Fig. 1-4 show the method's efficiency in terms of the number of iterations evaluated, the number of restarts, the number of functions and derivatives evaluated, and finally the time required for the two methods, in comparison to the proposed β i mHS and classical β i HS , as demonstrated below.

Discussion of experimental results
The numerical results demonstrated that the proposed mHS method is more effective and faster than the original HS formula. The proposed mHS method was faster than the HS method, as evidenced by the percentages in Table 2. The proposed algorithm is sensitive to changing any of the parameter values (σ 1 and σ 2 ) or the line search (3) and (4), any change that leads to either improving the method, increasing its efficiency, and obtaining the best results, or failing the method and obtaining poor and inefficient results.
Using universal test functions, which are always used to validate the proposed methods, the numerical results demonstrated that the proposed mHS method is more effective and faster than the original HS formula. Its goal is to evaluate the efficacy of the proposed method in the context of other lines of research, as well as the extent to which it influences numerical results. These methods are extremely sensitive to changing the σ 1 = 0.01 and σ 2 = 0.1 values in sWPLS, and any change in these values will result in either better results and completion of the tasks, or worse results and failure to complete the tasks.

Conclusions
1. It was mathematically proven that the proposed formula is non-negative, resulting in the method's efficiency.
2. The algorithm meets the condition of sufficient regression, which is critical for obtaining comprehensive convergence, as demonstrated by theorem (1), and this was demonstrated through the proof of theorem (3). The numerical results represent using cumulative by comparing the CG method with the proposed CG parameter with the CG parameter HS. Fig. 1-4 depict the performance profile for the number of iterations evaluated, the number of restarts, the number of functions and derivatives evaluated, and the CPU time. As a result, according to these figures, the CG method with the proposed parameter is always on top of the curve in the performance profile when compared to the other.