Consideration of estimation of logistic regression as an optimization problem

The logistic regression is one of the common methods of data classification in various spheres. The main goal of logistic regression is a separation of multiple input values with a linear boundary (a dividing line, plane or hyperplane) on two classes corresponding to two given region. Logistic regression predicts the probability of some events that are in the range from 0 to 1. Should indicate that the L classes can be more than two, in this case we have a multinomial regression. It can be built using L - 1 independent logistic regressions. The research of an estimation algorithm logistic regression on model data implemented in statistical packages, showed instability of the estimates. The goal of this article is research of the features of the procedure of estimation of logistic regression as an extreme problem, as well as the development of a stable algorithm of computing the coefficients of the separating equation. n i _T \ Nowadays the logistic regression is usually built as an optimization problem Q(b) = V ln (1 + e^-y,b x ) -» min that implements the r=1 V ' beR" criterion of maximum likelihood. In this article the features of estimation of logistic regression are considered as an optimization problem. There is shown that this problem has significant differences compared to the search of a local extremum. There are investigated causes of the instability of estimates of the logistic regression. It is shown that using b⁽⁰⁾ as initial approximation for the solve of the classification task as a linear regression often yields unsatisfactory results. The second reason for the instability of estimates of maximum likelihood is that the iterative descent algorithms are designed to find a local minimum. The target function with the correct classification of all precedents has zero as the lower bound at infinity. Moreover the quality of the building of the dividing line in the formal descent wouldn't be associated with the value of the objective function Q(b). So adjust of the strategy of the descent is needed. The third reason is that the some values, for example, y. b^T x. can be very large when the value of objective function Q(b) is small. It leads to fatal errors for computing and even can stop the operation of the algorithm due to memory overflow. The stable algorithm for calculating the coefficients of the separating equation of the logistic regression is described. As initial approximation a hyperplane is used. The hyperplane is orthogonal to the segment connecting the centers of gravity of classes, and is passed through the middle of this segment. The iterative algorithm involves a sequence of problems of minimizing zero-order based on a random search. At each k -th step the length of the vector of coefficients b^(k) is fixed. It is gradually increased until the objective function achieves the required value. The results of statistical modeling of comparative analysis of estimation of logistic regression with the existing algorithm are given.

Keywords

Authors

References