In many mountainous areas, landslides and slope instabilities frequently occur after heavy rainfall and earthquake, and result in enormous casualties and huge economic losses. In order to mitigate the landslides hazard efficiently, a method is required for a better understanding of stability analysis. Fortunately, upper bound theorem of limit analysis provides a practical and effective upper bound approach to evaluate the stability of slopes. And in this approach, the search for the minimum factor of safety can be formulated as a nonlinear constrained optimization. In general, the SQP-type algorithms are used to solve this optimization problem. However, it is quite time consuming and difficult to search the optimum from an arbitrary starting point based on the SQP-type algorithms. Fortunately, a QP-free algorithm based on penalty function and active-set strategy can be globally convergent toward the KKT points with arbitrary starting point, and the rate of convergence is local superlinear or even quadratic. Two classical problems of slope stability are solved by this QP-free algorithm. The results show that the QP-free algorithm would be the better choice than SQP-type algorithms for solving the nonlinear constrained optimization problem which is derived from the upper bound limit analysis of slope stability.