Stochastic optimization methods are fundamental in large-scale machine learning. However, understanding why these methods perform so well remains a major challenge. In this talk, I will present a theory for stochastic gradient descent (SGD) and its adaptive variant in high dimensions when the number of samples and problem dimensions is large. I will show that the dynamics of SGD applied to generalized linear models and multiindex problems with data possessing a general covariance matrix become deterministic in the large sample and dimensional limit. In particular, the limiting dynamics are governed by a set of low-dimensional ordinary differential equations (ODEs). This setup encompasses many optimization problems, including linear regression, logistic regression, and two-layer neural networks. In addition, it unveils the implicit bias inherent in SGD. For each of these problems, the deterministic equivalent of SGD enables us to derive a close approximation of the generalization error (with explicit and vanishing error bounds). Furthermore, we establish explicit conditions on the step size, ensuring the convergence and stability of SGD and a wide class of adaptive stochastic method.

The mean-risk model is a popular modeling paradigm, which quantifies the problem in a lucid form of only two criteria: the mean and the risk. Of particular interest of this paper, we consider the case that the risk is measured by a general lower partial moment (LPM) risk measure. When the underlying distribution of the random parameter is unknown, the distributionally robust approach is employed to handle the mean-LPM model by hedging against a range of possible distributions, which is the so-called ambiguity set. First of all, we discuss the attainability of the distributionally robust mean-LPM model regarding to both the decision variable and the worst probability distribution. After that, in the data-driven process, we investigate how the data size affects the parameters in the ambiguity set as well as the constraint endogenously when the ambiguity set is constructed by the first- and second-order moment information and the Wasserstein ball, respectively. This is not only curious in theory but also crucial in practical implementation. Finally, by defining some suitable optima, we give numerical procedures for computing these suitable optima. Some numerical results are reported by using real data in finance to validate the effectiveness and the efficiency of our method.

This paper considers a multiproduct pricing problem under pure characteristics demand models when the probability distribution of the random parameter in the problem is uncertain. We formulate this problem as a distributionally robust optimization (DRO) problem based on a constructive approach to estimating pure characteristics demand models with pricing by Pang, Su and Lee. In this model, the consumers' purchase decision is to maximize their utility. We show that the DRO problem is well-defined, and the objective function is upper semicontinuous by using an equivalent hierarchical form. We also use the data-driven approach to analyze the DRO problem when the ambiguity set, i.e., a set of probability distributions that contains some exact information of the underlying probability distribution, is given by a general moment-based case. We give convergence results as the data size tends to infinity and analyze the quantitative statistical robustness in view of the possible contamination of driven data. Furthermore, we use the Lagrange duality to reformulate the DRO problem as a mathematical program with complementarity constraints, and give a numerical procedure for finding a global solution of the DRO problem under certain specific settings. Finally, we report numerical results that validate the effectiveness and scalability of our approach for the distributionally robust multiproduct pricing problem.