Numerical Methods and Optimization in Statistics and Machine Learning (7 ECTS)

    Course Code: 
    6115
    Semester: 
    8th
    Elective Courses
    Διδάσκων: 

    YANNACOPOULOS ATHANASIOS

    Numerical linear algebra: Matrix decompositions and factorizations (LU, Cholesky, QR, SVD), numerical computation, and applications in statistics and machine learning. Iterative methods: Iterative methods, general theory and philosophy. Examples of iterative methods for solving linear systems or matrix inversion (Jacobi, Gauss-Seidel, over-relaxation) and applications in statistics and machine learning. Examples of iterative methods for solving eigenvalue problems (power-type methods, QR method) and applications in statistics and machine learning.

    Unconstrained optimization: Smooth problems: First-order methods (gradient descent, stochastic gradient descent, momentum methods, Adagrad, ADAM); higher-order methods (Newton methods, quasi-Newton methods). Nonsmooth problems: The subdifferential, proximal methods. Applications to the training of machine learning models, neural networks, and kernelized models.

    Constrained optimization: Projection methods (projected gradient descent and generalizations); duality-based methods and computational methods based on them. Saddle-point methods, descent and ascent methods based on Lagrange multipliers, etc. Applications to the training of machine learning models.

    Large-scale or large-volume data problems: Numerical linear algebra methods for large datasets (Krylov methods – iterated subspace methods). Non-convex optimization. Heuristic and evolutionary algorithms. Scalability and scalable methods. Applications to the training of machine learning models.

    Recommended Reading

    • Ακρίβης Γ., Β. Δουγαλής, Εισαγωγή στην Αριθμητική Ανάλυση, 5η Έκδοση, 2025 Πανεπιστημιακές Εκδόσεις Κρήτης
    • Chapra, S., Canale, R. (2016). Αριθμητικές Μέθοδοι για Μηχανικούς. Εκδόσεις Τζιόλα.
    • Ford, .,  Numerical Linear Algebra with Applications, 2014 Elsevier
    • Kravvaritis D. and A. N. Yannacopoulos Variational methods in nonlinear analysis with applications in optimization and partial differential equations, 2nd edition De Gruyter 2026 (ch. 2,4,5)
    • G. Lan, First order and Stochastic Optimization Methods for Machine Learning Springer, 2020
    • Lange, K. (2010). Numerical Analysis for Statisticians. Springer.
    • Monahan, J. F. (2011). Numerical methods of statistics. Cambridge University Press.
    • Α. Ν. Γιαννακόπουλος, Σημειώσεις μαθήματος
    • Εξειδικευμένη βιβλιογραφία από ερευνητικές εργασίες, στοχευμένες στα διάφορα θέματα των εργασιών (διαθέσιμη στους φοιτητές ανάλογα με το επιλεγμένο θέμα από τον διδάσκοντα).

    (old title: "Numerical Methods")