Course description

A systematic introduction to machine learning, covering theoretical as well as practical aspects of the use of statistical methods. Topics include linear models for classification and regression, support vector machines, regularization and model selection, and introduction to structured prediction and deep learning. Application examples are taken from areas like information retrieval, natural language processing, computer vision and others.

Prerequisites: Probability, Linear Algebra, Undergraduate Algorithms.

A detailed topic list (roughly in order of presentation)

• intro to ML: scope, motivation,  and goals of the class
• refresher on probability and algebra (TA)
• statistical framework for learning; loss/risk; least squares regression 
• noise models; error decomposition; bias/variance and overfitting 
• model complexity, sparsity (L1/L2) in regression; 
• classification; Fisher’s LDA, logistic regression and softmax 
• ensemble methods, boosting, stepwise methods
• generative models, Naive Bayes, multivariate Gaussians
• EM for mixture models and in general
• SVM and kernels 
• nonparametric methods; nearest neighbors, density estimation
• multilayer neural networks and deep learning 
• information theory and learning; information criteria, MDL and their connections to regularization
• experiment design and evaluation in ML
• advanced topics (time permitting)
• wrap-up and review of the class

Expected outcomes:

• Understand the notion of fitting a model to data and concepts such as model complexity, overfitting and generalization, and bias-variance tradeoff in estimation.
• Learn and be able to apply some of the fundamental learning methods, such as logistic regression, support vector machines, boosting, decision trees, neural networks.
• Learn the basics of optimization techniques such as gradient descent and the general EM algorithm.
• Familiarity with multivariate Gaussians and mixtures of Gaussians.
• Understand fundamental concepts in information theory (entropy, KL-divergence) and their relationship to machine learning.