TTK28 Modeling with neural networks

Teacher: Bjarne Grimstad

NOTE: THE COURSE WILL NOT BE GIVEN THIS FALL, 2024.

Deep neural networks are being widely deployed in the industry to solve complex tasks which conventional methods have struggled to solve. There are several reasons for this breakthrough: more data, more compute, and improved algorithms and software. A perhaps equally important factor to the success is the flexibility of neural networks as a modeling framework. Without explicitly programming the logic of an algorithm, neural networks can learn to solve complex tasks from data. In this paradigm, the job of the engineer has moved from programming logic, to designing meaningful machine learning tasks.

This specialization course gives an introduction to neural networks. We will see how neural networks, being computational graphs, can be used to build very generic hypothesis spaces of nonlinear models. With these models we can solve regression, classification or other types of learning tasks by simply selecting an appropriate cost function. The course ends with a couple of guest lectures on topics from ML research and/or an industrial applications of neural networks.

The goal with the course is to add neural networks to your problem solving toolkit. After completing the course, you should: understand the basic concepts of neural networks; be able to develop and apply a neural network; understand the advantages and limitations of neural networks; be able to digest new ML research and continue developing your skillset by self-study.

• Part I – Machine learning basics (8 hours)
  • 1.1 Course introduction and motivation – why learn from data?
  • 1.2 Learning algorithms – how to learn from data?
  • 1.3 Generalization and test sets
  • 1.4 Capacity and overfitting
  • 1.5 Hyper-parameters and validation sets
  • 1.6 Non-parametric models and representation learning
  • 1.7 Maximum likelihood estimation – Linear and logistic regression
• Part II – Deep learning (8 hours)
  • 2.1 Deep feed-forward neural networks: Representation
  • 2.2 Optimization of neural networks: Learning
  • 2.3 Regularization of neural networks
  • 2.4 Implementation and practice
  • 2.5 Popular architectures
• Part III – Programming and case study (5 hours)
  • 3.1 Programming in PyTorch (Python)
  • 3.2 Case study: industrial application of neural networks
  • 3.3 Orientation about the project and exam

The first lecture is scheduled to Monday 28th of August, 10:15-12:00 AM.

Most, if not all, of the lectures will be held digitally on Blackboard. Lectures will be live streamed to R3, Realfagbygget (https://i.ntnu.no/wiki/-/wiki/Norsk/R3+-+Undervisningsrom)

The lectures will be given in Norwegian, with slides written in English.

Students will be given a voluntary project assignment or topic to study.

The exam will consist of two parts: 1) an oral presentation of the assignment, and 2) questions from the syllabus.

Exam dates: December 14-15.

• Course in matrix algebra and calculus (e.g. TMA4115 - Calculus 3, or a similar course)
• Introductory course in statistics (e.g. TMA4240 – Statistics, or a similar course)
• Introductory course in mathematical optimization (e.g. TTK4135 - Optimization and Control, or a similar course)

• Lecture notes
• Deep learning by Goodfellow et al. (freely available at https://www.deeplearningbook.org/)
• Selection of papers (TBD)