# ME 470: Syllabus

ME 470: Uncertainty Quantification: Techniques and strategies for tackling unknowns in science and engineering research

## 1 Introduction

Uncertainty is a challenge. Representing a lack of knowledge, uncertainty stymies our attempts to draw scientific conclusions, and to confidently design engineering solutions. Failing to account for uncertainty can lead to false discoveries, while over-reacting to uncertainty can lead to overbuilt engineering designs. Overcoming these issues requires identifying, quantifying, and managing uncertainties through a combination of technical skills and an appropriate mindset. This class will introduce modern techniques for quantifying and propagating uncertainty. Issues of dimensionality will be discussed. Emphasis will be on applying techniques in genuine applications, through assignments, case studies, and student-defined projects.

Prerequisite: Basic probability and statistics at the level of CME 106 or equivalent.

## 2 Learning Goals

In this upper-level graduate course, students will learn how to formulate, analyze, and solve problems subject to uncertainty. Content will include technical discussion of state-of-the-art methods for propagating uncertainty, and case studies to illustrate their usage. By the end of the course, students will be able to address the following:

### 2.1 1. How uncertainty affects conclusions in science and engineering

a. In the context of open-ended assignments, students will:

• Identify sources of uncertainty
• Determine what – if any – conclusions can be drawn, given uncertainty
• Determine what additional data would be necessary to improve the analysis

Topics:

1. Risk, model validation, decision-making under uncertainty
2. Probability
3. Aleatory vs epistemic uncertainty

### 2.2 2. Why and how to employ practical skepticism towards models

a. In the context of analyzing scientific and engineering models, students will:

• Identify potential sources of model discrepancy
• Given data, determine the level of discrepancy with the model
• Use global sensitivity analysis to rank input uncertainties
• Survey additional techniques, including local sensitivity approaches

Topics:

1. Model-form uncertainty, model validation
2. Sensitivity analysis
3. Bayesian inference/calibration
4. Nonlinear regression
5. Model checking

### 2.3 3. How to handle issues of dimensionality

a. In the context of general uncertainty propagation, students will:

• Identify cases where dimensionality is an issue
• Employ dimension reduction techniques to make problems tractable

Topics:

1. Exponential cost of dimensionality
2. Sobol' indices
3. Active subspaces

### 2.4 4. How to quantify and understand uncertainty

a. In the context of data, students will:

• Explore and understand data using point and interval estimates
• Practice data skepticism and criticism
• Model uncertainty

Topics:

1. Summaries
2. Variability
3. Regression for visualization
4. Random variable modeling

### 2.5 5. How to propagate uncertainty

a. In the context of forward propagation, students will:

• Use the delta, Monte Carlo, and quadrature methods
• Select a reasonable technique for the problem at hand
• Gain familiarity with other methods, e.g. intrusive PCE, GPR
• Diagnose problems in propagation

Topics:

1. Delta method
2. Monte Carlo
3. Polynomial chaos
• Intrusive
• Non-intrusive

b. In the context of inverse propagation, students will:

• Use regression and Bayesian inference
• Build models for inference
• Diagnose problems in inference, and criticize inferential models

Topics:

1. Regression
• Linear
• Nonlinear
2. Bayesian inference/calibration
3. Model checking

## 3 Delivery Strategy

### 3.1 Format

Content will be delivered through a number of different formats, described below. All in-class delivery strategies will be adjusted to use modern, research-backed principles and strategies to promote learning outcomes – active learning.

#### 3.1.1 Lecture

Much of the class format will follow a fairly traditional lecture format. However, I will take steps to introduce small adjustments to make lectures more engaging (not all active learning involves large changes!). Lectures will generally start with some form of challenge statement – an open-ended question meant to get you thinking about high-level concepts. I am very much open to questions during lecture, and plan to turn discussion over to small groups or the full class as is appropriate.

#### 3.1.2 Interactive Case Studies

A handful (five planned) of in-class sessions will be structured around case studies. These will be interactive, instructor-guided, student-driven activities mean to reinforce concepts learned in previous lectures. These will be varied in format, but tend to use active learning strategies, and will emphasize in-class participation.

#### 3.1.3 Assignments

There will be a small number (four planned) of take-home problem sets, which will follow a fairly standard format. They will consist of a small number of fairly involved problems; some will be intended to solidify concepts covered in class, such as implenentation or derivation problems. Others will emphasize executive function – I will pose an open-ended problem, and you will need to select and justify an approach. In all problems, the emphasis will be on understanding and justifying the analysis – this will be reflected in the grading scheme as well.

#### 3.1.4 Project

There will be a student-initiated final project; you will pose a problem you would like to solve, describe your approach, and summarize your findings. The topic can be drawn from any area (ideally your research), but must involve some concept discussed in class. Projects will be carried out individually; if you are working with other students on the same research project, you must either choose different topics, or split the project into different components. The project Report itself is intended to involve an amount of work comparable to a single homework assignment; however, there are additional components around the project (Peer Review and Presentation) that will constitute additional work. Accordingly, the deliverables for the project are spread over the latter half of the course.

1. Proposal

By the midpoint of the class, you will submit a one-page Proposal for your project. In this assignment, you must

1. Describe the problem you aim to solve
2. State what approach you plan to use
3. Hypothesize what will be the result
4. State your personal learning goals for this course, and articulate how this project will serve those goals

The purpose of this assignment is to ensure you are on track to deliver a compelling project at the end of the course.

2. Draft and Peer Review

In week 8, you will submit a Draft report for your project. This will be used for a Peer Review exercise, in which you will review two other students' draft work. This will primarily be an exercise in providing useful commentary on other researchers work – a critical component of the academic enterprise.

For the Draft, you must:

4. Note what you plan to do for the final submission

For the Peer Review, you must:

1. Summarize what the work aims to do
2. State both strengths and weaknesses of the work
• From a writing perspective
• From a technical / methodological perspective
3. Provide recommendations on how to improve the work

The Draft will be graded for completeness. Peer Review will be graded for quality, based on a provided rubric. Note that, to simulate a more 'traditional' peer review process, we'll leave the learning goals out of this phase. This is a concession to normative research values in academia, but is not a reflection of the value of learning goals.

3. Final Presentation

The final days of the class will be reserved for student Final Presentations. These will (aspirationally) be 15-minute talks, though depending on enrollment we may need to adjust this amount of time. The purpose of presentations is to

1. practice communication skills, 2. introduce the class to the variety of

problems being tackled in UQ, and 3. reflect on how the project (and class) addressed your learning goals in taking the class.

4. Report

The final Report will be due during finals period.

For the Report, you must:

3. Argue how your results show that you solved the problem, OR note what would be necessary for more conclusive results
4. Reflect on how this class did (or did not) match your stated learning goals

### 3.2 Assignments

The majority of assignments are problem sets, with a small number of extensive problems intended to reinforce concepts learned in class. Each problem set will include at least one `open-ended' question that will stretch your executive reasoning, and require you to make and justify analysis choices. The project is essentially a large, student-defined assignment. It includes additional components above a normal problem set, described above.

Assignment Topic
HW 1 Probability basics
HW 2 Data analysis
HW 3 Sensitivity analysis
HW 4 Forward uncertainty propagation

### 3.3 Logistics and Procedures

Personnel and Office Hours:

 Name Email Office Hours Zachary del Rosario zdr@stanford.edu 500-500H TBD Gianluca Iaccarino jops@stanford.edu 500-500I TBD (TA TBD) TBD TBD TBD

Assignment late policy:

Assignment due-dates are listed below in the 'Content Sequence' subsection. All assignments are due at the beginning of class on the specified date. Assignments are considered late if not submitted at the specified time. Late assignments may be submitted for partial credit, though they will not be graded in detail, and will receive a maximum partial credit depending on the number of late days:

 Days Late Maximum Credit 1 75% 2 50% 3 25% 4+ 0%

Of course, in extenuating circumstances, exceptions to this policy will be made. Examples of reasonable exemptions include issues of health, family emergencies, and other unforeseeable circumstances. If you have planned events that may impede an assignment deadline (e.g. conference travel), please come discuss your situation with me, and we'll try to work something out.

Re-grades must be requested within one week of the assignment's return date; no exceptions will be made. If you think a grading decision is genuinely in error, please feel free to contact the teaching team (in a timely manner).

Assignment re-submission:

Generally, I would prefer that you take assignment feedback and incorporate it into future assignments. In extenuating circumstances, I may consider a re-submission of an assignment. In this case, the maximum grade that may be earned is the previous grade plus half the deficit; for example, if one scored a 50% on an assignment, the maximum re-submission grade will be 75%. Re-submissions will only be accepted until a week after the relevant assignment has been returned.

### 3.5 Content Sequence

Class Day Topics Assignment Due
1 Apr 2 Risk, validation, decision-making
2 Apr 4 Guest Lecture: Gianluca
3 Apr 9 Case Study 1: Warming up
4 Apr 11 Estimation and modeling HW 1. Prob modeling
5 Apr 16 Linear regression, intro
6 Apr 18 Linear regression as summary
7 Apr 23 Case Study 2: Data analysis
8 Apr 25 Dimensionality, sensitivity analysis HW 2. Estimation and modeling
9 Apr 30 Sobol' indices (ZDR travel) Proposal
10 May 2 Subspaces (PCA and AS)
11 May 7 Case Study 3: Model assessment
12 May 9 Monte Carlo: Basics HW 3: SA
13 May 14 Monte Carlo: QMC
14 May 16 Case Study 4: DOE
15 May 21 Quadrature HW 4: Fwd propagation
16 May 23 PCE
17 May 28 Bayesian inference Draft
18 May 30 MCMC
19 Jun 4 Case Study 5: Model-form errors Peer Review
Fin   Presentations Report

## 4 Resources

I have started a textbook to support teaching this content; however, it will not be complete in time to support this offering of the course. I hope to have a few chapters finished in time – particularly those that do not have a published equivalent.

The following book (Smith 2014) should be considered the primary resource for technical details, particularly when it comes to understanding and implementing algorithms. Copies are available in the Stanford library.

Ralph Smith, Uncertainty quantification: Theory, implementation, and applications (2014) SIAM

As Stanford students, we all have access to digital copies of ["A Certain Uncertainty"](https://www.cambridge.org/core/books/certain-uncertainty/65FB78B0DFC097572F2F9746A3A51C8F), which is a (rather detailed) treatment of relevant material. Consider this to be optional reading.

I will also point to technical papers when referencing material that lies outside the texbook.

## 5 Evaluation

### 5.1 Feedback

Both formative and summative feedback will be deployed in this class, though the emphasis will be on feedback to help students develop (formative), and less on assigning grades (summative).

Formative feedback will be provided through in-class case studies, as I will be able to gauge student understanding in real-time. Additional feedback will be provided through peer review, as students will both get commentary on their work, and be afforded to see how other students conceptualize the material. Finally, all assignments will include some amount of open-ended problem solving that will require executive thought – students are welcome and encouraged to attend office hours to discuss these not-straightforward problems.

Summative feedback will be deployed in grading the assignments and final project. More is stated below in the Grading Philosophy.

This is an upper-level graduate course. My assumption is that students taking this class are primarily engaged in research, and are now taking courses to support that activity. Thus, my expectation is that this class is of secondary importance in your current workload, and my grading philosophy is formulated with this assumption in mind.

There will be four homeworks and a final project, in order to minimize the external workload for the class. I would like to grade the in-class assignments, but I realize that upper-level graduates will need to miss classes for conferences and the like. The class has been designed around reinforcing concepts through these in-class activities, so please do not take my lack of point assignment as an indication of the relative importance of these activities.

The four homeworks will compose half of your grade, with the final project weighted equally. The final project is further broken down into the final report (delivered both through presentation and in writing), and the peer review stage. The relative weight of Project components is intentional.

Challenge Statements 6%
Homework 44%
- HW 1 (11%)
- HW 2 (11%)
- HW 3 (11%)
- HW 4 (11%)
Project 50%
- Draft (10%)
- Peer review (10%)
- Presentation (15%)
- Report (15%)

It is common in upper-level graduate courses to be lenient in grading, and I have every intent to do the same in this class. You are researchers at this stage – we should be beyond ``playing the game of school'' by now.

## 6 Requirements

### 6.1 Pre-requisites

Since this is an upper-level graduate course, I will assume a fair amount of background knowledge on your part. However, I realize some topics are often not well-covered in a `traditional' engineering curriculum, and you may feel uncomfortable with some of the necessary topics. I have sequenced the content with this fact in mind, and make specific my expectations for background knowledge below.

#### 6.1.1 Engineering Mathematics

All students in this class should be fairly mature in their mathematical ability. By this I do not mean proof writing, but strong proficiency with calculus and linear algebra is required. Many of our motivating examples will come from dynamical systems, so some familiarity with ODE's will be helpful, but not necessary.

#### 6.1.2 Probability

Most engineering curricula include some amount of probability theory, but many of the engineers I know are not as comfortable manipulating random variables as they are PDE's. To feel comfortable in this class, you should know what a continuous random variable is, have seen PDF's and CDF's before, have computed some expectations, and have done some Gaussian manipulations at some point in your life. We will review probability theory, so don't worry if you need to dust off these skills a bit. My hope is to reactivate this knowledge, and add a little bit more to your understanding of random variables.

#### 6.1.3 Statistics

Many of the engineers I know never took a statistics course; or, if they did, feel totally uncomfortable with even the basic concepts. We will have to play this topic by ear, you and I. I plan to teach the bare minimum content we'll need for understanding critical UQ concepts, though we may need to shuffle topics and borrow time, depending on how the class is feeling.

### 6.2 Post-requisites

This course is meant to start a shift in your mindset and toolkit. Like any kind of learning, grappling with uncertainty is a lifelong journey. I recommend the following 'post-requisites' to build upon the ideas introduced in this course. Note that these are current as of the 2018-19 academic year:

STATS 305A: Applied Statistics I

• All about (multivariate) regression, with a focus on practical concerns. I found this course very helpful for building intuition in statistics, but also found it rather challenging with only an engineer's background in statistics.

STATS 362: Topic: Monte Carlo

• Art Owen infrequently teaches this class on the Monte Carlo method. There are a lot of practical considerations and subtleties – I would take it if you find the chance!

CS 228: Probabilistic Graphical Models

• Specialty class on using probabilistic graphical models for (machine) learning. These form a very flexible, powerful framework for Bayesian inference, and are an exciting area of current research.

CS 238: Decision Making under Uncertainty

• I hear wonderful things about Mykel Kochenderfer's Decision making under uncertainty course. He has great skill in explaining complex topics in a lucid way.

ENGR 150: Data Challenge Lab

• Very unique course; by application only. Intensive course on data science – teaches visualization techniques and exploratory data analysis.

## 7 Accessibility and Affordability

The Stanford Diversity and Access Office provides accommodations related to disabilities.

The Stanford Financial Aid Office is your first resource for financial assistance. However, the Stanford Diversity and First-Gen Office also provides financial assistance related to:

• Travel costs associated with a death or illness
• Assistance with laptop repair
• Medical emergencies not covered by the Financial Aid Office
• Attending student conferences
• Expenses related to family attendance of major campus events
• Expenses related to graduate/professional exams

If you need further assistance or information, or just want to talk about your situation, please feel free to contact me.

## 8 Honor Code

The Stanford Honor Code was written by students in 1921. While there are no planned exams in this course, the Honor Code still applies to assignments. Please do not plagiarize material on your assignments. The full Honor Code, including violation policies, are listed on the aforementioned website.

## 9 Elastic Clause

The instructor shall have the responsibility to make all laws which shall be necessary and proper for carrying to realization the foregoing learning goals, and all other powers vested by this syllabus in the course of ME 470, or in any office hour or assignment thereof.