CSE 509 Computer System Security

Fall 2021

Course Description

In the class, we will discuss the principles and practice of computer system security, with particular emphasis on:

• software vulnerabilities and advances in exploit techniques
• vulnerability analysis and mitigation techniques
• binary analysis, reverse engineering and instrumentation
• malware trends and defenses against untrusted code
• advanced attack campaign detection and forensics

One of the main objectives of this course is adversarial thinking: students should be able to quickly zoom in on the weakest link in any security technology, or system design. Students should be able to imagine how an attacker might break their system, and build in protection and mitigation measures to ward off such attacks.

This is a hands-on course, where students learn by carrying out several short assignments and a final project. Some assignments will be aimed at in-depth understanding of software vulnerabilities by developing exploits. Others will be aimed at tools and techniques used for mitigating security threats. All of them are designed to prepare you for a final project that will be completed by groups of 2 to 4. All of these assignments and the projects provide a taste of research in software and systems security.

Some assignments are best carried out by teams of two. Please find a suitable project partner right at the beginning of the course in order to avoid problems later. You can do these assignments alone, but that obviously will mean more effort.

Course Topics

• Processor and virtual machine security
• Operating system security
• Cryptography and authentication
• Access control
• Software vulnerabilities and exploit techniques
• Vulnerability analysis
• Vulnerability mitigation techniques
• Intrusion detection
• Malware
• Defenses for untrusted code and malware
• Reverse engineering

Lectures

Note that lecture recordings are from a previous offering of this course. While most of the material has not changed, the depth of coverage in a few topics will differ to some extent. So, use these recordingly as supplementary material, but not as a substitute for in-person lectures.

Topic #	Topics and Lecture Recordings	Slides	Notes
1	Introduction	PDF
2a	Memory Corruption Vulnerabilities I Runtime memory organization: 3 mins C/C++ Object/Struct layout Stack-smashing attacks and defenses: 24 mins, 24 mins Attacking defenses: Brute force, information leakage & partial overwrite: 7 mins, 3 mins, 10 mins Code Injection and Code Reuse: 24 mins; ROP 4 mins, 8 mins Reading: Smashing the stack for fun and profit	PDF PDF	C Runtime Environment C/C++ Object Layout Memory Errors and Defenses
2b	Exploit Assignment Discussion Assignment 1 Overview: 24 mins, 11 mins Help session with gdb and code: 24 mins, 24 mins, 12 mins Follow-up discussion with Q&A session: 11 mins Behind the working of gdb: 21 mins
2c	Memory Corruption Vulnerabilities II Heap overflows: 11 mins, 22 mins Format-string attacks: 8 mins Integer overflows, more on memory exploits: 8 mins 8 mins, 15 mins Categorization of memory error defenses: 24 mins, 2 mins Randomization-based defenses: ASR, RAR, Code Randomization and DSR: 11 mins, 14 mins Memory errors: definition, detection and prevention: 4 mins, 24 mins, 24 mins	PDF
3	More Software Vulnerabilities Injection Attacks: 5 mins, 24 mins Taint-tracking: 24 mins Race Conditions: 24 mins, 24 mins, 2 mins CVE, CWE, and Principles of Secure system design: 19 mins, 24 mins	PDF
4	Malware Types and goals of malware: 24 mins, 18 mins Stealth, Obfuscation and Challenges of malware defense: 24 mins	PDF	PDF
5	Defenses for Untrusted Code and Malware Reference Monitors: 24 mins System call interception and sandboxing: 7 mins, 24 mins, 24 mins Inline Reference Monitors: 24 mins Software Fault Isolation: 7 mins, 24 mins Control Flow Integrity: 24 mins Native Client: 24 mins, 5 mins	PDF	PDF
6	Binary analysis and instrumentation Disassembly and Binary analysis: 6 mins, 6 mins, 24 mins, 22 mins, 2 mins Static binary instrumentation: 24 mins   16 mins   14 mins Dynamic binary translation: 24 mins, 20 mins	PDF PDF	PDF
6a	Midterm review discussion Questions related to Programming Assignment 1: 24 mins Questions related to Sample Problem Sets: 24 mins, 24 mins, 12 mins
7	Cryptography Basics* Introduction: 14 mins Symmetric (Secret Key) Cryptography: Overview 5 mins, Ciphers and algorithms 8 mins Asymmetric (Public Key) Cryptography: 16 mins Public vs Secret key encryption: 5 mins Cryptographic random numbers: 10 mins Digital signatures and message digests: 8 mins Digital certificates: 5 mins	PDF	PDF
8	Identification and Authentication   1h 11m   1h 6m Reading: Password Security: A Case History Reading: Lamport's One-Time Password Scheme	PDF	PDF
9	OS Security and Access Control File permissions and ACLs: 34 mins OS Capabilities: 10 mins Mandatory Access Control: 19 mins Domain and Type Enforcement: 11 mins Linux Capabilities: 9 mins Policies for untrusted code: 18 mins Policy Management: 9 mins Reading: Revisiting "Setuid Demystified" Reading: Confining Root Programs with Domain and Type Enforcement	PDF	PDF
10	Virtual Machines   1hr 6 mins	PDF
11	Web security Web overview: HTTP, Cookies, Javascript and DOM: 48 mins 14 mins Authentication: 7 mins Same Origin Policy: 22 mins CSRF and Clickjacking: 28 mins XSS and related attacks: 41 mins Network based attacks*: 13 mins Client-side attacks and summary*: 12 mins	PDF
12	Intrusion Detection   0:35   0:30	PDF
13	Vulnerability analysis: Fuzzing and Symbolic Execution   1:12	PDF
14	Side-channel attacks   0:53 Reading: Metdown and Spectre attacks	PDF
15	Course Summary	PDF

* Topics marked with an asterisk were recorded outside of normal class hours.

Class Place and Time:

• Lecture time/location: Tue/Thu at 11:30am to 12:50pm New CS 120
• First lecture: August 24, Tuesday
• Fall Break: October 11--12 (Mon, Tue)
• Thanksgiving Break: Nov 24 to Nov 28
• Last lecture: December 2, Thursday
• Final Exam: December 15, Wednesday, 11:15am to 1:45pm, New CS 120

Tentative Deadlines:

Dates for assignments and mid-term exams are subject to change.

Late submission policy: You can take a total of two late days across the three programming assignments/labs. Just inform the TA whenever you want to take a day off. A day is defined as 24 hours. You can use both late days for one assignment, or use one late day each for two of the assignments.

Instructor:

R. Sekar
Office: Rm 364 New Computer Science
Office Hours: Wed 11:30am to 12:30 on Zoom

If you experience difficulties in joining the zoom call, please send me email (my last name at cs.stonybrook.edu).

TA:

Rory Bennett
Office Hours: Mon, Fri 11am to noon on zoom
Email: rmbennett at cs dot stonybrook dot edu

Texts:

There is no textbook for this course. We will rely primarily on class notes.

Grading

Special Needs

If you have special needs, concerns or a disability, please contact the staff at Student Accessibility Support Center (SASC). SASC staff will review your concerns and determine, with you, what accommodations are necessary and appropriate. All information and documentation will remain confidential.