Foundations of Deep Learning for the Social Sciences

Overview

Deep learning has revolutionized how complex processes are modeled in fields including computer vision, natural language processing, computational biology, weather forecasting, and game playing. Despite these impressive breakthroughs, deep learning is rarely used to model complex behavioral processes in the social sciences.

This two-day workshop will provide a broad, practical introduction to deep learning concepts and methods, with a particular focus on the benefits and drawbacks of applying these methods to analyze behavioral data. During the first day, you will learn how to build, fit, and evaluate deep learning models for predicting behavioral outcomes. We will start the day by introducing fundamental deep learning concepts in the familiar context of logistic regression. We will finish the day with a hands-on overview of foundational deep learning models for analyzing both cross-sectional and longitudinal data.

During the second day, you will learn about how traditional latent variable models used in the social sciences can be enhanced using deep learning. We will begin by investigating how deep learning software and optimization methods provide a flexible framework for estimating structural equation models. We will conclude by exploring how to estimate highly flexible extensions of traditional structural equation and item response theory models in a computationally efficient manner using deep learning-based approximate inference methods.

What You Will Learn

The definition of a deep learing model
How to fit deep learning models using automatic differentiation and stochastic gradient-based optimization
How to select deep learning model hyperparameters including hidden layer sizes and learning rates
How to evaluate and compare fitted deep learning models
How to build deep learning models for predicting behavioral outcomes using both cross-sectional and longitudinal data
How to fit structural equation models using deep learning software
How to fit item response theory and other nonlinear latent variable models using deep learning-based approximate inference

Instructor

Christopher J. Urban, M.A.
Ph.D. Candidate in Quantitative Psychology, Univeristy of North Carolina at Chapel Hill
Homepage: https://cjurban.github.io/
E-mail:

Prerequisites

Necessary prerequisites:

Proficiency with generalized linear models, particularly linear regression and logistic regression
Proficiency in statistical programming (i.e., manipulating and analyzing data using a programming language like R or Python)
Familiarity with traditional latent variable modeling frameworks, particularly structural equation modeling and/or item response theory

Helpful (but not strictly necessary) prerequisites:

Familiarity with linear algrebra
Familiarity with multivariable calculus
Familiarity with basic machine learning concepts (e.g., hyperparameters, cross-validation)

Computing Resources

We will primarily use Python 3 and some R. If you don't have Python 3, you can install it a number of ways. Options include

the official installer on python.org and
a package manager like Anaconda.

If you don't have R, you can download it from cran.r-project.org. I also recommend installing RStudio, which is a helpful interface for writing a debugging R code.

We will use two deep learning frameworks: PyTorch and Tensorflow (via the Keras interface). PyTorch will be used entirely in Python and can be installed from pytorch.org. Tensorflow will be used in both Python and R. It can be installed

via Python at tensorflow.org/install and
via R at tensorflow.rstudio.com/install.

On the second day, we will use the Python packages tensorsem and DeepIRTools. Once you have Python 3, these packages can be installed via pip install https://github.com/vankesteren/tensorsem/archive/master.zip and pip install deepirtools, respectively.

Tentative Schedule

#	Date	Topic	Learning Outcomes	Materials	Supplemental Resources
1	10/6	Introduction and Foundational Deep Learning Concepts	You will learn: Some achievements of modern deep learning The definition of a deep learning model and some connections to traditional statistical models The basics of how deep learning models are fitted including the backpropagation algorithm, automatic differentiation, and stochastic gradient-based optimization	Lecture Slides Python Coding Example Web Page Notebook R Coding Example Web Page R Markdown	Deep Learning Overviews LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Urban, C. J., & Gates, K. M. (2021). Deep learning: A primer for psychologists. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Chapter 1: Introduction. Backpropagation Olah, C. (2015). Calculus on computational graphs: Backpropagation. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Chapter 6: Deep feedforward networks. Stochastic Gradient Methods Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Chapter 8: Optimization for training deep models.
2	10/6	Foundational Deep Learning Models	You will learn: The definition of the multilayer perceptron (MLP) and applications to cross-sectional data The defintion of the recurrent neural network (RNN) and applications to longitudinal data Why RNNs struggle with long-term dependencies and solutions based on gated architectures The definition of the convolutional neural network (CNN) and applications to image data Universal approximation properties of each model type The basics of tuning neural network hyperparameters How to evaluate and compare fitted deep learning models	See above	Recurrent Neural Networks Lipton, Z. C., Berkowitz, J., & Elkan, C. (2015). A critical review of recurrent neural networks for sequence learning. Olah, C. (2015). Understanding LSTM networks. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Chapter 10: Recurrent and recursive nets. Convolutional Neural Networks Olah, C. (2014). Conv nets: A modular perspective. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Chapter 9: Convolutional networks. Attention Olah, C., & Carter, S. (2016). Attention and augmented recurrent neural networks. Tuning Hyperparameters Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Chapter 11: Practical methodology. Smith, L. N. (2018). A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay. Agnihotri, A., & Batra, N. (2020). Exploring Bayesian optimization.
3	10/7	Connections to Psychometrics	You will learn: How to fit structural equation models using backpropagation and stochastic gradient-based optimization The defintion of the autoencoder and applications The basics of non-amortized and amortized variational inference How to use variational methods to fit complex latent factor models	Lecture Slides Python Coding Example Web Page Notebook Holzinger-Swineford Data Other Coding Examples Modeling Big-Five Personality Factors Data Generating Fake Handwriting	Deep Learning and Structural Equation Modeling van Kesteren, E.-J., & Oberski, D. L. (2022). Flexible extensions to structural equation models using computation graphs. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Chapter 13: Linear factor models. Autoencoders Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. Chapter 14: Autoencoders. Variational Inference Blei, D. M., Kucukelbir, A., & McAuliffe, J. D. (2017). Variational inference: A review for statisticians. Doersch, C. (2016). Tutorial on variational autoencoders. Kingma, D. P., & Welling, M. (2019). An introduction to variational autoencoders. Deep Learning and Item Response Theory Curi, M., Converse, G. A., Hajewski, J., & Oliveira, S. (2019). Interpretable variational autoencoders for cognitive models. Wu, M., ..., & Goodman, N. (2020). Variational item response theory: Fast, accurate, expressive. Urban, C. J., & Bauer, D. J. (2021). A deep learning algorithm for high-dimensional exploratory item factor analysis.