Logistics
Instructor: Arlei Silva
Lectures: Monday 2:00-2:50, room: DCC 113, building: Duncan Hall
Summary
Machine learning on graphs (MLG) is an exciting and growing research topic mainly for two reasons: (1) Many relevant real-world problems (in recommendation, infrastructure, healthcare, etc.) have some structure that can be captured as nodes, edges, and their attributes; and (2) machine learning has been the hottest topic in computer science in the past decade, so it has impacted the entire field. In this seminar, we will discuss some of the most recent papers on deep learning on graphs (Graph Neural Networks), which arguably has become the go-to approach for MLG. Many of these developments can be posed as 'How can I do X on a graph?', where 'X' might be convolution, pooling, reinforcement learning, etc. However, deep learning also provides a common framework with the potential to solve most (if not all) problems in network science, graph mining, and even graph algorithms, by fitting well-designed functions using data (sometimes lots of it). In the last five years, three books, about 20 surveys, and hundreds of journal and conference papers supporting this claim have been published. This seminar is an opportunity to read and discuss the state-of-the-art in deep learning on graphs.
Format
The seminar will be composed of an introductory lecture and paper presentations with discussions. For the presentations, we will adopt a
role-based format instead of the most traditional one. The discussion of each paper will be lead by a group of 4 students, each of them will select one of the following roles:
- Presenter: Provides an intuitive and concise description of the problem and key ideas of the paper (25 minutes)
- Reviewer: Writes a conference-style review and share it with the instructor before class
- Researcher: Tries to propose a novel research problem based on the paper
- Archeologist: Briefly discusses important previous and future work (if any) related to the paper (mini-survey)
- Hacker: Tries to find an implementation of the proposed approach and run (very) simple experiments
- Biographer: Tells a bit about the authors of the paper (names, backgrounds, research interests, affiliations, other publications, etc.)
- Ethicist: Discusses possible ethical issues (privacy, fairness, manipulation, etc.) related to the paper
- Practitioner: Describes how far is the proposed approach to be deployed as a technology
Roles 1-2 should be selected for each paper and 3-8 are optional. Role 2 will share the review with the instructor before class and the review will be shared with everyone after the class. Roles 3-8 have 5 minutes to summarize their work during the class (slides are optional). Students should not repeat roles during the semester but can perform multiple roles for the same paper if desired (other students will have priority for the second, third, and fourth roles). The remainder of the class participates in the discussion and submits a very short (like a paragraph/abstract) summary of the paper (problem, the idea of the approach, relevance, results, etc.) before the class. One paper will be presented per lecture (25-minute presentation, 5 minutes for roles 3-8 roles, 15-minute discussion). All students are expected to contribute to the discussion (with questions, comments, etc.). The expectation is that each student will perform 4 different roles throughout the semester.
Syllabus
- August 30: Seminar logistics, introduction to deep learning on graphs
- September 6: Holiday (Labor Day)
- September 13: Satorras et al. E (n) equivariant graph neural networks. ICML 2021
- September 20: Jin et al. Hierarchical Generation of Molecular Graphs using Structural Motifs. ICML 2020
- September 27: Wu et al. Simplifying Graph Convolutional Networks. ICML 2019
- October 4: Velickovic et al. Pointer Graph Networks. Neurips 2020
- October 11: Midterm recess
- October 18: Li et al. Training Graph Neural Networks with 1000 Layers. ICML 2021
- October 25: Hu et al. Open Graph Benchmark: Datasets for Machine Learning on Graphs. Neurips 2020
- November 1: Xu et al. What can Neural Networks Reason About? ICLR 2020
- November 8: Paper presentation 8
- November 15: Paper presentation 9
- November 22: Paper presentation 10
- November 29: Paper presentation 11 (or discussion)
Candidate Reading list
- Diego Mesquita, Amauri Souza, Samuel Kaski. Rethinking pooling in graph neural networks. Neurips 2020 [pooling, graph classification]
- Guohao Li, Matthias Müller, Bernard Ghanem, Vladlen Koltun. Training Graph Neural Networks with 1000 Layers. ICML 2021 [depth, performance, node classification]
- Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, Bo Yang. Geom-GCN: Geometric Graph Convolutional Networks. ICLR 2020 [new model, node classification, hyperbolic]
- Xinlei He, Jinyuan Jia, Michael Backes, Neil Zhenqiang Gong, Yang Zhang. Stealing links from graph neural networks. USENIX 2021 [privacy, attacks, node classification]
- Lu Mi, Hang Zhao, Charlie Nash, Xiaohan Jin, Jiyang Gao, Chen Sun, Cordelia Schmid, Nir Shavit, Yuning Chai, Dragomir Anguelov. HDMapGen: A Hierarchical Graph Generative Model of High Definition Maps. CVPR 2021 [generative models, computer vision]
- Muhan Zhang, Yixin Chen. Inductive Matrix Completion Based on Graph Neural Networks. ICLR 2020 [recommender systems, transfer learning]
- Lei Bai, Lina Yao, Can Li, Xianzhi Wang, Can Wang. Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting. Neurips 2020 [traffic, dynamic networks]]
- Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, Austin R. Benson. Combining label propagation and simple models out-performs graph neural networks. ICLR 2021 [criticism, node classification]
- Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, Jure Leskovec. Open Graph Benchmark: Datasets for Machine Learning on Graphs. Neurips 2020 [benchmarks]
- Eli A. Meirom, Haggai Maron, Shie Mannor, Gal Chechik. Controlling Graph Dynamics with Reinforcement Learning and Graph Neural Networks. ICML 2021 [reinforcement learning, epidemics]
- Kaveh Hassani, Amir Hosein Khasahmadi. Contrastive Multi-View Representation Learning on Graphs. ICML 2020 [self-supervised learning, node classification, graph classification]
- Daniel D. Johnson, Hugo Larochelle, Daniel Tarlow. Learning Graph Structure With A Finite-State Automaton Layer. Neurips 2020 [program analysis, reinforcement learning]
- Keyulu Xu, Jingling Li, Mozhi Zhang, Simon S. Du, Ken-ichi Kawarabayashi, Stefanie Jegelka. What Can Neural Networks Reason About? ICLR 2020 [theory, algorithms]
- Victor Garcia Satorras, Emiel Hoogeboom, Max Welling. E (n) equivariant graph neural networks. ICML 2021 [theory, chemistry, dynamical systems]
- Michael Sejr Schlichtkrull, Nicola De Cao, Ivan Titov. Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking. ICLR 2021 [NLP, interpretability]
- Deyu Bo, Xiao Wang, Chuan Shi, Meiqi Zhu, Emiao Lu, Peng Cui. Structural deep clustering network. WWW 2020 [clustering, encoders]
- Wengong Jin, Regina Barzilay, Tommi Jaakkola. Hierarchical Generation of Molecular Graphs using Structural Motifs. ICML 2020 [generative models, chemistry, encoders, hierarchical]
- Petar Velickovic, Lars Buesing, Matthew C. Overlan, Razvan Pascanu, Oriol Vinyals, Charles Blundell. Pointer Graph Networks. Neurips 2020 [new model, link prediction, algorithms]
References
- William L. Hamilton. Graph representation learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2020
- Yao Ma, Jiliang Tang. Deep Learning on Graphs. Cambridge University Press, 2020
- Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, Philip S. Yu. A Comprehensive Survey on Graph Neural Networks. arxiv 2019
- Ziwei Zhang, Peng Cui, Wenwu Zhu. Deep Learning on Graphs: A Survey. arxiv 2018
- Ines Chami, Sami Abu-El-Haija, Bryan Perozzi, Christopher Ré, Kevin Murphy. Machine learning on graphs: A model and comprehensive taxonomy. ArXiv 2020
- Faezeh Faez, Yassaman Ommi, Mahdieh Soleymani Baghshah, Hamid R. Rabiee. Graph deep learning: State of the art and challenges. IEEE Access 2021
- N. A. Asif et al. Graph Neural Network: A Comprehensive Review on Non-Euclidean Space. IEEE Access 2021
Other resources