Graduate Seminar on Machine Learning: Machine Learning with Graphs

Logistics

Instructor: Arlei Silva

Lectures: Monday 2:00-2:50, room: MEB 128, building: Mechanical Engineering Building

Summary

Machine learning on graphs (MLG) is an exciting and growing research topic mainly for two reasons: (1) Many relevant real-world problems (in recommendation, infrastructure, healthcare, etc.) have some structure that can be captured as nodes, edges, and their attributes; and (2) machine learning has been the hottest topic in computer science in the past decade, so it has impacted the entire field. In this seminar, we will discuss some of the most recent papers on deep learning on graphs (Graph Neural Networks), which arguably has become the go-to approach for MLG. Many of these developments can be posed as 'How can I do X on a graph?', where 'X' might be convolution, pooling, reinforcement learning, etc. However, deep learning also provides a common framework with the potential to solve most (if not all) problems in network science, graph mining, and even graph algorithms, by fitting well-designed functions using data (sometimes lots of it). In the last six years, at least three books, 20 surveys, and hundreds of journal and conference papers supporting this claim have been published. This seminar is an opportunity to read and discuss the state-of-the-art in machine learning on graphs.

Credit Hours: 1-3

This is a one credit seminar. It can be converted to 3 credits with the addition of a class project or a (high quality) survey (both subject to the instructor’s approval).

Mission

Read and discuss recent papers on machine learning on graphs.

Target Audience

  1. Graduate students working on machine learning, network science, graphs, etc. that want to broaden their knowledge on machine learning with graphs;
  2. Undergraduate students who want to be exposed to machine learning beyond the classical problems (clustering, classification, regression, etc.);
  3. Anyone else who is interested.

Recommended Prerequisites

A course on machine learning (some examples are COMP 502, COMP 540, COMP 542, COMP 576, COMP 602, COMP 640-3, COMP 680).

Course Materials

The seminar will be focused on research papers that are publicly available.

Format

The seminar will be composed of two introductory lectures and paper presentations with discussions. Each presentation will be 30 minutes long and will be followed by 20 minutes of discussion. Each student should submit reviews for five papers of their choice during the semester. The reviews should follow the NeurIPS'22 guidelines and should be submitted the day before the presentation.

Tips for a good presentation:
  1. Provide the relevant background;
  2. Make sure the problem, motivation, and assumptions are clear;
  3. Skip the details and focus on the major contribution;
  4. Don't limit yourself to the paper, make connections with other work;
  5. Criticise the paper, point out how it could be improved.
You can contribute to the discussion in the following ways:
  1. Complementing the presentation with relevant information;
  2. Criticizing the paper in terms of contributions, soundness, presentation, ethics, etc.;
  3. Asking questions you had while reading the paper;
  4. Proposing novel research based on the paper;
  5. Suggesting other papers for further reading;
  6. Commenting on the how the paper fits the author(s) broader research agenda.

Grading

Grading will be Pass/Fail.
  1. Read each paper presented;
  2. Write a review for 5 out of 10 papers;
  3. Participate of the paper discussions;
  4. Present at least one paper.

Logistics

Paper reviews and slides should be submitted on Canvas. Questions should be submitted on Piazza. For personal matters, please email the instructor.

Rice Honor Code

Students are expected to adhere to the Rice Honor Code. You are encouraged to collaborate and to find resources online. However, all the material to be graded is expected to be original. The exceptional use of someone else's work, such as a figure, should be properly recognized.

Students with Disabilities

If you have a documented disability that may affect academic performance, you should: 1) make sure this documentation is on file with Disability Support Services (Allen Center, Room 111 / adarice@rice.edu / x5841) to determine the accommodations you need; and 2) meet with me to discuss your accommodation needs.

Related Courses at Rice

  1. COMP 559: Machine Learning with Graphs (taught by Prof. Arlei Silva in the Spring).
  2. ELEC 573: Network Science and Analytics (taught by Prof. Segarra in the Fall);
  3. ECE677-001: Distributed Optimization and ML (taught by Prof. Uribe in the Fall);
  4. CAAM 570: Graph Theory (taught by Prof. Hicks in the Spring).

Schedule

Suggested topics and papers

  1. Architecture: e.g. pooling, aggregation, depth
    • Yang et al. Graph neural networks inspired by classical iterative algorithms. ICML, 2021
    • Alon and Yahav. On the bottleneck of graph neural networks and its practical implications. ICLR, 2020

  2. Theory: e.g. expressive power, invariances
    • Ganea et al. Independent se (3)-equivariant models for end-to-end rigid protein docking. ICLR, 2021
    • Xu et al. Optimization of graph neural networks: Implicit acceleration by skip connections and more depth. ICML, 2021

  3. Supervised learning: e.g. node classification, graph classification, link prediction
    • Qu et al. Neural structured prediction for inductive node classification. ICLR, 2021
    • Baek et al. Accurate learning of graph representations with graph multiset pooling. ICLR, 2020

  4. Scalability and systems: e.g. distributed algorithms, simplified architectures
    • Sriram et al. Towards training billion parameter graph neural networks for atomic simulations. ICLR, 2021
    • Kaler et al. Accelerating training and inference of graph neural networks with fast sampling and pipelining. MLSys, 2022

  5. Fairness and privacy: e.g. information leakage, private computation
    • Agarwal et al. Towards a unified framework for fair and stable graph representation learning. UAI, 2021
    • Liao et al. Information obfuscation of graph neural networks. ICML, 2021

  6. Reinforcement learning: e.g. policy learning
    • Paliwal et al. Reinforced genetic algorithm learning for optimizing computation graphs. ICLR, 2019
    • Trivedi et al. Graphopt: Learning optimization models of graph formation. ICML, 2020.

  7. Alternatives: e.g. label propagation, Markov Random Fields, transformers
    • Ying et al. Do transformers really perform badly for graph representation? Neurips, 2021.
    • Huang et al. Combining label propagation and simple models out-performs graph neural networks. ICLR, 2020

  8. Interpretability: e.g. GNN explainers
    • Miao et al. Interpretable and generalizable graph learning via stochastic attention mechanism. ICML, 2022.

  9. Generalizations: e.g. hypergraphs, heterogenous graphs, dynamic graphs
    • Chen et al. Tamps2gcnets: coupling time-aware multipersistence knowledge representation with spatio-supragraph convolutional networks for time-series forecasting. ICLR, 2021.
    • Bodnar et al. Weisfeiler and lehman go topological: Message passing simplicial networks. ICML, 2021.

  10. Unsupervised learning: e.g. autoencoders, self-supervised learning
    • Hassani and Khasahmadi. Contrastive multi-view representation learning on graphs. ICML, 2020.
    • Zhang and Li. Nested graph neural networks. Neurips, 2021.

  11. Adversarial learning: e.g. attacks, robustness
    • Wu et al. Graph information bottleneck. Neurips, 2020.
    • Xi et al. Graph backdoor. USENIX Security, 2021.

  12. Healthcare and bioinformatics: e.g. drug discovery, diagnosis, epidemics forecasting
    • Kim et al. Learning dynamic graph representation of brain connectome with spatio-temporal attention. Neurips, 2021

  13. NLP and vision: e.g. translation, word embeddings, question-answering, text classification
    • Shen et al. Unsupervised dependency graph network. ACL, 2022.
    • Yuxian Meng, Shi Zong, Xiaoya Li, Xiaofei Sun, Tianwei Zhang, Fei Wu, and Jiwei Li. Gnnlm: Language modeling based on global contexts via gnn. In International Conference on Learning Representations, 2021.

  14. Knowledge graphs: e.g. completion, alignment, reasoning
    • Yu et al. Kg-fid: Infusing knowledge graph in fusion-in-decoder for open-domain question answering. ACL, 2022.
    • Zhang et al. Greaselm: Graph reasoning enhanced language models. ICLR, 2021.

  15. Traffic and mobility: e.g. traffic forecasting, trajectory prediction
    • Wang et al. Metro passenger flow prediction via dynamic hypergraph convolution networks. IEEE Transactions on Intelligent Transportation Systems, 2021.
    • Bai et al. Adaptive graph convolutional recurrent network for traffic forecasting. Neurips, 2020.

  16. Physical sciences: e.g. physical reasoning, simulation, PDE solving
    • Lienen and Gunnemann. Learning the dynamics of physical systems from sparse observations with finite element networks. ICLR, 2021.

  17. Program analysis: e.g. type inference, code summarization, bug detection
    • Dinella et al. Hoppity: Learning graph transformations to detect and fix bugs in programs. ICLR, 2020.
    • Pashakhanloo et al. Codetrek: Flexible modeling of code using an extensible relational representation. ICLR, 2021.

  18. Hyperparameter tuning and AutoML: e.g. architecture search, graph learning
    • Chen et al. A unified lottery ticket hypothesis for graph neural networks. ICML, 2021.
    • Zhang et al. Deep and flexible graph neural architecture search. ICML, 2022.

  19. Robotics: e.g. motion planning
    • Zhou et al. Multi-robot collaborative perception with graph neural networks. IEEE Robotics and Automation Letters, 2022.
    • Yu and Gao. Reducing collision checking for sampling-based motion planning using graph neural networks. Neurips, 2021.

  20. Algorithms: e.g. learning graph algorithms
    • Bai et al. Glsearch: Maximum common subgraph detection via learning to search. ICML, 2021.
    • Meirom et al. Optimizing tensor network contraction using reinforcement learning. ICML, 2022.

Additional references

Other resources