GCNs

Semi Supervised Classification with Graph Convolutional Networks

MIM Lab
Katsuya Ogata

1. Basic Knowledge of GNN
2. Introduction
3. Fast Approximate Convolutions on Graphs
4. Semi-Supervised Node Classification
5. Experiments
6. Results
7. Discussion
8. Appendix

Basic Elements of a Graph

• Node
  A vertex in the graph, e.g., a user in a social network or an atom in a molecule.

• Edge
  A connection between two nodes, e.g., friendship between users or bonds between atoms.

• Adjacency Matrix
  A matrix representing the connectivity of nodes. If node $i$ and node $j$ are connected, $A_{ij} = 1$; otherwise, $0$.

Graph Spectral Theory

• Study the properties of graphs by analyzing the eigenvalues and eigenvectors of matrices associated with the graph.

• We reveal crucial structural and global properties, such as:

  • Connectivity (how well the graph is connected)
  • Bipartiteness (if the graph can be divided into two independent sets)
  • The presence of certain motifs or communities

Graph Laplacian

• Definition
  $L := D - A$

  Where:

  • $D$ = Degree matrix (diagonal, $D_{ii}$ = degree of node $i$)
  • $A$ = Adjacency matrix

• Properties

  • Symmetric (if undirected graph)
  • Captures the difference between a node and its neighbors

Normalized Laplacian

• Definition
  $\tilde{L} := D^{-1/2} L D^{-1/2} = I - D^{-1/2} A D^{-1/2} = I -\tilde{A}$

  Where:

  • $\tilde{A} = D^{-1/2} A D^{-1/2}$ is the normalized adjacency

• Why normalize?

  • Removes scale differences due to node degrees
  • Makes it easier to compare graphs with different structures

Graph Fourier Transform

• Forward transform

  $$F(x) = U^T x$$

• Inverse transform

  $$F^{-1}(x) = U x$$

Where:

• $U$: matrix of eigenvectors ($\tilde{L} = U \Lambda U^T$)
• $x$: graph signal

Spectral Convolution

• Definition (using Fourier domain):

  $$g * x = F^{-1}(F(g) \odot F(x)) = U (U^T g \odot U^T x)$$

Where:

• $\odot$: element-wise multiplication

Practical Filtering

• Direct use of $U^T g$ is often impractical.

• Instead, we typically use a learnable diagonal matrix $g_w$:

  $$g_w * x = U g_w U^T x$$

This simplifies the filter design and makes learning scalable.

Summary

1. Transform the graph signal ( x ) into the spectral domain
2. Apply a filter in the spectral domain
3. Return to the original space

Loss Functions of GNN

$$L = L_0 + \lambda L_{\text{reg}}$$

where:

• $L_0$ : supervised loss over the labeled part of the graph
• $\lambda$ : weighting factor controlling the strength of the regularization
• $L_{reg}$: graph regularization term

Regularization Term

$$L_{\text{reg}} = \sum_{i, j} A_{ij} \| f(X_i) - f(X_j) \|^2 = f(X)^\top \Delta f(X)$$

where:

• $f(\cdot)$: neural network-like differentiable function
• $X$: matrix of node feature vectors $X_i$
• $A$: adjacency matrix
• $\Delta = D - A$: unnormalized graph Laplacian

Homophily Hypothesis

• Homophily refers to the tendency of connected nodes to share similar attributes or labels.

• In graph learning, it is assumed that:

  "Connected nodes are likely to belong to the same class."

Proposed Methods

$$loss = L_0$$

$$output = f(X, A)$$

Where:

• $X \in \mathbb{R}^{N \times D}$: Node feature matrix,
  where $N$ = number of nodes, $D$ = number of features
• $A \in \mathbb{R}^{N \times N}$: Adjacency matrix
• $f(\cdot)$: Neural network mapping features and graph structure
• $L_0$: Supervised loss on labeled nodes

Computational Cost Issues

$$g_\theta \star x = U g_\theta U^T x$$

$$( \theta \in R^N, x \in R^N )$$

1. Multiplication with eigenvector matrix $U$:

   • Computational complexity is $O(N^2)$ ($N$: number of nodes)
   • Very expensive for large graphs

2. Eigendecomposition of graph Laplacian $L$:

   • Necessary to obtain $U$
   • Also computationally prohibitive for large graphs

Solution: Approximation via Chebyshev Polynomials

Proposal by Hammond et al. (2011):
Approximate $g_\theta(\Lambda)$ by a truncated expansion in terms of Chebyshev polynomials $T_k(x)$ up to $K^{th}$ order.

Approximation (Eq. 4):

$$g_{\theta'}(\Lambda) \approx \sum_{k=0}^K \theta'_k T_k(\tilde{\Lambda})$$

Components of Eq. (4) and Chebyshev Polynomials

• $\tilde{\Lambda}$ (Rescaled eigenvalues):

  $$\tilde{\Lambda} = \frac{2}{\lambda_{max}}\Lambda - I_N$$

  • $\lambda_{max}$: Largest eigenvalue of $\tilde{L}$
  • The range of $\tilde{\Lambda}$ becomes $[-1, 1]$, matching the domain of Chebyshev polynomials.

• Chebyshev polynomials $T_k(x)$:

  • $T_0(x) = 1$
  • $T_1(x) = x$
  • $T_k(x) = 2xT_{k-1}(x) - T_{k-2}(x)$ (Recursive definition)

Convolution using the Approximation

Applying the approximation (Eq. 4) to the original convolution definition
$g_\theta \star x = U g_\theta(\Lambda) U^T x$:

$$g_{\theta'} \star x \approx U \left( \sum_{k=0}^K \theta'_k T_k(\tilde{\Lambda}) \right) U^T x$$

$$g_{\theta'} \star x \approx \sum_{k=0}^K \theta'_k U T_k(\tilde{\Lambda}) U^T x$$

Using

$$(U\tilde{\Lambda}U^T)^k = U\tilde{\Lambda}^kU^T$$

$$T_k(\tilde{L}) = U T_k(\tilde{\Lambda}) U^T$$

we get:

$$g_{\theta'} \star x \approx \sum_{k=0}^K \theta'_k T_k(\tilde{L})x$$

where:

$$\tilde{L} = \frac{2}{\lambda_{max}}L - I_N$$

Important Properties:

1. $K$-localized:

   • $T_k(\tilde{L})$ is a $K^{th}$-order polynomial in the Laplacian.
   • The convolution result depends only on nodes that are at maximum $K$ steps away from the central node (Kth-order neighborhood).

2. Computational Complexity:

   • Evaluating Eq. (5) is $O(|E|)$ ($|E|$: number of edges).
   • $T_k(\tilde{L})x$ can be computed efficiently through repeated sparse matrix-vector multiplications.

Approximation and Simplification

We approximate $\lambda_{max} \approx 2$ and $K=1$.(It is expected that neural network parameters will adapt to this change in scale during training.)

Under these approximations, Eq. 5 simplifies to:

$$g_{\theta'} \star x \approx {\theta'}_0 x + {\theta'}_1 (L-I_N)x$$

$$g_{\theta'} \star x = {\theta'}_0 x - {\theta'}_1 D^{1/2}AD^{1/2}x$$

$$g_{\theta'} \star x \approx \theta(I_N + D^{1/2}AD^{1/2})x$$

Renormalization Trick

To alleviate the problem of numerical instabilities, the following renormalization trick is introduced:

$$I_N + D^{-1/2}AD^{-1/2} \rightarrow \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}$$

Where:

• $\tilde{A} = A + I_N$ (Adjacency matrix with self-connections added)
• $\tilde{D}_{ii} = \sum_j \tilde{A}_{ij}$ (Degree matrix of $\tilde{A}$)

Generalization to Multiple Channels/Filters

$$Z = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}X\Theta \quad (Eq. 8)$$

Where:

• $X \in \mathbb{R}^{N \times C}$ is the matrix of input graph signal.
• $\Theta \in \mathbb{R}^{C \times F}$ is the matrix of filter parameters.
• $Z \in \mathbb{R}^{N \times F}$ is the convolved signal matrix.
• This filtering operation has a complexity of $O(|E|FC)$, as $\tilde{A}X$ can be efficiently implemented as a product of a sparse matrix with a dense matrix.

Two-Layer GCN

$$H^{(l+1)} = \sigma(\tilde{D}^{1/2}\tilde{A}\tilde{D}^{1/2}H^{(l)}W^{(l)})$$

$$Z = f(X, A) = softmax((\hat{A} ReLU(\hat{A}XW^{(0)})W^{(1)}))$$

Where:

• $\hat{A} = \tilde{D}^{1/2}A\tilde{D}^{1/2}$, $W^{(0)} \in R^{C \times H}$, $W^{(0)} \in R^{H \times F}$

Loss Function

Evaluate the cross-entropy error over all labeled examples：

$$L = -\sum_{l \in \mathcal{Y}_L} \sum_{f=1}^{F} Y_{lf} \ln Z_{lf}$$

Explanation of symbols:

• $\mathcal{Y}_L$: index set of labelled nodes
• $F$: number of output classes
• $Y_{lf}$: true label for class $f$ of node $l$.
• $Z_{lf}$: predicted probability for class $f$ of node $l$.

Datasets Overview

Label Rate: Number of labeled nodes used for training / Total nodes

Citeseer, Cora, and Pubmed (Citation networks)

Structure:

• Nodes: Documents
• Edges: Citation links (treated as undirected)
• Features: Sparse bag-of-words vectors
• Adjacency Matrix: Binary, symmetric

Training Setup:

• Only 20 labels per class for training
• All feature vectors available
• Each document has a class label

Knowledge Graph Structure (NELL)

Original Format:

• Entities connected with directed, labeled edges (relations)
• Example: (entity₁, relation, entity₂)

Preprocessing:

• Convert to bipartite graph: (e₁, r₁) and (e₂, r₂)
• 55,864 relation nodes + 9,891 entity nodes
• Extended features: 61,278-dim sparse vectors

Extreme Semi-supervised Setting:

• Only 1 labeled example per class (210 classes total)

For Runtime Analysis (Random graphs)

Generation Process:

• $N$ nodes → $2N$ edges assigned uniformly at random
• Feature Matrix: Identity matrix $I_N$ (featureless approach)
• Each node represented by unique one-hot vector
• Dummy labels: $Y_i = 1$ for all nodes

Purpose: Measure training time per epoch across different graph sizes

Model Configuration

• Architecture: 2-layer GCN (Section 3.1)
• Test Set: 1,000 labeled examples
• Validation Set: 500 labeled examples for hyperparemeter optimization (dropout rate, the number of hidden units and L2 regularization factor)
• Optimizer: Adam (learning rate = 0.01)
• Max Epochs: 200
• Learning Rate 0.01
• Early Stopping: Window size = 10
• Weight Initialization: Glorot & Bengio (2010)

Semi-Supervised Node Classification Results

Propagation Model Evaluation

Comparing Different Variants

Training Time Analysis

Key Finding

Linear scalability enables application to very large graphs

Why GCN Outperforms Traditional Methods

1. End-to-End Learning

• Other methods: Multi-step pipeline (embedding learning → classifier training)
• GCN: Unified optimization with single loss function

2. Efficient Information Propagation

• Graph Laplacian methods: Limited by assumption that edges = node similarity
• GCN: Propagates feature information through neighbors at each layer

3. Computational Efficiency

• Complexity: O(|E|) - linear in number of edges
• Speed: 3-4x faster than Planetoid (Cora: 13s→4s)

Limitations and Future Work

Memory Requirements

• Mini-batch SGD, Approximate methods, Distributed training

Directed edges and edge features

• Native directed graph support, Edge feature integration, Heterogeneous graph handling

Current Limiting Assumptions

$$\tilde{A} = A + λI_N$$

• λ: Learnable trade-off parameter