GDL4LLM: Each graph is a new language: Graph Learning with LLMs

Sketched Overview

GDL4LLM introduces a novel approach to text-attributed graph learning with LLMs by treating graphs as a language rather than using natural language descriptions. Recognizing that natural language is too verbose and unstructured for modeling complex graph relationships, GDL4LLM translates graphs into a concise corpus on which LLMs can be pre-trained. This enables efficient representation of subgraphs with minimal tokens during fine-tuning. Experiments on real-world datasets show GDL4LLM outperforms description-based and embedding-based approaches by effectively modeling multi-hop neighborhoods.

📌 We convert the problem of modeling graph structures for LLMs into a graph language learning problem. We justify this approach by proving that the graph language learning objective enables LLMs to learn graph structural information.

🔧 We introduce GDL4LLM, a simple yet effective framework. It generates a graph language corpus from the given graph and pre-trains LLMs on this corpus to understand the graph. The framework then samples from the graph language corpus to represent subgraphs centered around target nodes for fine-tuning on downstream tasks.

📊 Through extensive experiments on three real-world datasets, we demonstrate that GDL4LLM outperforms competitive baselines. It surpasses both description-based and textual attribute embedding-based approaches by efficiently modeling different orders of neighbors with LLMs.

Overall Framework and Pipeline

The figure demonstrates a comparison between mainstream methods and GDL4LLM for node-classification task. Figure (a) utilizes LLMs to embed node attributes and leverages GNN to aggregate the embeddings. Figure (b) presents the descriptions of graph structure centered around target nodes. Figure (c) illustrates how LLMs are pre-trained to capture graph structures through graph language learning, and how textual attributes are further integrated to enhance LLMs fine-tuning.

Observations and Findings

(i) Pre-training significantly enhances GDL4LLM performance, especially when combined with textual attributes in prompts, creating a synergistic effect where structural understanding complements semantic comprehension.

(ii) GDL4LLM performs better with Llama-3 than Llama-2 due to architectural improvements and better training data, with ablation studies confirming the effectiveness of the pre-training objective across different LLM architectures.

Discipline-specific insights

We examine two critical hyperparameters: the length of sampled graph sentences l and the number of sampled sentences k. The figure shows optimal performance at l=5 and k=10, and the performance gain is marginal when approaching this value.

These results demonstrate our framework's effectiveness in modeling high-order structural information, such as inter-order dependencies. For instance, a length of 5 captures fourth-order structural information, whereas GNNs, often converging in about two layers, typically capture only second-order information.