Expressivity of Graph Neural Networks: ReLU vs. Sigmoid GNNs

Graph Neural Networks (GNNs) form a powerful computational framework for machine learning on graphs with many applications. However, many open questions exist regarding how the chosen activation function affects expressivity, trainability and generability. In this short communication, we show that ReLU GNNs are more expressive than Sigmoid GNNs. However, we also show that Sigmoid GNNs are expressive enough so that this gap is not significant in practice, meaning that the comparison between ReLU and Sigmoid GNNs should be done in terms of trainability and generability.

This is work accepted and presented at ICLR'25 [3], which is of interest to ISSAC.

In this short communication for ISSAC, we present briefly the result of [3] focused on the case of ReLU vs. Sigmoid GNNs. As of today, the effect of the activation function on the properties of GNNs (expressivity, trainability and generability) is widely unknown. The work in [3] focused in the expressivity of GNNs through the perspective of logical queries following [1, 2].