E2062 - xHC: Expanded Hyper-Connections

🤗 Upvotes: 47 | cs.LG, cs.CL

Authors:
Xiangdong Zhang, Xiaohan Qin, Sunan Zou, Tuo Dai, Xiaoming Shi, Huaijin Wu, Yebin Yang, Zhuo Xia, Shaofeng Zhang, Lin Yao, Yuliang Liu, Yu Cheng, Junchi Yan

Title:
xHC: Expanded Hyper-Connections

Arxiv:
http://arxiv.org/abs/2607.14530v1

Abstract:
Hyper-Connections (HC) expand the residual stream of Transformers into $N$ parallel streams, providing a form of memory scaling beyond model width and depth. Manifold-Constrained HC (mHC) stabilizes this formulation at scale. The large gains from $N{=}1$ to $N{=}4$ suggest residual-stream expansion as a promising sca...