Bio

Standard Vision Transformers flatten 2D images into 1D sequences, disrupting the natural spatial topology. While Rotary Positional Embedding (RoPE) excels in 1D, it inherits this limitation, often treating spatially distant patches (e.g., at row edges) as sequence neighbors. Existing 2D approaches typically treat spatial axes independently, failing to decouple this false sequential proximity from true spatial distance. To restore the 2D spatial manifold, we introduce Geometric Positional Embedding (GeoPE), a framework that extends rotations to 3D Euclidean space using quaternions. To overcome non-commutativity and ensure symmetry, GeoPE constructs a unified rotational operator by computing the geometric mean in the Lie algebra. This creates a geometrically coupled encoding that effectively separates spatial dimensions. Extensive experiments on image classification, object detection, and 3D semantic segmentation demonstrate that GeoPE consistently outperforms existing 2D RoPE variants and significantly enhances shape bias, confirming its ability to capture true geometric structure.

The neural network (NN) looks for the hypothesis function in the search space, where the hypothesis function can be conceptualized as a continuous interpolating function. Nevertheless, the issue of overfitting commonly arises in interpolation methods that focus on function values, such as Lagrange interpolation. We construct the geometric intuition to improve extrapolation and induce the Jacobian Regularization (JR) by Hermite interpolation. The concept of Jacobian Regularization resembles the gradient penalty (GP) employed in the Wasserstein GAN, with an experiment verifying the feasibility of our method.