Kernel Regression with Tree-Induced Metrics

This paper presents a family of kernel models based on metrics derived from the structure of decision trees and random forests. Unlike the standard random-forest proximity, which only checks whether two observations fall into the same leaf, the proposed metrics also use information about decision paths, the depth of their divergence, and the depth of their lowest common ancestor. A weighted variant with trainable depth-dependent parameters is also introduced. The tree-based metrics are aggregated over a forest and used to construct exponential kernels. They can also be combined with the Euclidean distance in the original feature space. We prove that the resulting kernels are positive semidefinite and can therefore be used in standard kernel methods, including Nadaraya-Watson regression and kernel ridge regression. The proposed models are evaluated on classification datasets and in a controlled source-to-target regression experiment with a limited number of labeled target observations. The results show that information about the internal structure of decision trees can be useful beyond simple leaf coincidence. Several proposed variants improve upon conventional geometric kernels and the standard random-forest proximity kernel, while the best-performing metric depends on the prediction method and the dataset.