Learning Contextual Embeddings for Structural Semantic Similarity using Categorical Information

Tree kernels (TKs) and neural networks are two effective approaches for automatic feature engineering. In this paper, we combine them by modeling context word similarity in semantic TKs. This way, the latter can operate subtree matching by applying neural-based similarity on tree lexical nodes. We study how to learn representations for the words in context such that TKs can exploit more focused information. We found that neural em- beddings produced by current methods do not provide a suitable contextual similar- ity. Thus, we define a new approach based on a Siamese Network, which produces word representations while learning a bi- nary text similarity. We set the latter con- sidering examples in the same category as similar. The experiments on question and sentiment classification show that our se- mantic TK highly improves previous re- sults.