Understanding Generalization: Learning Quantifiers and Negation with Neural Tensor Networks
Michael Repplinger

Abstract:
The following investigation is focused on the intersection of symbolic and distributional approaches to natural language semantics. Broadly speaking, we analyze symbolic approaches to semantics in order to learn how the performance of distributional models can be improved that are applied to compositional language tasks. Specifically then, we set out to discover empirical justification for the often claimed advantage of employing higher-order tensors in semantic vector space models.
Using an earlier natural language inference experiment, and adjusting it towards a more stringent test of generalization performance, we are able to find clear signs in support of the conjectured advantage of tensor models. In our experiments, a clear difference emerges in generalization performance between a conventional matrix-based tree-structured neural network and a tensor-based variant of the architecture.
To discover the cause for this performance difference, we visualize sentence representations of models trained on the semantic task. We find evidence of linearization of complex data in the tensor model, such as hyperplane separation of negated expressions, and the systematic organization of quantified expressions. We then suggest an explanation of our prediction results, by linking the observed properties of the model representations with the logical requirements of the inference task.