Oct 31, 2023 GS.09
In the machine learning community, structured representations have demonstrated themselves to be hugely beneficial for efficient learning from limited data and generalization far beyond the training set. Examples of such structured representations include the spatially organized feature maps of convolutional neural networks, and the group structured activations of other equivariant models. To date however, the integration of such structured representations with deep neural networks has been limited to explicitly geometric transformations (such as spatial translation or rotation), known a priori to model developers. In the real world, we know that natural intelligence is able to efficiently handle novel transformations and flexibly generalize in a manner reminiscent of these structured artificial models, but crucially with respect to a broader class of non-geometric transformations. In this talk, we investigate how naturally intelligent systems might accomplish this through what we denote ‘natural representational structure’. Specifically, we investigate two types of structure observed in the brain — topographic organization, and traveling waves. By developing novel deep neural network architectures which exhibit such natural structure, we show empirically that such models indeed learn approximately ‘equivariant’ representations, similar to their explicitly geometrically structured counterparts, but in a much more flexible manner, where structure is learned directly from the data itself. We show that this structure both improves artificial neural networks, and simultaneously helps to understand observations from neuroscience itself.