From Scenes To Spikes: Understanding Vision From The Outside In