Efficient And Secure Learning Across Memory Hierarchy