The extreme flexibility of digital circuits has allowed modern processors based on the Von Neumann architecture to not only efficiently implement algorithms for a wide variety of problems, but to consistently improve system performance at an exponential rate. However, with continued aggressive device scaling constrained by power- and voltage-considerations, the time and energy spent transporting data between memory and processor (across the so-called “Von- Neumann bottleneck”) has become problematic for data-centric applications such as real-time image recognition and natural language processing. One path to non-Von Neumann (non-VN) systems capable of full real time learning could be designed moving from reliable but binary devices to dense and analog (but less reliable) nanoscale elements. Previous work has been carried out, for example,with phase-change devices, obtaining high machine learning performance in large-scale hardware/software systems, with more recent developments achieving classification accuracies equivalent to those of fully software-based training. This presentation will provide a perspective on how neuromorphic engineering could complement modern computers in commercially relevant tasks and successively give a critical review of non-VN systems implemented using non- volatile memory elements with focus on a particular backpropagation accelerator implemented with non-filamentary resistive memory devices.