A Domain-Specific Architecture for Deep Neural Networks | September 2018 | Communications of the ACM

Moore’s Law is dead. Let’s not mince words.

DRAM chips introduced in 2014 contained eight billion transistors. Chips with 16-billion will not be mass produced until 2019. At the very least, the growth exponent has flattened.

Meanwhile, the company associated with Moore, Intel, is struggling to stay relevant with its CISC architecture. The RISC-based ARM architecture used by Apple in the iPad line is approaching the performance of Intel, while using significantly less power.

The lesser-known Dennard Scaling law is also dead. Power utilization is scaling more linearly with the number of circuits.

What this means, in practice, is new computing nodes based on traditional architectures will require more power to feed smaller advances in performance. Consumers haven been aware of this trend already, but it will only get worse until new architectures achieve mainstream status. And then advances will depend a lot more on what you are planning to do with it.

We are already at the early edge of this. Or not. Gamers have long been aware of the benefits of graphical processing units (GPUs) for improve gaming performance and would pay dearly for those benefits. GPUs exploit parallelism in matrix calculations (see linear mathematics) to significantly improve performance in specific domains (I.e. gaming) while leaving other forms of computation untouched. Few other desktop computing applications exploit GPUs capabilities.

There are a couple popular exceptions:

  • Bitcoin mining

  • Neural network learning

The rise in these applications has been a boon for Nvidia, while the recent declines in the former has dropped the bottom out of their growth projections.

What the death of Moore’s Law and Denning Scaling portends is:

  • Flattening of general purpose processing CPUs.

  • A shift from CISC to RISC CPU architectures. Apple and Microsoft have already shown signs they want to shift their PCs towards ARM-based processors that use less power — this will be pronounced by 2020.

  • Increasing reliance on alternative, domain-specific computation architectures that excel in specific areas.

Other buzzwords include:

  • Field Programmable Gate Arrays (FPGAs)

  • Custom ASICs

  • Tensor Processing Unit (TPU)

  • Neuromorphic computing

Most of these concepts are not new, but all are becoming newly relevant.

I have to admit I find the literature on computing architecture to be tedious and dull. I stopped following advances in Intel chips a long time ago. However, if computing architecture is your thing, it is a great time to be alive.

A Domain-Specific Architecture for Deep Neural Networks | September 2018 | Communications of the ACM
— Read on