TensorRT optimizes deep learning inference by quantizing and fusing operations for NVIDIA GPUs
Image: BigRiz, CC BY-SA 3.0, via Wikimedia Commons
TensorRT does: NVIDIA's inference optimizer that quantizes and fuses operations
TensorRT optimizes deep learning inference by quantizing and fusing operations for NVIDIA GPUs
Related concepts
tensor cores are
Why can computers crunch numbers faster than humans?
quantization to INT8 doubles throughput
Quantization to INT8 doubles throughput because tensor cores process INT8 2x faster
Adam has bias correction: divides by (1-β^t) in early steps
Why do we sometimes need to fix mistakes in computer decisions?
CUDA
CUDA enables parallel computation on GPUs
GPTQ quantization does
Post-training quantization using second-order information for model compression
XLA does for TensorFlow/JAX: compiles computation graphs for TPU/GPU execution
XLA accelerates TensorFlow models
Swipe through 100 ML concepts daily
Open Pocket Polymath