Just-in-time Compilation#
The @polyblocks_jit_* decorators#
The @polyblocks_jit_* decorators can be attached to a function to specify that
the latter should be compiled and executed using the PolyBlocks JIT compiler.
Additionally, compile options can be specified as a trailing argument. The
target option is an important one and can be ‘cpu’, ‘nvgpu’, or ‘amdgpu’. It
defaults to ‘cpu’ when not provided. PyTorch (2.0) and JAX are supported.
PolyBlocks-powered PyTorch#
The PolyBlocks compiler supports the PyTorch framework. PyTorch/PolyBlocks
uses TorchDynamo with
the PolyBlocks engine as a backend to generate fast optimized code by compiling
the captured Torch FX graphs. One can either use an annotation or use PolyBlocks
as the backend compiler with torch.compile.
@polyblocks_jit_torch(compile_options: dict)
An example with PyTorch.
from polyblocks.torch_polyblocks_jit import polyblocks_jit_torch
...
@polyblocks_jit_torch(compile_options={'target': 'nvgpu'})
def blur_x_blur_y(img):
# We are doing a regular image convolution in the X direction and then in
# the Y direction.
kernel = torch.tensor([0.0625, 0.25, 0.375, 0.25, 0.0625], device = img.device).float()
blur_kernel = kernel.view(1, 1, 1, 5)
img_reshaped = img.view(3, 1, height, width)
# The output is two short at each end.
blurx = torch.nn.functional.conv2d(img_reshaped,
blur_kernel, stride=1, padding=0)
blur_kernel = kernel.view(1, 1, 5, 1)
blury = torch.nn.functional.conv2d(blurx, blur_kernel,
stride=1, padding=0)
output = blury.view(3, height - 4, width - 4)
return output
The function can be then called the usual way:
img = torch.rand(3, height, width, device=device).float()
out = blur_x_blur_y(img)
Another way is to decorate the function at the callsite. In this case, there is no need to annotate the blur_x_blur_y definition. The following example demonstrates it:
img = torch.rand(3, height, width, device=device).float()
blur_x_blur_y_polyblocks_compilable = polyblocks_jit_torch(
compile_options={'target': 'nvgpu'}
)(blur_x_blur_y)
out = blur_x_blur_y_polyblocks_compilable(img)
When executing on a system with NVIDIA GPUs, the CUDA_VISIBLE_DEVICES
environment variable can be used to control which one(s) the function can
execute.
Use torch.compile with “polyblocks” backend#
A more torch-native way of using PolyBlocks would be via torch.compile, and
the following is functionally equivalent to the polyblocks annotation-based approach:
# Register polyblocks as a backend for Torch.
import polyblocks.torch_polyblocks_jit
...
img = torch.rand(3, height, width, device=device).float()
polyblocks_compiled_func = torch.compile(blur_x_blur_y, backend="polyblocks")
out = polyblocks_compiled_func(img)
PolyBlocks compiled PyTorch functions support inputs of type torch.Tensor
irrespective of where the inputs reside. The only restriction is that all of the
inputs should be on the host, or all of them should be on a GPU.
For optimal end-to-end performance PolyBlocks/PyTorch GPU runtime does not
synchronize unless the data is brought back to CPU. However, cuda_synchronize
function from polyblocks.device_helpers can be used to make sure that all the
PolyBlocks accelerated work running on GPU is finished.
PolyBlocks-powered JAX#
Functions written in JAX can be compiled via PolyBlocks using the
@polyblocks_jit_jax annotation. @polyblocks_jit_jax has the following
syntax. (The annotation does not require an input specification.)
@polyblocks_jit_jax(compile_options: dict)
An example with JAX.
from polyblocks.jax_polyblocks_compiler import polyblocks_jit_jax
class CNN(nn.Module):
@nn.compact
def __call__(self, x):
x = nn.Conv(
features=out_channels,
kernel_size=filter_shape,
padding="VALID",
use_bias=True,
dtype=jnp.dtype("float16"),
param_dtype=jnp.dtype("float16"),
)(x)
x = nn.relu(x)
return x
# Care must be taken when passing the model parameters. They should always be
# passed as function arguments to the function that is being JITted. If we
# do not pass them in that manner and obtain them from the enclosing scope, all those parameters
# will be captured as constant in the resultant IR, and that will negatively impact
# compilation times as well as performance. When they are explicitly passed,
# they are captured as function parameters, which is better for compilation time
# and performance.
@polyblocks_jit_jax(compile_options={'target': 'nvgpu'})
def ConvBiasRelu(variables, img):
out = cnn.apply(variables, img)
return out
The JAX JIT annotation also supports passing static arguments (arguments to be
treated as compile-time constants) via the
static_argnums
compile option.
@polyblocks_jit_jax(
compile_options={
"target": "nvgpu",
"debug": False,
"static_argnums": (2, 3),
}
)
def gradient_descent(
X, D, learning_rate=0.0001, num_iterations=1000
):
In the above example, arguments at indices 2 and 3 are treated as compile-time constants.
All inputs (except static arguments) to a JAX function being compiled via
PolyBlocks must be JAX arrays obtained using
jax.device_put().
D_JAX = jax.device_put(D, jax.devices("cpu")[0])
Compare with other backends: polymage_jit_* annotations#
The flexibility to execute via separate backends also exists. A different
annotation, polymage_jit_jax, must be used to enable execution via different
backends. For JAX, there exist three different backends that can be selected
using the execution_backend option:
execution_backendcan be one ofstdfor JAX without JIT compilation,jitfor the JAX JIT, andpolyblocksfor the PolyBlocks backend. As an example:
@polymage_jit_jax(execution_backend="jit", jit_args={"target": "nvgpu"})
def conv_bias_add_relu(img, filters, bias):
conv = jax.lax.conv_general_dilated(
img,
filters,
window_strides=(1, 1),
padding="VALID",
dimension_numbers=("NHWC", "HWIO", "NHWC"),
)
return jax.nn.relu(conv + bias)
Tooling and developer ecosystem#
All well-known tools are able to run on PolyBlocks-optimized code and provide useful performance insights. This includes profiling, performance monitoring, and memory sanity-checking tools. For example, NVIDIA Nsight systems nsys can be used to profile and understand performance, and the CUDA compute sanitizer can be used to check for memory errors.
As an example, the execution profile from nsys below shows that PolyBlocks was able to fuse all operators into a single GPU kernel.
nsys profile -s none --cpuctxsw=none --trace=cuda -o gpu_ python blur_x_blur_y.py -gpu \
-skip-jax-std -skip-jax-jit -debug && nsys stats -q --report gpukernsum --format table gpu_.nsys-rep
** CUDA GPU Kernel Summary (gpukernsum):
+----------+-----------------+-----------+-----------+-----------+----------+----------+-------------+----------------+----------------+---------------------------------------+
| Time (%) | Total Time (ns) | Instances | Avg (ns) | Med (ns) | Min (ns) | Max (ns) | StdDev (ns) | GridXYZ | BlockXYZ | Name |
+----------+-----------------+-----------+-----------+-----------+----------+----------+-------------+----------------+----------------+---------------------------------------+
| 100.0 | 651,070 | 1 | 651,070.0 | 651,070.0 | 651,070 | 651,070 | 0.0 | 682 33 3 | 96 1 1 | __inference_BlurXY_15_Conv2D_1_kernel |
+----------+-----------------+-----------+-----------+-----------+----------+----------+-------------+----------------+----------------+---------------------------------------+