AOT Compilation#
The PolyBlocks compiler compiles PyTorch models ahead of time (AOT) from a
torch.export program, either into a library or into artifacts loaded and
run from Python.
Generating libraries for deployment/offline use#
torch_polyblocks_generate_library generates a library and its header,
<name>_mlir_artifact.o and <name>_mlir_artifact.h. An AOT workload
generates them with -libgen into its -artifact-dir, with -artifact-name
naming the function and the files. The generated library can be called from any
C/C++ application or from any language that can use a C API. Its only
dependence is on the MLIR runtime utils library (which is just a thin wrapper)
and on the CUDA runtime (if executing on the NVIDIA GPUs) or the OpenMP runtime
(for multicore CPUs).
The generated function takes the model’s parameters and buffers, then its
inputs, then its outputs. Tensors the model creates as constants are compiled
into the library. It does not take or provide ownership of any buffers/memory.
The caller is expected to manage input and output data’s memory.
-on-gpu-tensors can be used in conjunction with -libgen to specify that the
passed buffers are already on the GPU when compiling for GPUs.
AOT Features#
User-provided memory pool: The user can provide pre-allocated memory and have the generated library use that memory pool instead of allocating and deallocating on its own.
User-supplied stream (CUDA stream): This allows the user to provide an existing CUDA stream. User-provided memory pool and stream go together.
On-GPU tensors: The user can specify that the inputs and outputs of the generated function already reside on the device (GPU). The generated library will not perform any host-to-device transfers.
Example#
$ python examples/pytorch/aot/single_ops/blur_x_blur_y_aot.py -gpu -libgen \
-artifact-name blur_x_blur_y_polyblocks_gpu -artifact-dir .
The function is named after -artifact-name, and its arguments after the
graph’s operations:
/// Signature of the PolyBlocks-generated function.
/// Requires 402260672 bytes (3.836257e+02 MB) of memory to be supplied via the trailing argument.
/// Type/shape of view_default is memref<3x4096x4096xf32>.
/// Type/shape of output0 is memref<3x4092x4092xf32>.
/// Type/shape of user_device_memory is memref<402260672xi8>.
/// Type/shape of stream is !gpu.async.token.
extern "C" void blur_x_blur_y_polyblocks_gpu(float *view_default, float *output0, int8_t *user_device_memory, void *stream);
# Compile and link with it. The driver file has a few lines to call the
# generated function.
$ clang++ -O3 -I/usr/local/cuda/include blur_x_blur_y.cpp \
blur_x_blur_y_polyblocks_gpu_mlir_artifact.o -L/usr/local/cuda/lib64 -lcuda \
-lcudart -L/ws/release/llvm-mlir/lib -lmlir_cuda_runtime -o blur_x_blur_y
$ ./blur_x_blur_y
# Generate a shared object.
$ clang++ -O3 -fPIC -shared blur_x_blur_y_polyblocks_gpu_mlir_artifact.o \
-L/ws/release/llvm-mlir/lib -lmlir_cuda_runtime -L/usr/local/cuda/lib64 \
-lcuda -lcudart -o blur_x_blur_y_polyblocks_gpu.so
examples/libgen/pytorch builds and runs this example for the CPU and the
GPU. A library can also be generated from Python:
import torch
from polyblocks.torch_polyblocks_aot import torch_polyblocks_generate_library
if __name__ == "__main__":
model = MyModel().eval()
inputs = (torch.randn(1, 3, 224, 224),)
exported = torch.export.export(model, inputs)
torch_polyblocks_generate_library(
exported, inputs, "my_model", "lib_out", {"target": "nvgpu"}
)
Compiling and loading PyTorch models#
torch_polyblocks_compile_aot compiles a torch.export program ahead of
time, and torch_polyblocks_load_aot loads the result and runs it from
Python.
import torch
from polyblocks.torch_polyblocks_aot import (
torch_polyblocks_compile_aot,
torch_polyblocks_load_aot,
)
if __name__ == "__main__":
model = MyModel().eval().cuda()
inputs = (torch.randn(1, 3, 224, 224, device="cuda"),)
exported = torch.export.export(model, inputs)
artifact_dir = torch_polyblocks_compile_aot(
exported, inputs, "my_model", "aot_out", {"target": "nvgpu"}
)
module = torch_polyblocks_load_aot(artifact_dir)
outputs = module(*inputs)
The compiler runs parts of itself in processes that import the main script, so
a script that compiles keeps its top-level code under
if __name__ == "__main__":.
torch_polyblocks_compile_aot writes the artifacts that
torch_polyblocks_load_aot runs to aot_out/my_model:
lib<name>.so: the compiled model.weights.pt: the model’s weights.manifest.json: what each argument of the library is.build/: the compiler’s intermediate files.
The module torch_polyblocks_load_aot returns allocates the model’s output
tensors once, when it is loaded. Each call of the module writes its results
into those same tensors and returns them, so a call overwrites the results of
the call before it. To keep results across calls, copy them, e.g. with
.clone().