TorchScript

• TorchScript Language Reference

TorchScript is a way to create serializable and optimizable models from PyTorch code. Any TorchScript program can be saved from a Python process and loaded in a process where there is no Python dependency.

We provide tools to incrementally transition a model from a pure Python program to a TorchScript program that can be run independently from Python, such as in a standalone C++ program. This makes it possible to train models in PyTorch using familiar tools in Python and then export the model via TorchScript to a production environment where Python programs may be disadvantageous for performance and multi-threading reasons.

For a gentle introduction to TorchScript, see theIntroduction to TorchScripttutorial.

For an end-to-end example of converting a PyTorch model to TorchScript and running it in C++, see the Loading a PyTorch Model in C++tutorial.

Creating TorchScript Code

script	Script the function.
trace	Trace a function and return an executable orScriptFunctionthat will be optimized using just-in-time compilation.
script_if_tracing	Compilesfnwhen it is first called during tracing.
trace_module	Trace a module and return an executableScriptModulethat will be optimized using just-in-time compilation.
fork	Create an asynchronous task executingfuncand a reference to the value of the result of this execution.
wait	Force completion of atorch.jit.Future[T]asynchronous task, returning the result of the task.
ScriptModule	Wrapper for C++ torch::jit::Module with methods, attributes, and parameters.
ScriptFunction	Functionally equivalent to aScriptModule,but represents a single function and does not have any attributes or Parameters.
freeze	Freeze ScriptModule, inline submodules, and attributes as constants.
optimize_for_inference	Perform a set of optimization passes to optimize a model for the purposes of inference.
enable_onednn_fusion	Enable or disables onednn JIT fusion based on the parameterenabled.
onednn_fusion_enabled	Return whether onednn JIT fusion is enabled.
set_fusion_strategy	Set the type and number of specializations that can occur during fusion.
strict_fusion	Give errors if not all nodes have been fused in inference, or symbolically differentiated in training.
save	Save an offline version of this module for use in a separate process.
load	Load aScriptModuleorScriptFunctionpreviously saved withtorch.jit.save.
ignore	This decorator indicates to the compiler that a function or method should be ignored and left as a Python function.
unused	This decorator indicates to the compiler that a function or method should be ignored and replaced with the raising of an exception.
interface	Decorate to annotate classes or modules of different types.
isinstance	Provide container type refinement in TorchScript.
Attribute	This method is a pass-through function that returnsvalue,mostly used to indicate to the TorchScript compiler that the left-hand side expression is a class instance attribute with type oftype.
annotate	Use to give type ofthe_valuein TorchScript compiler.

Mixing Tracing and Scripting

In many cases either tracing or scripting is an easier approach for converting a model to TorchScript. Tracing and scripting can be composed to suit the particular requirements of a part of a model.

Scripted functions can call traced functions. This is particularly useful when you need to use control-flow around a simple feed-forward model. For instance the beam search of a sequence to sequence model will typically be written in script but can call an encoder module generated using tracing.

Example (calling a traced function in script):

importtorch

deffoo(x,y):
return2*x+y

traced_foo=torch.jit.trace(foo,(torch.rand(3),torch.rand(3)))

@torch.jit.script
defbar(x):
returntraced_foo(x,x)

Traced functions can call script functions. This is useful when a small part of a model requires some control-flow even though most of the model is just a feed-forward network. Control-flow inside of a script function called by a traced function is preserved correctly.

Example (calling a script function in a traced function):

importtorch

@torch.jit.script
deffoo(x,y):
ifx.max()>y.max():
r=x
else:
r=y
returnr


defbar(x,y,z):
returnfoo(x,y)+z

traced_bar=torch.jit.trace(bar,(torch.rand(3),torch.rand(3),torch.rand(3)))

This composition also works fornn.Modules as well, where it can be used to generate a submodule using tracing that can be called from the methods of a script module.

Example (using a traced module):

importtorch
importtorchvision

classMyScriptModule(torch.nn.Module):
def__init__(self):
super().__init__()
self.means=torch.nn.Parameter(torch.tensor([103.939,116.779,123.68])
.resize_(1,3,1,1))
self.resnet=torch.jit.trace(torchvision.models.resnet18(),
torch.rand(1,3,224,224))

defforward(self,input):
returnself.resnet(input-self.means)

my_script_module=torch.jit.script(MyScriptModule())

TorchScript Language

TorchScript is a statically typed subset of Python, so many Python features apply directly to TorchScript. See the fullTorchScript Language Referencefor details.

Built-in Functions and Modules

TorchScript supports the use of most PyTorch functions and many Python built-ins. SeeTorchScript Builtinsfor a full reference of supported functions.

PyTorch Functions and Modules

TorchScript supports a subset of the tensor and neural network functions that PyTorch provides. Most methods on Tensor as well as functions in thetorchnamespace, all functions intorch.nn.functionaland most modules fromtorch.nnare supported in TorchScript.

SeeTorchScript Unsupported PyTorch Constructsfor a list of unsupported PyTorch functions and modules.

Python Functions and Modules

Many of Python’sbuilt-in functionsare supported in TorchScript. Themathmodule is also supported (seemath Modulefor details), but no other Python modules (built-in or third party) are supported.

Python Language Reference Comparison

For a full listing of supported Python features, seePython Language Reference Coverage.

Debugging

Disable JIT for Debugging

PYTORCH_JIT

Setting the environment variablePYTORCH_JIT=0will disable all script and tracing annotations. If there is hard-to-debug error in one of your TorchScript models, you can use this flag to force everything to run using native Python. Since TorchScript (scripting and tracing) is disabled with this flag, you can use tools likepdbto debug the model code. For example:

@torch.jit.script
defscripted_fn(x:torch.Tensor):
foriinrange(12):
x=x+x
returnx

deffn(x):
x=torch.neg(x)
importpdb;pdb.set_trace()
returnscripted_fn(x)

traced_fn=torch.jit.trace(fn,(torch.rand(4,5),))
traced_fn(torch.rand(3,4))

Debugging this script withpdbworks except for when we invoke the @torch.jit.scriptfunction. We can globally disable JIT, so that we can call the@torch.jit.script function as a normal Python function and not compile it. If the above script is calleddisable_jit_example.py,we can invoke it like so:

$ PYTORCH_JIT=0 python disable_jit_example.py

and we will be able to step into the@torch.jit.scriptfunction as a normal Python function. To disable the TorchScript compiler for a specific function, see @torch.jit.ignore.

Inspecting Code

TorchScript provides a code pretty-printer for allScriptModuleinstances. This pretty-printer gives an interpretation of the script method’s code as valid Python syntax. For example:

@torch.jit.script
deffoo(len):
# type: (int) -> torch.Tensor
rv=torch.zeros(3,4)
foriinrange(len):
ifi<10:
rv=rv-1.0
else:
rv=rv+1.0
returnrv

print(foo.code)

AScriptModulewith a singleforwardmethod will have an attribute code,which you can use to inspect theScriptModule’s code. If theScriptModulehas more than one method, you will need to access .codeon the method itself and not the module. We can inspect the code of a method namedfooon aScriptModuleby accessing.foo.code. The example above produces this output:

deffoo(len:int)->Tensor:
rv=torch.zeros([3,4],dtype=None,layout=None,device=None,pin_memory=None)
rv0=rv
foriinrange(len):
iftorch.lt(i,10):
rv1=torch.sub(rv0,1.,1)
else:
rv1=torch.add(rv0,1.,1)
rv0=rv1
returnrv0

This is TorchScript’s compilation of the code for theforwardmethod. You can use this to ensure TorchScript (tracing or scripting) has captured your model code correctly.

Interpreting Graphs

TorchScript also has a representation at a lower level than the code pretty- printer, in the form of IR graphs.

TorchScript uses a static single assignment (SSA) intermediate representation (IR) to represent computation. The instructions in this format consist of ATen (the C++ backend of PyTorch) operators and other primitive operators, including control flow operators for loops and conditionals. As an example:

@torch.jit.script
deffoo(len):
# type: (int) -> torch.Tensor
rv=torch.zeros(3,4)
foriinrange(len):
ifi<10:
rv=rv-1.0
else:
rv=rv+1.0
returnrv

print(foo.graph)

graphfollows the same rules described in theInspecting Codesection with regard toforwardmethod lookup.

The example script above produces the graph:

graph(%len.1: int):
%24: int = prim::Constant[value=1]()
%17: bool = prim::Constant[value=1]() # test.py:10:5
%12: bool? = prim::Constant()
%10: Device? = prim::Constant()
%6: int? = prim::Constant()
%1: int = prim::Constant[value=3]() # test.py:9:22
%2: int = prim::Constant[value=4]() # test.py:9:25
%20: int = prim::Constant[value=10]() # test.py:11:16
%23: float = prim::Constant[value=1]() # test.py:12:23
%4: int[] = prim::ListConstruct(%1, %2)
%rv.1: Tensor = aten::zeros(%4, %6, %6, %10, %12) # test.py:9:10
%rv: Tensor = prim::Loop(%len.1, %17, %rv.1) # test.py:10:5
block0(%i.1: int, %rv.14: Tensor):
%21: bool = aten::lt(%i.1, %20) # test.py:11:12
%rv.13: Tensor = prim::If(%21) # test.py:11:9
block0():
%rv.3: Tensor = aten::sub(%rv.14, %23, %24) # test.py:12:18
-> (%rv.3)
block1():
%rv.6: Tensor = aten::add(%rv.14, %23, %24) # test.py:14:18
-> (%rv.6)
-> (%17, %rv.13)
return (%rv)

Take the instruction%rv.1:Tensor=aten::zeros(%4,%6,%6,%10,%12)#test.py:9:10for example.

• %rv.1:Tensormeans we assign the output to a (unique) value namedrv.1,that value is ofTensortype and that we do not know its concrete shape.

• aten::zerosis the operator (equivalent totorch.zeros) and the input list(%4,%6,%6,%10,%12)specifies which values in scope should be passed as inputs. The schema for built-in functions likeaten::zeroscan be found atBuiltin Functions.

• #test.py:9:10is the location in the original source file that generated this instruction. In this case, it is a file namedtest.py,on line 9, and at character 10.

Notice that operators can also have associatedblocks,namely the prim::Loopandprim::Ifoperators. In the graph print-out, these operators are formatted to reflect their equivalent source code forms to facilitate easy debugging.

Graphs can be inspected as shown to confirm that the computation described by aScriptModuleis correct, in both automated and manual fashion, as described below.

Tracer

Tracing Edge Cases

There are some edge cases that exist where the trace of a given Python function/module will not be representative of the underlying code. These cases can include:

• Tracing of control flow that is dependent on inputs (e.g. tensor shapes)

• Tracing of in-place operations of tensor views (e.g. indexing on the left-hand side of an assignment)

Note that these cases may in fact be traceable in the future.

Automatic Trace Checking

One way to automatically catch many errors in traces is by usingcheck_inputs on thetorch.jit.trace()API.check_inputstakes a list of tuples of inputs that will be used to re-trace the computation and verify the results. For example:

defloop_in_traced_fn(x):
result=x[0]
foriinrange(x.size(0)):
result=result*x[i]
returnresult

inputs=(torch.rand(3,4,5),)
check_inputs=[(torch.rand(4,5,6),),(torch.rand(2,3,4),)]

traced=torch.jit.trace(loop_in_traced_fn,inputs,check_inputs=check_inputs)

Gives us the following diagnostic information:

ERROR: Graphs differed across invocations!
Graph diff:

graph(%x: Tensor) {
%1: int = prim::Constant[value=0]()
%2: int = prim::Constant[value=0]()
%result.1: Tensor = aten::select(%x, %1, %2)
%4: int = prim::Constant[value=0]()
%5: int = prim::Constant[value=0]()
%6: Tensor = aten::select(%x, %4, %5)
%result.2: Tensor = aten::mul(%result.1, %6)
%8: int = prim::Constant[value=0]()
%9: int = prim::Constant[value=1]()
%10: Tensor = aten::select(%x, %8, %9)
- %result: Tensor = aten::mul(%result.2, %10)
+ %result.3: Tensor = aten::mul(%result.2, %10)
?++
%12: int = prim::Constant[value=0]()
%13: int = prim::Constant[value=2]()
%14: Tensor = aten::select(%x, %12, %13)
+ %result: Tensor = aten::mul(%result.3, %14)
+ %16: int = prim::Constant[value=0]()
+ %17: int = prim::Constant[value=3]()
+ %18: Tensor = aten::select(%x, %16, %17)
- %15: Tensor = aten::mul(%result, %14)
?^ ^
+ %19: Tensor = aten::mul(%result, %18)
?^ ^
- return (%15);
?^
+ return (%19);
?^
}

This message indicates to us that the computation differed between when we first traced it and when we traced it with thecheck_inputs.Indeed, the loop within the body ofloop_in_traced_fndepends on the shape of the inputx,and thus when we try anotherxwith a different shape, the trace differs.

In this case, data-dependent control flow like this can be captured using torch.jit.script()instead:

deffn(x):
result=x[0]
foriinrange(x.size(0)):
result=result*x[i]
returnresult

inputs=(torch.rand(3,4,5),)
check_inputs=[(torch.rand(4,5,6),),(torch.rand(2,3,4),)]

scripted_fn=torch.jit.script(fn)
print(scripted_fn.graph)
#print(str(scripted_fn.graph).strip())

forinput_tuplein[inputs]+check_inputs:
torch.testing.assert_close(fn(*input_tuple),scripted_fn(*input_tuple))

Which produces:

graph(%x:Tensor){
%5:bool=prim::Constant[value=1]()
%1:int=prim::Constant[value=0]()
%result.1:Tensor=aten::select(%x,%1,%1)
%4:int=aten::size(%x,%1)
%result:Tensor=prim::Loop(%4,%5,%result.1)
block0(%i:int,%7:Tensor){
%10:Tensor=aten::select(%x,%1,%i)
%result.2:Tensor=aten::mul(%7,%10)
->(%5,%result.2)
}
return(%result);
}

Tracer Warnings

The tracer produces warnings for several problematic patterns in traced computation. As an example, take a trace of a function that contains an in-place assignment on a slice (a view) of a Tensor:

deffill_row_zero(x):
x[0]=torch.rand(*x.shape[1:2])
returnx

traced=torch.jit.trace(fill_row_zero,(torch.rand(3,4),))
print(traced.graph)

Produces several warnings and a graph which simply returns the input:

fill_row_zero.py:4: TracerWarning: There are 2 live references to the data region being modified when tracing in-place operator copy_ (possibly due to an assignment). This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.
x[0] = torch.rand(*x.shape[1:2])
fill_row_zero.py:6: TracerWarning: Output nr 1. of the traced function does not match the corresponding output of the Python function. Detailed error:
Not within tolerance rtol=1e-05 atol=1e-05 at input[0, 1] (0.09115803241729736 vs. 0.6782537698745728) and 3 other locations (33.00%)
traced = torch.jit.trace(fill_row_zero, (torch.rand(3, 4),))
graph(%0: Float(3, 4)) {
return (%0);
}

We can fix this by modifying the code to not use the in-place update, but rather build up the result tensor out-of-place withtorch.cat:

deffill_row_zero(x):
x=torch.cat((torch.rand(1,*x.shape[1:2]),x[1:2]),dim=0)
returnx

traced=torch.jit.trace(fill_row_zero,(torch.rand(3,4),))
print(traced.graph)

Frequently Asked Questions

Q: I would like to train a model on GPU and do inference on CPU. What are the best practices?

First convert your model from GPU to CPU and then save it, like so:

cpu_model=gpu_model.cpu()
sample_input_cpu=sample_input_gpu.cpu()
traced_cpu=torch.jit.trace(cpu_model,sample_input_cpu)
torch.jit.save(traced_cpu,"cpu.pt")

traced_gpu=torch.jit.trace(gpu_model,sample_input_gpu)
torch.jit.save(traced_gpu,"gpu.pt")

#... later, when using the model:

ifuse_gpu:
model=torch.jit.load("gpu.pt")
else:
model=torch.jit.load("cpu.pt")

model(input)

This is recommended because the tracer may witness tensor creation on a specific device, so casting an already-loaded model may have unexpected effects. Casting the modelbeforesaving it ensures that the tracer has the correct device information.

Q: How do I store attributes on aScriptModule?

Say we have a model like:

importtorch

classModel(torch.nn.Module):
def__init__(self):
super().__init__()
self.x=2

defforward(self):
returnself.x

m=torch.jit.script(Model())

IfModelis instantiated it will result in a compilation error since the compiler doesn’t know aboutx.There are 4 ways to inform the compiler of attributes onScriptModule:

1.nn.Parameter- Values wrapped innn.Parameterwill work as they do onnn.Modules

2.register_buffer- Values wrapped inregister_bufferwill work as they do onnn.Modules. This is equivalent to an attribute (see 4) of type Tensor.

3. Constants - Annotating a class member asFinal(or adding it to a list called __constants__at the class definition level) will mark the contained names as constants. Constants are saved directly in the code of the model. See builtin-constantsfor details.

4. Attributes - Values that are asupported typecan be added as mutable attributes. Most types can be inferred but some may need to be specified, see module attributesfor details.

Q: I would like to trace module’s method but I keep getting this error:

RuntimeError:CannotinsertaTensorthatrequiresgradasaconstant.Considermakingitaparameterorinput,ordetachingthegradient

This error usually means that the method you are tracing uses a module’s parameters and you are passing the module’s method instead of the module instance (e.g.my_module_instance.forwardvsmy_module_instance).

• Invokingtracewith a module’s method captures module parameters (which may require gradients) asconstants.

• On the other hand, invokingtracewith module’s instance (e.g.my_module) creates a new module and correctly copies parameters into the new module, so they can accumulate gradients if required.

To trace a specific method on a module, seetorch.jit.trace_module

Known Issues

If you’re usingSequentialwith TorchScript, the inputs of some of theSequentialsubmodules may be falsely inferred to be Tensor,even if they’re annotated otherwise. The canonical solution is to subclassnn.Sequentialand redeclareforward with the input typed correctly.

Appendix

Migrating to PyTorch 1.2 Recursive Scripting API

This section details the changes to TorchScript in PyTorch 1.2. If you are new to TorchScript you can skip this section. There are two main changes to the TorchScript API with PyTorch 1.2.

1.torch.jit.scriptwill now attempt to recursively compile functions, methods, and classes that it encounters. Once you calltorch.jit.script, compilation is “opt-out”, rather than “opt-in”.

2.torch.jit.script(nn_module_instance)is now the preferred way to create ScriptModules, instead of inheriting fromtorch.jit.ScriptModule. These changes combine to provide a simpler, easier-to-use API for converting yournn.Modules intoScriptModules, ready to be optimized and executed in a non-Python environment.

The new usage looks like this:

importtorch
importtorch.nnasnn
importtorch.nn.functionalasF

classModel(nn.Module):
def__init__(self):
super().__init__()
self.conv1=nn.Conv2d(1,20,5)
self.conv2=nn.Conv2d(20,20,5)

defforward(self,x):
x=F.relu(self.conv1(x))
returnF.relu(self.conv2(x))

my_model=Model()
my_scripted_model=torch.jit.script(my_model)

• The module’sforwardis compiled by default. Methods called fromforwardare lazily compiled in the order they are used inforward.

• To compile a method other thanforwardthat is not called fromforward,add@torch.jit.export.

• To stop the compiler from compiling a method, add@torch.jit.ignoreor@torch.jit.unused.@ignoreleaves the

• method as a call to python, and@unusedreplaces it with an exception.@ignoredcannot be exported;@unusedcan.

• Most attribute types can be inferred, sotorch.jit.Attributeis not necessary. For empty container types, annotate their types usingPEP 526-styleclass annotations.

• Constants can be marked with aFinalclass annotation instead of adding the name of the member to__constants__.

• Python 3 type hints can be used in place oftorch.jit.annotate

As a result of these changes, the following items are considered deprecated and should not appear in new code:

• The@torch.jit.script_methoddecorator

• Classes that inherit fromtorch.jit.ScriptModule

• Thetorch.jit.Attributewrapper class

• The__constants__array

• Thetorch.jit.annotatefunction

Modules

Warning

The@torch.jit.ignoreannotation’s behavior changes in PyTorch 1.2. Before PyTorch 1.2 the @ignore decorator was used to make a function or method callable from code that is exported. To get this functionality back, use@torch.jit.unused().@torch.jit.ignoreis now equivalent to@torch.jit.ignore(drop=False).See@torch.jit.ignore and@torch.jit.unusedfor details.

When passed to thetorch.jit.scriptfunction, atorch.nn.Module's data is copied to aScriptModuleand the TorchScript compiler compiles the module. The module’sforwardis compiled by default. Methods called fromforwardare lazily compiled in the order they are used inforward,as well as any @torch.jit.exportmethods.

torch.jit.export(fn)

This decorator indicates that a method on annn.Moduleis used as an entry point into a ScriptModuleand should be compiled.

forwardimplicitly is assumed to be an entry point, so it does not need this decorator. Functions and methods called fromforwardare compiled as they are seen by the compiler, so they do not need this decorator either.

Example (using@torch.jit.exporton a method):

importtorch
importtorch.nnasnn

classMyModule(nn.Module):
defimplicitly_compiled_method(self,x):
returnx+99

# `forward` is implicitly decorated with `@torch.jit.export`,
# so adding it here would have no effect
defforward(self,x):
returnx+10

@torch.jit.export
defanother_forward(self,x):
# When the compiler sees this call, it will compile
# `implicitly_compiled_method`
returnself.implicitly_compiled_method(x)

defunused_method(self,x):
returnx-20

# `m` will contain compiled methods:
# `forward`
# `another_forward`
# `implicitly_compiled_method`
# `unused_method` will not be compiled since it was not called from
# any compiled methods and wasn't decorated with `@torch.jit.export`
m=torch.jit.script(MyModule())

Functions

Functions don’t change much, they can be decorated with@torch.jit.ignoreortorch.jit.unusedif needed.

# Same behavior as pre-PyTorch 1.2
@torch.jit.script
defsome_fn():
return2

# Marks a function as ignored, if nothing
# ever calls it then this has no effect
@torch.jit.ignore
defsome_fn2():
return2

# As with ignore, if nothing calls it then it has no effect.
# If it is called in script it is replaced with an exception.
@torch.jit.unused
defsome_fn3():
importpdb;pdb.set_trace()
return4

# Doesn't do anything, this function is already
# the main entry point
@torch.jit.export
defsome_fn4():
return2

TorchScript Classes

Warning

TorchScript class support is experimental. Currently it is best suited for simple record-like types (think aNamedTuplewith methods attached).

Everything in a user definedTorchScript Classis exported by default, functions can be decorated with@torch.jit.ignoreif needed.

Attributes

The TorchScript compiler needs to know the types ofmodule attributes.Most types can be inferred from the value of the member. Empty lists and dicts cannot have their types inferred and must have their types annotated withPEP 526-styleclass annotations. If a type cannot be inferred and is not explicitly annotated, it will not be added as an attribute to the resultingScriptModule

Old API:

fromtypingimportDict
importtorch

classMyModule(torch.jit.ScriptModule):
def__init__(self):
super().__init__()
self.my_dict=torch.jit.Attribute({},Dict[str,int])
self.my_int=torch.jit.Attribute(20,int)

m=MyModule()

New API:

fromtypingimportDict

classMyModule(torch.nn.Module):
my_dict:Dict[str,int]

def__init__(self):
super().__init__()
# This type cannot be inferred and must be specified
self.my_dict={}

# The attribute type here is inferred to be `int`
self.my_int=20

defforward(self):
pass

m=torch.jit.script(MyModule())

Constants

TheFinaltype constructor can be used to mark members asconstant.If members are not marked constant, they will be copied to the resultingScriptModuleas an attribute. UsingFinalopens opportunities for optimization if the value is known to be fixed and gives additional type safety.

Old API:

classMyModule(torch.jit.ScriptModule):
__constants__=['my_constant']

def__init__(self):
super().__init__()
self.my_constant=2

defforward(self):
pass
m=MyModule()

New API:

fromtypingimportFinal

classMyModule(torch.nn.Module):

my_constant:Final[int]

def__init__(self):
super().__init__()
self.my_constant=2

defforward(self):
pass

m=torch.jit.script(MyModule())

Variables

Containers are assumed to have typeTensorand be non-optional (see Default Typesfor more information). Previously,torch.jit.annotatewas used to tell the TorchScript compiler what the type should be. Python 3 style type hints are now supported.

importtorch
fromtypingimportDict,Optional

@torch.jit.script
defmake_dict(flag:bool):
x:Dict[str,int]={}
x['hi']=2
b:Optional[int]=None
ifflag:
b=2
returnx,b

Fusion Backends

There are a couple of fusion backends available to optimize TorchScript execution. The default fuser on CPUs is NNC, which can perform fusions for both CPUs and GPUs. The default fuser on GPUs is NVFuser, which supports a wider range of operators and has demonstrated generated kernels with improved throughput. See theNVFuser documentationfor more details on usage and debugging.

References

• Python Language Reference Coverage
• TorchScript Unsupported PyTorch Constructs