Curated AI & Developer Learning Resources

Migrating from functorch to torch.func

PyTorch documentation e72dc1e80af126643b62a7e7f3b5940a9ede4d91_0

torch.func, previously known as “functorch”, is JAX-like composable function transforms for PyTorch. functorch started as an out-of-tree library over at the pytorch/functorch repository. Our goal has ...

UX Limitations

PyTorch documentation d1d449ac6cfcb231c5a5eec72d71f256877a3133_0

torch.func, like JAX , has restrictions around what can be transformed. In general, JAX’s limitations are that transforms only work with pure functions: that is, functions where the output is complete...

Patching Batch Norm

PyTorch documentation 43dedf09639a6877987113a7f4e87f1ab3810c6c_0

What’s happening? Batch Norm requires in-place updates to running_mean and running_var of the same size as the input. Functorch does not support inplace update to a regular tensor that takes in a batc...

torch.func.replace_all_batch_norm_modules_

PyTorch documentation 246cd1e3ce2f797bf04e7f4e4cfe9fd8aded578d_0

In place updates root by setting the running_mean and running_var to be None and setting track_running_stats to be False for any nn.BatchNorm module in root Module

torch.func.stack_module_state

PyTorch documentation f6b4dd8b7340feb2e03b438fd485722e25d0ee6f_0

Prepares a list of torch.nn.Modules for ensembling with vmap() . Given a list of M nn.Modules of the same class, returns two dictionaries that stack all of their parameters and buffers together, index...

POE0004:operator-supported-in-newer-opset-version

PyTorch documentation 196d8a930299182d21a440bb5546e027e52032d9_0

Operator is supported in newer opset version. Example:

torch.func.functional_call

PyTorch documentation 7441a15740e671fc7567a283d3e96d6dc0b839ed_0

Performs a functional call on the module by replacing the module parameters and buffers with the provided ones. Note If the module has active parametrizations, passing a value in the parameters_and_bu...

POE0003:missing-standard-symbolic-function

PyTorch documentation 40f23a1024d640264b7b09a5b114d140340ecb65_0

Missing symbolic function for standard PyTorch operator, cannot translate node to ONNX.

torch.func.functionalize

PyTorch documentation 6eae871f973e9da798eda5c41f3ad5d8129c3997_0

functionalize is a transform that can be used to remove (intermediate) mutations and aliasing from a function, while preserving the function’s semantics. functionalize(func) returns a new function wit...

POE0002:missing-custom-symbolic-function

PyTorch documentation cb89286fae3a798f206e5b26cdaa3fc30d28da61_0

Missing symbolic function for custom PyTorch operator, cannot translate node to ONNX.

torch.signal.windows.nuttall

PyTorch documentation afda63101b5c8045e85c10027af8ccd378da7c83_0

Computes the minimum 4-term Blackman-Harris window according to Nuttall. where z_n = 2 π n/ M . The window is normalized to 1 (maximum value is 1). However, the 1 doesn’t appear if M is even and sym i...

torch.func.hessian

PyTorch documentation 79690a4e9a24303c9c5a4e470b04477b2126dc01_0

Computes the Hessian of func with respect to the arg(s) at index argnum via a forward-over-reverse strategy. The forward-over-reverse strategy (composing jacfwd(jacrev(func)) ) is a good default for g...