Rework forward pass to remove old gradients - #46

Open

Arkay92 wants to merge 13 commits into

openai:mainfrom

Arkay92:fix/residual-cross-attention-rework

Commits on Dec 27, 2022

Rework perceiver tensor logic
Arkay92
committed
d40d34d

Commits on Dec 29, 2022

Add apex lib to setup
Arkay92
committed
4830080
Switch to apex fused layer norm
Arkay92
committed
400535e
Tidy up repeating code and apex import
Arkay92
committed
032673c
Add apex gitlib to install requires
Arkay92
committed
80a2920
Add np_grad to every forward pass in percevier
Arkay92
committed
16b3492

Commits on Dec 30, 2022

Change to fusedlayernorm lib, fix forward pass
Arkay92
committed
bfa5020
Add back cuda check
Arkay92
committed
cbbf405
Revert to std layernorm
Arkay92
committed
816f343
Remove apex from setup and import
Arkay92
committed
9aff88e

Commits on Jan 2, 2023

Remove device to pass until figured out the logic
Arkay92
committed
fadbf06
Add back .to() with torch.device directly
Arkay92
committed
51a3f95
remove .to()
Arkay92
committed
1a7a7ea