Though OPE Prop seems like a very efficient solution for model training, there is a big drawback with it

Jumping straight to the point, OPE Prop is very efficient for the selected network types it describes. The challenge is that for every new activation and error function pair, additional formula deriviation and testing is required. This is quite slow and hard because it isn't a one fit solution to all problems like gradient descent, for example, is. Gradient descent or other commonly used algorithms have a single formula you always follow, for example, taking the derivative and subtracting it from the parameter. OPE Prop on the other side, relies on solving equations, which can be hard and slow to do. For every new layer there is, you'll have to solve new equations that can be quite big and apply them in code.

But all hard work pays off, and by solving unique equations each time by hand, your model can train faster and more efficient, which could pay off in the long run.

There are new ideas and concepts I am currently exploring by trying new algorithms in the Deep Learning training process, or completely re-structuring the modern neural network achitecture. I can't wait to share the working ideas with everyone!

As always, if you have any suggestions, new formulas or ideas - you can contact me on my portfolio website under the 'Let’ talk' section, or by email (kyryloshy@gmail.com). Thank you!

OPE Prop formulas on this website are licensed under the CC BY-SA 4.0 License. More details here