Thanks for asking!
There are several programs that we used so far. Initially, that was Procreate, but we changed to GoodNotes now because of the PDF annotation capabilities and some great whiteboarding tricks that it supports (like shape recognition) . I personally use Penultimate because I like the Evernote integration.
All three have support for Apple Pencil, so the experience is really very close to real-life drawing and writing (except the lines are much more straight with GoodNotes ).
If I remember correctly what originally gave @Jim_Meegan and I the idea for using an iPad for the whiteboarding in our learning videos came from reading this article here about how artist Kyle Lambert created the poster/art style for Stranger Things.
We used Procreate for a long time, however, once I started to use my own iPad Pro more and more, I discovered GoodNotes, which, I love, for all sorts of reasons. I can take notes easily, import graphics, annotate PDF's, and draw and scribble all to my heart's content.
We will have to check out Penultimate though, who knows what great features may lurk there!
What I really want to find is a whiteboarding/sketching/annotating app that makes it easy to work with icons. Any ideas out there?