Have you checked out Xournal++ already?

I use Rnote to write freely on a big whiteboard and Xournal++ for taking notes over PDFs of lectures.

Rnote has a much more modern and user-friendly interface while Xournal++ has more features and has been around for longer. I use both where I feel each excels.

Gilles Castel (RIP) published a couple articles on taking math notes with Vim + LaTeX which I found eye-opening. In another article he posted about how he integrates Inkscape into the process. If you haven’t read the articles, they may be helpful for what you are doing.

Xournal++ should work well

If you use org mode, you can use this https://github.com/misohena/el-easydraw. Its a basic drawing tool in emacs.