Post navigation module within a sub category

Hi there. I’m attempting to use the post navigation module on a theme layout page for a few categories on my site. I have navigate within the same taxonomy enabled and set to “category.” However it doesn’t seem to be grabbing the correct posts within the category.

The categories I am trying to target with the themer layout are sub categories, but even when I set it to only one of those categories it goes to a random post within the parent category on both next and previous.

I cannot see a pattern as they aren’t even actually was is previous and next within the admin on the parent category so it seems like a couple things could be going wrong here.

Any help is much appreciated.