I don't know if I can help.
I also have a NOOK and am having a problem with a Library book, but I hasten to add that it is a different problem,... trouble in downloading.
However, how are you trying to "return" the book ?
When I return them early I do so from ADE, clicking on "Item Options" then "Return borrowed Item". This immediately "returns" it and it is removed from my Library account.
How are you doing this via your NOOK ?