Yes, you can leave a pocket that is shared with you by tapping the "Leave" button inside the shared pocket at any time. But, please note that the power of attorney provided for the pocket will only be formally rejected after confirmation from the pocket-owner side.