Choose who to go to
In your absence, you need to have someone in the workplace who can respond in your place if necessary.
Whether it's a co-worker, your assistant, or your boss, that person needs to know what you're doing on those days so they can act in your place.