Currently, the way this is set up is to remove access for one Role, all other Roles need to be given access, by manually checking on each. If we had a bulk update option, it will be easier to simply remove the restricted users.
Thank you for submitting your idea. We understand the desire for it and will consider it for a future release.