There would be a lot of people who don’t want to be in government, like jury duty, so there would need to be a process for entering the lottery so that only interested parties get drawn.
After that, I still have this problem with the idea: even if everyone who was selected was vetted by experts to ensure soundness of mind and stuff like that, those people would then be able to enact whatever policy they want. Voting allows us to decide if we agree with someone’s policy or not, before they enact it. In this scenario we wouldn’t have that ability. Your idea then to have experienced civil servants decide if someone needs replacing is essentially creating an autocracy; they’d be the ones with all the power. Voting gives the people the power, random selection gives individuals the power, and random selection with a selection committee gives the committee all the power. So for me, voting is still the safest and fairest option.