I disagree (shocker!) that its different to grant a right versus removing it. If we believe as a society that we can appropriately judge competence using age as the criteria, then that should extend to both ends of the spectrum. Revoking licenses and "rights" should 100% be on the table if people cannot sufficiently maintain competence. People lose eyesight as they age. People lose cognitive ability as they age. People lose mobility as they age. People lose motivation as they age. Its part of being human, and I think society should discuss the possibility of removing certain privileges as people get older.
A great example is my grandparents continuing to drive into their 80's and even 90's, and they should NOT have been on the road at a certain point. They were outright, objectively dangerous behind the wheel. Its not because they were less of a human. Its because they couldn't hear, see, or judge sufficiently to make them competent drivers. Yet they insisted on driving, and they were a hazard to everyone else on the roads. I fully support society finding a way to protect others by respectfully finding a way to get those people off the road.
I'd support extending that many other arenas, including politics, based on age alone, just like we do for the lower end of the spectrum.