How can we use digital identity for authentication in the physical world without compromising user privacy? This central question is an underlying concern for further developments in ubiquitous computing scenarios: enabling individuals to – for example – use public transport and other payment/ticketing applications, access physical doors, access computing resources on public terminals, or even cross country borders without carrying any form of physical identity document or trusted mobile device. Moving towards such a device-free infrastructure-based authentication could be easily facilitated by centralized databases with full biometric records of all individuals, authenticating and therefore tracking people in all their interactions in the digital and physical worlds. However, such centralized tracking is not compatible with fundamental human rights to data privacy. One option to gain the utility of such digital authentication without sacrificing privacy rights is a fully decentralized approach to digital user authentication in the physical world. An ensemble of biometric sensors, different verifiers, and decentralized personal identity agents gives each individual better control over their digital and physical world interactions and data traces they leave.