Not necessarily - pavucontrol switched to GTK4, and there are a lot of other applications that I use that are on it as well. If XFCE stays on X11, I wouldn’t be able to run any application that updates to GTK5 (except through some hack like running Weston nested in X, which I used to do when I used Waydroid).
Debian is on the right track. XFCE might work - I remember it running pretty well on a laptop with 4 gigs.