I could have included those factors in the model, and then used only the content-related variables to rate quality.

Y'know, now that you mention it, I realize it didn't really occur to me to go that route, once I settled on the XGBoost approach. It's almost like I have an economist mode, which is focused on primarily linear models with a strong concern for endogeneity, omited variables, and the interpretation of the error term, and a machine-learning mode, which doesn't worry about any of those things.

I remember back when I first started working with industry ML folks, I wanted to include company fixed-effects in a model, and a computer-science trained ML guy got made at me and told me it was inappropriate. I tried explaining what the purpose was, but he couldn't really understand. They're not trained to think about endogeneity much, it seems.