I really like [name_u]Spencer[/name_u] on a girl. I think judging as “they really wanted a boy,” is, well, exactly that–super judgy. If someone is going to think that name is meant for a boy or think it is too fancy for a normal middle class family, it sounds like they are going to judge no matter what you do, so I wouldn’t even take those kind of things into account.
I don’t think most people would think twice about it. I certainly wouldn’t, and I don’t really like the boy names on girls thing. I think of [name_u]Spencer[/name_u] as a reasonably equal unisex name, as opposed to something like [name_m]Matthew[/name_m] or [name_u]James[/name_u] that is very distinctly male.