I think they are fine together. [name_u]James[/name_u] is obv. more classic and not quirky like [name_m]Rafferty[/name_m], but they are both handsome and complement each other well imo. I find [name_u]James[/name_u] so versatile that it works with a lot of different styles.
[name_m]Rafferty[/name_m] [name_m]George[/name_m] is adorable!!
Well since [name_f]Esme[/name_f] is [name_m]French[/name_m], you already have some diversity in your sibset. I see no problem with [name_f]Esme[/name_f], [name_u]James[/name_u], and [name_m]Rafferty[/name_m]. They are all classy & stylish names. I would be envious of this sibset!