As big bands went out of vogue toward the end of the ’40s, Baker retired from the road and married Charleston tennis star and restauranteur Delaney “Wag” Wagner, who owned Wag’s Barbecue, a downtown hot spot. While in Charleston, Baker continued to perform, backed by local combos like The Earl Tate Trio, The Francis Taylor Trio, and the Billy J Trio, which featured Hubert “Rabbit” Jones on bass. The couple also started a West Side club called The Shalamar. There, Baker sat in with a succession of big names from Nat King Cole to Hampton to pianist Errol Garner. Before integration, the club attracted lawyers, doctors, politicians - clientele from among the elite of Charleston. Baker was a regular at the old West Virginia Jazz Festival, and later had a jazz series at Charleston’s Multifest named in her honor.