And even Barra’s significant rise to CEO of GM comes with a caveat, notes Heidi Hartmann, the president of the Institute for Women’s Policy Research, a Washington think tank: Unlike her predecessor, Dan Akerson, Barra won’t be chairman of the company’s board, though she will get a seat.

“It’s kind of like Chinese water torture, drip by drip,” Hartmann said. “I certainly hope [Barra’s appointment] moves the needle for getting more women in CEO positions, but you have to admit that there aren’t very many.”

When Barra takes over on Jan. 15, just 23 of the Fortune 500 companies will have female CEOs. That means businesses need to do more to help women advance through the pipeline, like ensuring they have opportunities to take on visible roles that could one day land them in a CEO or executive spot, according to Deborah Gillis, the chief operating officer of Catalyst.

“Her appointment is so important because we’re celebrating another first and we’re delighted to see her moving into this role, but we’ll be more delighted when we stop counting firsts,” she said.