Among the gymnasts Béla and Marta Károlyi have trained are Nadia Comăneci (first 10), Mary Lou Retton, Betty Okino, Kerri Strug, Teodora Ungureanu, Kim Zmeskal, Kristie Phillips and Dominique Moceanu. In total, Károlyi has coached nine Olympic champions, fifteen world champions, sixteen European medalists and six U.S. national champions.

Károlyi pioneered the Romanian centralized gymnastics training system in Romania in the late 1960s and early 1970s. He worked as a coach at the boarding school in Gheorghe Gheorghiu-Dej (now Oneşti), training young girls especially chosen for their athletic potential. One of the first students at the school was six year old Nadia Comăneci, who lived near the town and commuted from home.

Károlyi debuted as an international coach in 1974. He had to persuade the Romanian gymnastics federation to have Comăneci and his other athletes named to the 1975 European Championships and the 1976 Olympic team, as the Federation favored athletes from the competing Dinamo club in Bucharest. At the 1976 Summer Olympics in Montreal, he was Head Coach of the Romanian squad, and most of the members of the team were Gheorghe Gheorghiu-Dej athletes. The team took the silver medal and Comăneci was one of the outstanding performers of the Games, scoring the first-ever perfect 10 in Olympic competition. Altogether the Romanians won seven medals in Montreal: three gold, two silver and two bronze.

Following Comăneci's astounding success in Montreal, Károlyi's importance as a coach was recognized. He was named Head Coach of the Romanian team at the 1980 Olympics. Károlyi came under fire from Romanian officials due to his score protests at several international meets, including the 1980 Olympics. After the Olympics, Károlyi again clashed with Romanian Federation officials, and tension escalated. During a 1981 gymnastics tour, Béla, Márta, and Romanian team choreographer Géza Pozsár defected and sought political asylum in the United States. They settled in Oklahoma.

In 1981, a group of businessmen invited Karolyi to join a gymnastics business venture. He decided to invest in the business and the Károlyis relocated to Houston, Texas. The gym ran into financial problems, and Károlyi ended up buying the gym.

Károlyi's status as "Nadia's coach" quickly attracted gymnasts to his club. Only three years after his defection, Károlyi was back at the 1984 Olympics as the individual coach of all-around champion Mary Lou Retton and uneven bars gold medalist Julianne McNamara. Károlyi's clout in America grew after 1984, and by the time of the 1988 Olympics, he was influential enough to be made head coach of the women's Olympic gymnastics team. When Károlyi's status as the 1988 Olympic coach was jeopardized by the fact that he had not yet fulfilled the five-year residency requirement to become a U.S. citizen, two U.S. senators sponsored a special bill to waive the waiting period and grant him early citizenship. Károlyi was also the personal coach of three athletes on the squad: balance beam bronze medalist Phoebe Mills, the only female U.S. gymnast to medal in Seoul; Chelle Stack and Brandy Johnson.

After the 1988 Olympics, Károlyi's sphere of influence continued to grow, as did the number of elites training in his gym. At one meet in 1990, a journalist dubbed six top Károlyi gymnasts the "Karolyi six-pack." Although the members of the six-pack would change, the name stuck and increased Károlyi's prominence in the sport.

At the 1991 World Championships, four of the six athletes on the U.S. women's team—Kim Zmeskal, Betty Okino, Hilary Grivich and Kerri Strug—were trained by Károlyi; the other two, Shannon Miller and Michelle Campi, were trained by ex-Károlyi club coaches. The situation was almost repeated at the 1992 Olympics, where Károlyi was head coach and five members of the seven-gymnast squad (six competitors; one alternate) were either trained by him or one of his proteges.

Károlyi mostly acted as a personal coach for his athletes Dominique Moceanu and Kerri Strug at the 1996 Olympics, but still managed to draw the spotlight. After Strug was injured during the U.S. team's final rotation on vault, Károlyi carried her to the podium to accept her gold medal. The moment was photographed and widely distributed, and became what was arguably among the most enduring memories of the 1996 Olympics.

Károlyi retired from coaching after the 1996 Olympics. He and Márta still have a ranch and gymnastics camp in New Waverly, Texas. The following year, in 1997, Béla Károlyi was inducted into the International Gymnastics Hall of Fame.