Rome fell in 476 AD and took the majority of its culture with it. However, the Eastern half of the Roman Empire remained, carrying on for another thousand years and becoming its own entity, the Byzantine Empire.
Fighting as a sport did mostly decline without its Roman center, though, plus, the constant state of war in Europe made it so that there was simply no time for fighting sports like boxing or wrestling. There was always the next war, and soldiers were needed, not athletes. Further, the Byzantines had their own view and culture, distinct from their predecessors.
The Ancient Greeks viewed the events of boxing, wrestling, and pankration as the celebration of athleticism, treating them as symbolic of strength and skill. The Ancient Romans saw them more like a spectacle, enjoying violence as entertainment. Though the Greek sports were part of the Roman events, their most famous and favored spectacle was gladiatorial combat.
None of these had the same prominence in the Byzantine view. The importance of athletics was to produce good and quick soldiers, for the Byzantine Empire was a warring culture, not a sporting one. Further, with Christianity in the picture, the fighting sports were viewed as synonymous with paganism and were therefore not tolerated.
It was Christianity that kept fighting sports down. Saint Augustine of Hippo specifically pointed to the gladiatorial combat in the Coliseum as the height of pagan excess. And with enemies on the borders, the Byzantines could not waste time or energy with petty games when there were wars to fight.
With warfare becoming a way of life, it didn’t take long for it to get formalized. Emperor Leo VI wrote the Tactica, which served as the Empire’s military manual, and was the first among the multitude of martial arts texts that would arise later in the Middle Ages. Eventually, the Empire fell, but by then the military outlook had shifted back to individual skill and prowess, thanks to the pugilistic spirit of Ancient Rome coming together with the skill of the Byzantine military. The manuals moved with the times, seeking to guide the individual in his achievement of martial mastery.
What do you know of combat as it was practiced in the Middle Ages? Have you ever read through a Medieval fight manuals?