Since the adoption of the M16, controversy has surrounded the platform. When it was first fielded, the military took shortcuts - such as changing the gunpowder and not chrome lining the barrel - that resulted in repeated and catastrophic failures. However, these were fixed, and by the time the M16A2 was released, the US military fielded the most modern and arguably the most reliable small arm on the planet.
Enter the M4A1. In 1994 the US Army adopted the M4 as its general use small arm. It was put to the test in the Middle East. Naturally, problems arose.
The M4 "cheats" its way to a lighter design by reduction of barrel length. Additionally, the gas tube (which operates the bolt, causing the carbine to chamber the next round automatically) is shortened. This creates a harsher environment for the bolt (the main moving part in the carbine). Subsequently, more failures were documented.
In other words, the "cheat" came at a cost. And the harsh environments of the Middle East exposed those costs and weaknesses.
So another controversy was born. This time, other gun manufacturers keep lobbying the US military, with factual claims that they can produce rifles and carbines that do not fail as often as an M4. In practicality, nobody argues the facts, as the M4, even when lubricated properly (something the Army has historically not been fond of doing), still failed once every 120 rounds tested. Most of the other manufacturer's firearms only failed approximately once every 250 rounds (this link - page 101).
What the other manufacturers don't want you to know is this: The M16A4 only suffered stoppages about once every 225 rounds - statistically indistinguishable from the HK and the SCAR.
Why change to a new platform? Why not just run it the way it was designed -as a rifle, with a rifle-length barrel (more accuracy, higher bullet velocity), with a rifle length gas system (more reliability, less recoil)???
Oh, wait... the US Marine Corps already does.