The Apple Lightning connector was introduced on September 12, 2012 to replace the 30-pin dock connector on the iPhone 5. It went on to replace the 30-pin dock connector on all new Apple products including popular accessories like the Apple Pencil, Magic Keyboard, and Siri Remote.
More compact than the 30-pin dock connector, the Lightning connector can be inserted with either side facing up. But as far as Apple's customers are concerned, that is where Lightning's advantages end.
The 30-pin dock connector introduced new capabilities by extending the existing 30-pin layout and utilizing dedicated hardware on the device. The Lightning connector emulates these capabilities across Lightning's 8-pins by way of the device's on board CPU. Often this emulation requires including expensive integrated circuits inside the Lightning adapter itself. This is one reason why Lightning adapters often cost more than their 30-pin counterparts.
The other reason is the Apple MFi Program. The MFi Program is a licensing program that third-party manufacturers must join in order to produce Lightning compatible accessories. Failure to pay the MFi Program tax could prevent a manufacturer's Lightning accessories from working with Apple devices. The program is enforced by the use of DRM. But even after the DRM was hacked, Lightning accessories that do much more than USB are still complicated to produce.
Apple created Lightning to retain control of its products They did so from a position of power. Designing the next interface for the world's most popular phone, tablet, and digital music player. Unlike the iMac's early adoption of USB, no suitable alternative was available at the time. Standards driven USB 3.1 and the reversible Type-C connection were still over two years away. Apple needed a smaller, convenient, future-proof connector in time for the introduction of the new thinner iPhone. What better way to ensure that Apple's products have the features they need and at a schedule of Apple's choosing, than to design a extensible connector and control the delivery of its capabilities. Lightning is that connector.
I am not surprised Apple is removing the headphone jack from the iPhone 7 and replacing it with the existing Lightning connector. But I don't think the removal of the floppy drive from the iMac is a comparable analogy.
The headphone jack's popularity isn't in decline. It has not been superseded by technologies like the ZIP drive, magnetic optical, and rewritable CD. Nor is it a legacy port, kept on for compatibility. If anything its popularity has increased in recent years with the surge of mobile devices and the digital music revolution. If anything its capabilities have grown with the introduction of an on board microphone, and remote control.
Apple has two reasons to get rid of the headphone jack. One is to ensure even greater control over its platform by getting more third-party accessory manufactures to join the MFi Program. The other is to announce a new feature of the upcoming iPhone 7 that would not be possible if the headphone jack is included.
Either way the Lightning connector is here to stay.