A new IBM Redbook*, ‘Accelerating Modernization with Agile Integration’ (AMAI) is a 650-page treatise on current thinking in software development. Unlike many Redbooks, AMAI is not closely coupled with an IBM technology. Instead it represents a comprehensive overview of many of the generic trends that are currently touted in the industry. We were particularly interested in AMAI in the context of OSDU, the Open Subsurface Data Universe, that appears to share many of the concepts exposed in the IBM oeuvre.
In the introduction, ‘agile integration’ is presented as key to digital transformation initiatives, enabled by ‘agile connectivity through APIs and architectures that interconnect digital solutions from cloud vendors, system developers, start-ups, and others’. The digital transformation sets out to ‘connected customer experiences across a network of applications that use data of all types’. Unfortunately (perhaps surprisingly given the many years of effort in data standardization) ‘bringing these processes and information sources together at the correct time and within the correct context has become increasingly complicated’. The introduction has it that new AI capabilities require API access to information sources ‘propagated in near real time by event streams’ and a ‘multitude of other mechanisms’. ‘At the heart of the digital economy is the basic need to connect disparate data’. AMAI is said to draw on learnings from hundreds of customer interactions and ‘takes note of the dramatic changes that affect the integration landscape’.
What is ‘agile integration’? AMAI has a stab at defining agile thus.
‘The pace of innovation in IT has changed dramatically. Iterations on requirements are in near real-time, prototypes are prepared in weeks or even days, and new mobile apps are made available in months. Application development techniques needed to keep pace by introducing new approaches such as microservices that enable teams to work more independently. Integration is maturing towards a more API-led approach where interfaces that are easy to use enable teams to rapidly share data and functions. However, there is much more subtlety to this change than is apparent. You must think differently about how you align the people and skillsets that relate to integration. You must consider how to ensure that integration components embrace new architectural tenets such as microservices and that they capitalize on the benefits of new infrastructure platforms such as containers’.
AMAI presents integration as ‘the secret weapon behind the great innovations of today’. New ideas ‘always require data and functions from other applications within the enterprise and often from other enterprises’. Agility is required to transform ideas into production quickly ‘so that new niches can be leveraged’. This requires ‘highly empowered and autonomous teams that can self-provision the integration capabilities that they need and interact efficiently with other teams’.
Agile is further divvied up into three aspects, ‘decentralized ownership’, a ‘delivery-focused architecture’ and, you guessed, a ‘cloud-native infrastructure’. Decentralization means a shift from yesterday’s service-oriented architectures that tended to result in ‘heavily centralized integration teams’ to ‘fine-grained integration deployment, enabling distributed ownership of the creation and maintenance of the integrations.’ Delivery-focus implies an ‘API-led integration strategy for connectivity between applications’, API management refines the SOA paradigm with ‘standards around how interfaces are shared between applications and between enterprises in an API economy’. A cloud-native infrastructure allows (encourages?) developers to ‘hand-off the burden of many of their previously proprietary mechanisms for cluster management, scaling, and availability to the cloud platform in which they are running’. Software must make the best use of orchestration capabilities such as Kubernetes and other cloud frameworks.
As Oil IT Journal has been reporting on the ‘basic need to connect disparate data’ for 25 years notably with earlier attempts at data integration such as business objects and the service bus. AMAI discusses the failings in such earlier work. ‘SOA turned out to be a more complex than just the implementation of an enterprise service bus (ESB)’, resulting in ‘heavy centralization [… of …] the resulting topology.’ AMAI provides other reasons for ESB failure, from funding development across different departments, maintaining different OS versions to support a service, and the use of integration specialists with poor knowledge of the applications, and the now frequently dissed ‘waterfall’ approach to development.
AMAI cites a major driver for agile as the development of the ‘API economy’ whereby services from third parties can be called upon over the wire and integrated into an application. APIs are said to ‘play a significant role in the disruption of industries’. AMAI devotes a chapter to the subject of API management.
AMAI stresses the use of ‘microservices’. These improve agility as they are small enough to be understood in isolation and changed independently. They provide scalability as resource usage can be tied to the business model. They also provide resilience as a change to a suitably decoupled microservice does not affect others at run time’. One might reasonably doubt the veracity of such claims. AIAM acknowledges that microservices are not the solution to every problem. ‘The approach is complex. In many cases, existing enterprise solutions can and should continue running with a more traditional architecture’. However, ‘for suitably selected new solutions or for pockets of modernization within existing systems, microservices can provide an order of magnitude increase in benefits that are hard to achieve any other way. It will be interesting to see what OSDU comes up with in this arbitration between the ‘traditional architecture’ and ‘microservices’ – for more on this see the editorial in this issue.
A chapter on ‘capability perspective’ covers API management, application integration, messaging and business-to-business B2B exchange and hybrid and multi cloud considerations. AMAI gets into its swing here, discussing topics such as stateless components (Kubernetes), image-based deployments (Docker) and elastic, agnostic infrastructure, and container orchestration platforms. Unfortunately, here AMAI has changed gear from a rather verbose explanation of relatively well-trodden ground to a hasty exposition that is harder to follow.
The B2B section is better developed and should be of interest working in the PIDX e-business data exchange space. Here ‘Traditional B2B and EDI patterns were focused around supporting messages for purchasing and supply chains in a batch mode’. APIs better enable real-time data exchanges between applications and business partners.
AMAI includes a short section on hybrid and multi cloud deployment. Users ‘want to use the strength and unique offerings from different cloud vendors, but also want to have a consistent operation and runtime environment so that they can achieve portability without cloud platform lock-in’. But here again, coverage is limited to an enumeration of some overriding issues with little on the thorny issue of avoiding lock-in.
Other chapters cover cloud-native concepts and technology, practical agile integration and ‘notes from the field’ on various topics, one of which, ‘field notes on modernizing messaging’ focuses on the IBM MQ environment and may be of interest to scada developers.
On the subject of messaging, one topic that intrigued us earlier in AMAI is the event stream. Messaging is seeing ‘renewed importance’ as enterprises need ‘robust, secure, and reliable ways to move data asynchronously’. This may be one of the most appropriate fields for agile development as messages pass from one company system to another, and where scalability is required to handle a sudden increase in traffic. Here Apache Kafka is the ‘de facto standard’. The event stream is represented as a pipeline of events from different consumers coming in asynchronously, rather like the clickstream we discussed back in our 2017 editorial ‘The Great Misunderstanding. At the time we postulated (rather unadventurously) that big data technology was more aligned with this kind of use case than suited to replacing a monolithic software application.
One of the oft-touted virtues of the cloud (and of agile) is the perceived access that these technologies provide to AI/ML functions. AMAI has little to say on the topic except that agile promises ‘quick utilization of artificial intelligence (AI) services’. ‘Cloud-based tooling enables enrichment of data in flight and the implementation of digital agents by using pre-built connectors to IBM Watson’s cognitive services’. These allow for AI capabilities to be ‘trivially embedded into clients’ integrations’. Trivial AI, well that’s something new!
The Redbook is a free download from IBM.
© Oil IT Journal - all rights reserved.