The Software Factory becomes true
The Production Line software
DevOps in conjunction with Continuous Delivery promises not less than the realization of the software factory. Management is in the process of creating the right conditions for ist realization – or the path to failure.
A brief historical excursion – how it does not work
The promise of the software factory is by no means a new one. Since the beginnings of software development, efforts have been made to streamline the software development process. The goal was -and is- the plannability and quantifiability of the development efforts as well as achieving a substantial increase in the development speed (see also „DevOps Strategies“).
Inadequate approaches to achieve rationalization across the software industry are:
- Re-use of already manufactured components or purchased standard components. Admittedly, today this technology is the basic prerequisite for any commercial software development. However, re-use has not enabled the software factory in the industry.
One reason for this is that many software developments properly represent a project; within a limited time, something new is created that makes cross-departmental efforts necessary. It is almost self-evident that in most cases such projects cannot be realized solely by the combination of standard components.
- Another attempt to streamline software development was -and is- the use of Computer Aided Software Engineering (CASE). CASE includes the automated generation of program code through the use of graphical tools. These tools should abstract from the actual source code. One of the reasons why case tools have not prevailed are:
- The fact that CASE tools cover the source code’s complexity only superficially leads to the situation that developers almost always have to go down to the code level, thereby making CASE modeling and source code diverge. For this reason, according to the author’s experience, CASE tools have been established only in niche areas in the long term.
- Another approach, Model Driven Architecture (MDA) is quite popular, as a careful modeling of the architecture is always beneficial to a successful implementation afterwards. However, like CASE, MDA is also less efficient in heterogeneous IT landscapes and the resulting different architectural approaches. Similar to CASE, MDA is perceived by the developers as an obstacle to the development of simple components due to the additional abstraction level and the resulting overheads.
How it can work
As mentioned above, DevOps, in conjunction with Continuous Delivery, has entered into this promise, but with a completely different approach (DevOps Strategies).
The following quotation attributed to Albert Einstein fits very well to the principles of DevOps and Continuous Delivery:
„One should make everything as simple as possible, but not simpler“
There is no attempt to shield the developers from the complexity of the source code – instead, the following approaches are followed:
- DevOps culture is established according to the CALMS principles (DevOps Strategies). This should anchor the appropriate mind set in the organization, e.g. that the program code does not „belong“ to individual programmers, which requires a new sense of responsibility.
- The agile processes are designed and organized as simply as possible (e.g. Lean Management).
- Systems are designed in strictly modular ways (for example, micro services).
- Progress is made in small increments, i.e. releases are performed at short intervals. Thus the changes between successive releases are relatively small and thus in many cases more or less uncomplicated.
- Automation. All the steps routinely executed by developers and admins are carried out automatically, such as the building and distribution of the software packages as well as, if possible, the testing.
Smaller companies that start with a new product are in a better position than large companies, that either:
- Build and operate new complex products
- Want to replace existing complex products
It is much easier for small companies to design and develop a micro service architecture that is directly deployed in front of customers. In extreme cases, this can even happen without an intermediate test department if:
- There is a sufficiently short feedback loop between customer and development
- The increments, i.e. steps between the releases are sufficiently small and manageable
- No SLAs against customers are to be complied with
- In order to carry out such small development steps, as described above, the agile mode of development is appropriate.
In contrast, the establishment of a continuous delivery pipeline within a DevOps adoption for complex products represents a financial and organizational challenge (DevOps Strategies): Existing processes and the corporate DevOps culture have to be established. This refers not only to the above-mentioned CALMS, but also to the agile software development processes – these must be actually agile for a CD pipeline. The technical know-how for the construction of the CD pipeline and its operation must be established. The CD pipeline has to be created – this may lead to the situation that existing applications and systems have to be ported to the CD pipeline.
Infrastructure and CD pipeline must be designed on the basis of the software products to be developed. This can be done in the form of two project phases in which the core CD pipeline is first built and afterwards adapted to the software to be produced in the second project phase. Afterwards, the production is carried out further releases within the line organization using the CD pipeline.
There is no way around the implementation of agile process and presents an additional challenge to the organization.
As shown schematically in the figure above, this may result in three successive phases:
Building the core CD Pipeline:
First, a CD pipeline must be built up in its basic structure. The core of the CD pipeline is created in the first phase of the project, with minor reworking still taking place in project phase 2.
Adapt the CD-Pipeline to software product:
Already in the last third of the project phase 1, developers begin with adaptations of the core CD pipeline to the software product. The main work, however, takes place in the second project phase. Final activities will take place in the line organization.
Functional Software Development:
The main focus of the functional software development is in the first phase 1, in which a first release with functionality is already implemented. However, this has to be be synchronized with the structure and demands of the CD pipeline. If the developments are already diverging here, the project’s success is already highly endangered.
It is evident that it would be a completely missed strategy if an organization would organize its DevOps Engineers in their own team („Silo“), which presents the developers the CD pipeline „. Many DevOps introductions have failed.
The above figure also shows the resources used:
The functional programmers are already present in phase 1 and phase 2, when, for example, functional prototypes are created. At this point, it must be ensured that the functional software is not built in contradictory ways to the requirements of the CD pipeline.
The work of the DevOps Engineers has the greatest scope in the first two phases of the development of the core CD pipeline and its adaptation to the software product to be developed. Afterwards, in the production phase of the line organization DevOps take changes and maintenance work as well as the third-level support.
⇒ It is important to ensure that developers and DevOps work closely together. From the middle of project phase 2, the utilization and / or the quantity of resources used to increase, but this happened in a manageable way. The author has made good experiences with Kanban in this context.
From the outset, operations is to be involved. The first is to help build a CD pipeline at the infrastructure level. Then it is about their operation in the creation of the actual functional software.
⇒ Once again: Developers, Operations and DevOps Engineers do not work separated from each other, as the upper figure might seem to suggest. The project will fail if the efforts during the project phases are not synchronized. A common mistake is that DevOps Engineers appear as a silo, create a CD pipeline, and then throw it „over the fence“ to developers and administrators,
Such a phase plan must be set up in order to be able to control the management’s expectations. The creation of a CD pipeline for complex systems is very large invest. It must be made clear to the management that the releases do not go into production at the same speed from the outset, as if they would be produced with a purely agile approach – without a CD pipeline.
So, it is not only important that DevOps Engineers work closely with developers and administrators. Rather, the business or the customers, who are informed about the content of each release as a result, must also be involved. The whole thing has to be. Through a correspondingly agile release planning and project planning.