30 May 2021
Semat's Journey in 2020 and Early 2021
Dr. Carlos Zapata has stated, “400 years from now people will talk about Essence in the same way as people today talk about what Newton and Galileo did 400 years ago.” Please read more here on Four centuries of the Essence of Software Engineering.
As background, let us first quickly review the value proposition of Essence.
2.0 The value proposition of Essence
Essence has a profound impact on many different personas concerned with software development. One of those personas is the Executive which we address below along with the case for which Essence was originally developed. We recommend you read the full value proposition related to three key personas.
However, there are many more benefitting from Essence, almost everyone concerned with software engineering.
2.1 Values to the Executive
The Essence common language and way of thinking about process makes it easier to form teams and have them coordinate with others. The organization can still have common practices but not constrained within a one-size fits all process. This is future-proofed, rather than following the latest ‘fashion’ in industry standard process frameworks, they can evolve over their own over time, practice by practice, as the organization learns from within and from the outside world.
- Moving from primarily being a craft to primarily being an engineering discipline
- Improved governance of ongoing projects and programs
Originally, our objectives with Essence were to eliminate the methods war and create an eco-system from which teams and organizations could select the practices that worked for them and compose these practices to methods. Published in ACM Queue ‘Tear down the method prisons! Set free the practices!’. Essentially the same message was presented last year in a keynote in Norway.
However, as we addressed this problem, we learnt many more problems that could be addressed with the same approach. Many of these problems are presented in the value proposition. The most dramatic of those is by Ivar Jacobson identified as 'The Achilles’ Heel of Agile Method Adoptions' to be discussed further down.
It has taken many years to be able to realize concretely the underlying values of Essence to the software community. Appendix A presents the journey from the perspective of one company, Ivar Jacobson International. Other perspectives are included in section 3 on the differentiating values.
IJI has worked with a client Hapag-Lloyd wanting to use Essence to redesign all their methods as a practice eco-system as we originally had in mind when we designed Essence. Hapag-Lloyd is very successful in their mission as presented in this case study!
Over the past few years, we have come to realize a number of additional differentiating values of Essence that have helped fuel other personas including the team, and the coach. These are discussed below.
3.0 Differentiating values – the team and the coach
When using Essence, essential and practice information is available as a constant reminder to the team’s way of working, reducing the reliance on the coach over time for most basic things, as we said in the value proposition. The team can become self-sufficient quicker allowing the coach to spend time on the more difficult challenges. A few of those more difficult challenges that Essence can help with are discussed below.
Essentializing a practice means among other things that the practice is represented by a set of cards. These cards have many use cases played as games:
- Individual and teams learning new practices and enhancing training events.
- Team retrospectives improving the team’s way of working.
- Planning activities while efficiently coordinating the team and other stakeholders.
- Agreeing on responsibilities within the team and, importantly, with external stakeholders.
- Tracking progress of the key items under development.
- Determining the current status of the endeavor.
This is published in ACM Queue: ‘Scrum Essentials Cards – Experiences of Scrum Teams Improving with Essence’. However, similar values are achieved for most other practices. The value of game playing can best be perceived by actually playing it. The enthusiasm and engagement of the teams playing games are phenomenal. It is hard to achieve that experience by just reading a paper so we have created a video to demonstrate how Essence can dramatically improve team performance. An absolute must see!
3.2 Improving Existing Practices and Methods
While a major value of Essence is a kernel and language to describe and compare practices and methods, another value recently demonstrated is improving existing practices and methods.
Dr. Jeff Sutherland, co-founder of Scrum, uses Essence in his Scrum training and has stated:
“These Essence Cards flush out what is Scrum, what people are doing with it and how it is working…and we now have extensive data showing which parts of Scrum are implemented well, which are implemented poorly, and which are implemented not at all… And on the average, we find that about a third are implemented well, a third poorly, and a third are not implemented at all…. Essence immediately flushes out where the problems are, and then the next question is how can we use Essence to fix it?”
Dr. Jeff Sutherland says “Essence the key to success” in his blog ‘Better Scrum with Essence’.
For a discussion on how Essence and the Heart of Agile can be used together to build a better Scrum refer to Paul McMachon's 'Build a Better Scrum using Essence and the Heart of Agile'.
One of the challenges faced in getting many organizations onboard with Essentializing their practices is the perceived upfront investment. To learn more about a practical cost-effective approach to Essentialization refer to Paul McMahon's 'Incremental Essentialization: A cost-effective and practical value of “Stealth Essence”'.
3.3 The Method Adoption Challenge
Method creators focus on the learning cycle (books, papers, videos, slides, web sites, etc. and associated training and certification), but provide little or no support in guiding people in moving to the delivery cycle. The user of a practice, in particular a new practice or a rarely used practice, has to remember what she/he has learnt or go back to the teaching material to find out what to do. There is a gap, a big gap, to bridge for the user moving from the learning cycle to the delivery cycle. Further down in the delivery cycle, in particular in the Plan, Do and Check activities, the users can get tool support, such as Jira, Gitlab, but the gap between learning and delivery is still there, and it is significant. We call this gap “The Achilles’ Heel of Method Adoption”.
3.4 The human dimension and the challenge of structure versus culture
Alistair Cockburn has started a new initiative referred to as the Heart of Agile to address the concern that agile has become overly decorated. There is also concern that too many of today’s varying agile approaches are missing the essential human values as intended by the original agile manifesto.
One of the differentiating values of Essence is its use as a bridge between the “Heart of Agile” and today’s multitude of agile approaches. This use of Essence as a bridge is discussed in the following ACM Sigsoft Webinar, titled “Making the Heart of Agile Concrete”.
The Heart of Agile initiative has also stirred discussion on the structure versus culture debate. Read about how Essence can help with this challenge in the following two blog posts by Paul McMahon:
- How do you fix a problem with–or improve– a culture?
- Essence Patterns: A simple way to apply just the right amount of structure to fix a problem with– or improve– a culture
3.5 A thinking framework and using Essence in “stealth mode” to aid critical-thinking
One approach that has proven effective and is gaining momentum helping on-the-job critical thinking is the use of Essence in a natural conversation and root cause analysis mode. This has been referred to as using Essence in “stealth mode.,” For more information including stealth mode case studies refer to the following blog posts by Paul McMahon:
- Using Essence in “stealth mode” to help solve a Cybersecurity Challenge
- Scaling “Essence in stealth mode” and why you should care
- Critical thinking and Essence: Why every engineer and consultant should care
4.0 The Academic World
In academia we have at the forefront teachers and researchers and this area may well be where Essence has made its greatest achievements to date.
4.1 Essence Education Forum
The latest book “The Essentials of Modern Software Engineering” (2019), designed for first year students in software engineering, got a very positive reception in the academic world. As a consequence, we formed the Essence Education Forum with the mission to give teachers a platform to discuss Essence education, to develop training material, and to generally learn from one another. Jöran Pieper from Stralsund’s Hochschule serves as a chair for this community, which by now has had 8 meetings, one monthly. About 60 university teachers have joined this group. If you are interested in joining, please contact Bara Buhnova .
4.2 A MeetUp
Inspired by Sam Zawadi, the founder of Agnostic Agile, a meetup called Essence for Agility was created late 2020. The meetup is intended to create a grassroot movement. We have had five meetups with many important topics and many well-known speakers such as Jeff Sutherland, Kent Beck, Grady Booch, Scott Ambler, Ivar Jacobson. All meetups are recorded and you can find them here (see recorded meetups). If you want to join the meetup you can do it here https://www.meetup.com/essence-for-agility/.
The Essence for Agility meetup has got 2,400+ members in just a short period of time. It seems to meet our objectives.
4.3 Teaching software practitioners in the university to better prepare them for industry
Pekka Abrahamsson recently observed in an Essence Education Forum meeting that what we have been traditionally teaching our future software developers in the Universities is significantly different from what many practitioners are experiencing in the real world. This topic has been discussed in multiple Essence Education forum meetings. For more information we recommend you listen to a short talk recorded at one of those meetings.
4.4 Extending the use of Essence to other courses in Computer Science Curriculum
While Essence was originally developed with Software Engineering in mind, it has been observed that the framework can be extended to aid in teaching other computer science courses including cybersecurity, artificial intelligence/machine learning, and computer ethics. Each of these courses provides its own unique perspective into critical challenges related to developing software. For more information on how Essence could help in each of these courses as a general problem-solving aid refer to:
- Let’s start a conversation on integrating Essence into day to day activities as a thinking framework and general problem-solving tool
For more information on using Essence to help with cybersecurity challenges refer to
- Cybersecure Software: New twists to well-known questions every developer should be asking
- Using Essence in “stealth mode” to help solve a Cybersecurity Challenge
4.5 1st International Essence Conference
The 1st International Workshop on Essence in Education and Training was conducted in November, 2020 in conjunction with the 32nd IEEE International Conference on Software Engineering Education & Training. The workshop included multiple presentations on teaching Essence at the University and high school level, as well as a presentation on an Essence based game. Participants included representatives from both industry and academia and included some with prior experience and some attendees completely new to Essence.
Some of the most valuable aspects of the conference included discussions that ensued after each presentation. This included a discussion on how the Essence checklists were developed and how they can be used in the Universities to teach our next generation of software developers the essential goals of practices. Here to access the full conference presentations.
Here access a 10-minute keynote address from the conference, titled: An Industry perspective on what we should be teaching our next generation of software practitioners in the Universities.
The Swedish government organization Vinnova financed an effort by Teyi AB, Sweden, to develop an innovation method called Innovation Essentials. Innovation Essentials includes 9 practices helping “organizations to form effective collaborative partnerships to develop new business models based on innovative market offerings”. The practices have been well received by both large multi-international companies as well as start-ups.
Innovation Essentials is an example of using Essence for other objects than software development.
6.0 Concluding Remarks
The work on Essence and its use cases started about 15 years ago in Ivar Jacobson International, moved to SEMAT in 2009, resulted in a standard in 2014. Many people have joined this journey over the years. Many have not been as patient as the current leadership. We admit that we had underestimated the effort it would take to fundamentally change the software industry. Moving from a world of a multitude of methods with no common ground to a world of methods standing on top of a common ground is a human achievement without any comparison. It takes a major effort of say two days of work to “get under the skin” of Essence. Most methodologists, exceptions being Jeff Sutherland and Scott Ambler, see Essence as a competitor instead of an enabler. Practitioners need to move from seeing software development as primarily a craft to primarily being an engineering discipline. Organizations need to realize that their teams are often just paying lip services to their method adoptions.
However, today we see more progress and adoption than ever before. We are seeing more and more new use cases in the method’s space being realizable thanks to Essence. As a consequence, we believe we are getting closer and closer to the inflexion point when wider adoption of Essence by the industry and the academia will accelerate. Onward Towards Total World Domination.
Essence Development Journey
Being a formal standard, Essence is in itself a mature product reviewed and improved by hundreds of experts working through the standardization process of OMG. It is a product relying on several innovations derived from the international software development community, but now composed in a way that allow us to create practice libraries, compose practices into methods, make practices come alive, etc.
However, given that Essence provides so many use cases to so many different kinds of users – executives, program, product and project managers, teams and team members (analysts, developers, testers), researchers, teachers, students etc. it has taken a very long time to learn and experiment what values were required by the different communities needing Essence. To understand this, we think that you will benefit from understanding the journey we have taken.
In 1968 a NATO conference in Garmisch, Germany, coined the term software engineering. This was seen partly as the resolution to the software crisis the industry was living with. One of the biggest impediments to success was the ‘methods war’ which every company and every developer had to live with then, and that we are all still fighting to this day.
Since 2006 IJI has been working actively on what in 2014 became an international standard ‘Essence’ and its application in the form of practices and methods. Our objective was to eliminate the methods war, get rid of the dependence on method gurus and free the practices from their method prisons.
Given our huge ambition the journey has been long but fruitful:
- In 2007 we demonstrated that every method was just a composition of a number of mini-methods called practices and that a better user experience than traditional handbooks could be offered; every practice could effectively be described by using poker-sized cards. KPN in Holland became our first serious client of the Essential Unified Process. In collaboration with KPN we developed what became KPNUP which attracted high praise from the CIO and company method experts. During this time, we worked with several other companies such as Sony in Japan and Samsung Electronics in Korea.
- We realized that we could apply the same ‘practice’ idea for all methods in the world, not just the Unified Process, if we first identified a common ground which they could all share. This resulted in what we called Essence. We used it with clients such as Munich Re and Fujitsu UK, with very successful results among process engineers, but they had difficulties deploying it amongst teams. The process engineers wanted tools to author practices and methods, so we developed what is now, after two previous attempts (in 2008 with TCS and in 2010 with IJI China), the Essence in Practice Workbench™. This new kind of tool required new techniques and a new user experience and involved significant investment by IJI.
- In 2009 we created the SEMAT community with support from a large number of corporations, universities and famous software industry figures. Thousands of other individuals signed up for the idea of revolutionizing software engineering as a rigorous discipline, based on a kernel of widely accepted elements. The SEMAT community adopted the IJI idea of Essence and succeeded to make it an OMG standard in 2014.
- Having a standard ‘Essence’, including a simple language to describe practices, IJI was liberated to develop a large number of practices and create a public practice library. Given the limited resources we had, it took us into 2017 until we had reached this state applauded by industry process engineers.
- Although we had made great strides, we understood that something fundamental was missing. Our practices and the library did not deliver sufficient ongoing value to the teams:
- They needed an easy to understand user interface to navigate the practices,
- They needed a practical way to visualize the progress they were making, and
- They needed support in their daily work, the technology we now call Live Guidance™
Thus, our latest step in the journey has been to develop what we today call Essence in Practice TeamSpace™.
In summary, the journey has been very long since our ambitions were very extensive. We have had many clients on this trip. We have been very successful in getting process engineers to love what we have developed. But getting practice users (practitioners and teams) has been a longer journey. See the LinkedIn article ”Curing the Achilles’ Heel of Agile Methods’ Adoption”