What is quality engineering?
February 3, 2020
Quality Engineering (QE)
Building quality into all aspects of engineering guarantees less risk, increased market competitiveness and greater agility for the business (see also: What does a Software Quality Engineer do?). Lower incident and defect rates lead to better customer experiences, faster product rolls with lower testing cycle durations and less costs due to automation. Further, uncovering new efficiencies are known to be benefits to Quality Engineering and DevOps. Both disciplines synergise and enable world class engineering practices.
Quality Engineering enables “Quality at speed”
The picture below is a good illustration of how QE enables faster speed to market. The flow depicted at the top is the traditional approach to Quality Assurance – a waterfall process that has testing tacked on to the end, rather than embedded into the process from the start.
Quality Engineering focuses primarily on two approaches to achieve “Quality at speed”. Firstly, through process changes during the earliest stages in the Software Development Lifecycle (SDLC) – assuming the engineering practices are industry standard. Secondly, through tooling and technology improvements at the start of the SDLC – assuming the business function are industry standard.
Quality Engineering – Process Improvement
Approaches to Quality Engineering in any organisation are completed with the following stages from a process perspective:
- Understand the pain points of all stakeholders
- Analyse and gain insights from existing data
- Organise meetings and workshops to determine actionable items with a timeline
Understanding Pain points
Pulse surveys and meetings with groups of people and individuals, at all levels are required for obtaining a baseline for the current environment. It is through these activities that consistent themes are identified and initial investigations can be conducted to further identify the root cause of the issues.
Analysing data and gaining insights from the existing data (if present) is crucial in determining and evaluating all issues. Not all issues may present through the initial activities. Data from departments are needed for building the full list of issues. At the same time, if not enough data is present, new tool customisations need to be implemented so that the right data is gathered for visualisation on Business Intelligence solutioning or SDLC management tooling via dashboards.
Value stream mapping activities should also occur as part of activities conducted by agile coaches during this stage, to identify bottlenecks and further process improvements for the next big room planning session.
Quality Engineering – technology improvement
DevOps, Continuous Delivery and SAFe are methodologies for enabling digital transformation. Their adoption results in the modernisation of an organisation’s people, process and technology.
“Shifting left” is the best way to test!
Quality Engineering differs to Quality Assurance in how quality is achieved and the speed at which it is done – quality is built in throughout the process and technology changes at the start of the Software Development Lifecycle (SDLC). It means that all testing activities are done prior to where our first test quality gates would traditionally be – i.e. system and system integration test. It is the cheapest and most efficient means of ensuring quality as most defects are introduced during the earliest stages of development – as shown in Figure 1.0 below:
Figure 1.0 – where defects are introduced
A result of Continuous Delivery (CD) with Quality Engineering is streamlined, standardised, Continuous Testing. It is highly efficient and effective and allows for platform and pipeline integration resulting in engineering efforts unseen in the traditional Quality Assurance discipline.
- Testing in the earliest stage of software development in an Agile Development environment means that the entire testing paradigm changes: “Everything as code” is as relevant to Test as every other engineering practice. This means that engineering standards, tools and technologies are a must for any quality engineer
- No departmental silos are created, instead cross functional teams engineer a solution and own it – this means that any task within the team is owned by the team (including quality engineers)
- Quality engineers can provide engineering solutions that are not just tests
- Quality engineers can assist with governance of code quality
- Process improvements are also a responsibility of a quality engineer as you can enhance the quality of a product or service through process alone (this is the Atlassian model of operation)
- The majority of tests are automated verses manual effort. This results in effort spent writing self-documenting code that is stored in a central location and visible to all engineers and; also running exploratory test sessions to find defects faster and cheaper (in source code repositories).
- Application test code is coupled with application code for visibility, efficiency and re-usability
- Application test code is written before the application code is even implemented and allows for Test Driven Development (TDD) as well as Behavioral Driven Development (BDD)
- Software Engineering frameworks are used in development and test due to the change in software engineering expertise – quality engineer skillsets are higher
- DevOps tooling and patterns are also mixed into test tool and technologies for on- demand test infrastructure, reporting, monitoring, code scanning and security and vulnerability scanning tools.
- Business teams can create, edit and have access to all acceptance criteria (business requirements) as they are translated into code and exist as the central source of truth
- Application layers are separated by concerns as part of design patterns that are industry recognised. This means that tests can now be written into different layers for added efficiencies and less flakiness
- Test patterns follow the same software engineering principles as all the other engineering departments as the code is now in the same repositories as all other production code and not separated and “siloed”
- Generic modularity and containerisation is now expected as part of any deployment and test is no different. All test code should follow the modular patterns that are abundant in DevOps and Continuous Delivery.
- Test data is no longer an issue. Data can be injected into containers so that states are repeatable and perfect for automation
- Regardless of how applications are being tested, binaries that are created by trunk style deployment result in the same test approach. Engineering teams can build features faster and test more frequently by having smaller incremental software updates that exist for a shorter period.
- Modern web architecture and newer software design patterns have led to a three-tiered architecture. Testing as a result is more targeted due to less coupling within systems. This means that only two tiers of a modern web application need to have a targeted test level and that is all that is required to mitigate risk – the integration layer gives feedback from the Application Data Layer. Advances in tooling have also allowed for even cheaper and more efficient test levels like contract & integration testing. (ensures that integration between systems are unbroken)
- Exploratory testing is a level that can be introduced to cheaply and efficiently identify defects to allow quality engineers to focus more on valuable tasks – like building quality into the lifecycle.
- This means that System and System Integration Tests, E2E tests, Regression tests are no longer required. Targeted tests with limited scope should only be completed assuming that all tests used as part of Continuous Integration are up to date, test coverage is increased or decreased depending on the code impacts from code merge and all levels of tests are adequate and follow the test pyramid. See the diagram at Figure 2.0 below:
Figure 2.0 – Test Levels in Venn Diagrams
So why are System and System Integration Tests, E2E tests and Regression tests no longer required?
Unit, Integration & Contract, Acceptance Tests
Systems are tested by separate cross functional teams at varying test levels. The technical depth is deeper than purely black box methods of testing as there are white box testing levels in this approach.
System & System Integration Tests
In System test, the application under test is treated as one system. There is no separation of concerns (treated like a black box) from a test perspective and typically System test is focused purely on front end testing. System Integration Test, tests the integration between two systems and emphasises the functionality of API web services. The combination of both approaches is a waterfall approach to testing and is purely black box with little technical depth meaning it is less efficient and cost-effective finding defects with these two levels of functional tests.
In agile development environments, testing usually follows the test pyramid. In some circumstances, regression test is added as the last quality gate to add further confidence to the testing function. Regression isn’t necessary in this regard and adds a level of cost and redundancy that is not efficient. Furthermore, Regression testing is a “blanket” or non-targeted test level that is not typically maintained well and, in most organisations, adds no value as the tests are duplicated, flaky because they are normally front end and add to the test cycle length unnecessarily. At the same time, Regression test cannot replace all functional testing due to the nature and risk that the tests for a code change, are covered by the suite.
In many organisations, different teams end up ensuring that functionality that spans their system under test to other integrated systems work. The problem is that with tightly coupled systems with older architecture, some of these functional flows span multiple systems. This leads to massive amounts of redundancy across multiple teams. This can be done effectively through communication between teams, but this rarely happens and what you mostly see are test suites that are much larger than they should be. This is usually a symptom of team “siloing” even in the same department.