Why Human Touch Still Matters in Automated Process Testing

Emerging patterns in the coded items suggest tool and methodological recommendations. This analysis aligns with the advanced coding phase of Grounded Theory. The inferred suggestions and concepts provide concrete examples and recommendations for implementing AB-BPM tools.

The integration of the ABBPM tool with a process model repository (PMR) was highlighted in one interview and supported by other participants. The main aim of the Integrated Process Model Repository (IPMR) is to facilitate the introduction of new process variants to participants and collect their feedback. Notifications about process updates would include key information and links to further materials, potentially accompanied by a short test to ensure comprehension. Feedback from participants could be evaluated by process experts.

To mitigate failed process instances, an instance recovery mechanism (IRM) is proposed. Process experts could set thresholds for specific KPIs before starting an experiment. If a process instance reaches a threshold, experts would be notified to intervene. This intervention should be limited to erroneous cases to avoid experimental bias. Including manually salvaged instances in model evaluations with a penalty value could better assess the process version. IRM relates to real-time business process monitoring.

Interviews also indicated the need for increased human control, reflected in perceived risks and desired tool features. An initial prototype enhancing AB-BPM methodology with human control features has been presented previously.

The study has limitations, including the use of a shortened RTDM, which may affect result validity. The qualitative nature of the study suggests rankings should serve as guidance rather than definitive conclusions. Participation in the validation round decreased, but higher participation in the ranking round and feedback options helped address validity concerns. All participants were from the same company, but were selected based on diverse experiences.

This study aimed to gather practitioners' perspectives on the ABBPM method using qualitative research methods. Participants generally viewed the methodology as beneficial compared to current practices. Key conclusions include the need for increased human intervention, transparency, participation features, and integrated process execution for seamless deployment of process variants. Future research opportunities include studying unstructured process tests in production environments and tool-driven training of participants. Insights from different business improvement methods could further enhance comparative studies.

Aguilar-Savén, R.S.: Business process modelling: Review and framework. International Journal of Production Economics 90(2), 129–149 (Jul 2004) Bass, L., Weber, I., Zhu, L.: DevOps: A Software Architect’s Perspective. Addison-Wesley, 1st edition (May 2015) Brinkmann, S., Kvale, S.: InterViews: Learning the Craft of Qualitative Research Interviewing. SAGE Publications, Inc, Los Angeles, third edition (Apr 2014) Chun Tie, Y., Birks, M., Francis, K.: Grounded theory research: A design framework for novice researchers. SAGE Open Medicine 7, 205031211882292 (Jan 2019) Dalkey, N., Helmer, O.: An Experimental Application of the DELPHI Method to the Use of Experts. Management Science 9(3), 458–467 (Apr 1963) Dumas, M., La Rosa, M., Mendling, J., Reijers, H.A.: Fundamentals of Business Process Management. Springer Berlin Heidelberg, Berlin, Heidelberg (2018) Holland, C.W., Cochran, D.: Breakthrough Business Results With MVT. Wiley, Hoboken, NJ, 1st edition (Apr 2005) Ivanov, S.Y., Kalenkova, A.A.: BPMNDiffViz: A Tool for BPMN Models Comparison. Proceedings of the Demo Session of the 13th International Conference on Business Process Management (2015) Kampik, T., Weske, M.: Event log generation: An industry perspective. Enterprise, Business-Process and Information Systems Modeling. Springer International Publishing, Cham (2022) Kang, B., Kim, D., Kang, S.H.: Real-time business process monitoring method for prediction of abnormal termination using KNNI-based LOF prediction. Expert Systems with Applications 39(5), 6061–6068 (Apr 2012) Kohavi, R., Longbotham, R.: Online Controlled Experiments and A/B Testing. In: Sammut, C., Webb, G.I. (eds.) Encyclopedia of Machine Learning and Data Mining. Springer US, Boston, MA (2017) Kohavi, R., Longbotham, R., Sommerfield, D., Henne, R.M.: Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery 18(1), 140–181 (Feb 2009) Kurz, A.F., Santelmann, B., Großmann, T., Kampik, T., Pufahl, L., Weber, I.: HITL-AB-BPM: Business Process Improvement with A/B Testing and Human-in-the-Loop. Proceedings of the Demo Session of the 20th International Conference on Business Process Management (2022) La Rosa, M., Reijers, H.A., Van Der Aalst, W.M., Dijkman, R.M., Mendling, J., Dumas, M., Garcia-Banuelos, L.: Apromore: An advanced process model repository. Expert Systems with Applications 38(6), 7029–7040 (2011) Malinova, M., Gross, S., Mendling, J.: A study into the contingencies of process improvement methods. Information Systems 104, 101880 (2022) Okoli, C., Pawlowski, S.D.: The Delphi method as a research tool: an example, design considerations and applications. Information & Management 42(1), 15–29 (Dec 2004) Paré, G., Cameron, A.F., Poba-Nzaou, P., Templier, M.: A systematic assessment of rigor in information systems ranking-type Delphi studies. Information & Management 50(5), 207–217 (Jul 2013) Pedrinaci, C., Lambert, D., Wetzstein, B., van Lessen, T., Cekov, L., Dimitrov, M.: SENTINEL: a semantic business process monitoring tool. In: Proceedings of the first international workshop on Ontology-supported business intelligence. OBI '08, Association for Computing Machinery, New York, NY, USA (Oct 2008) Pietsch, P., Wenzel, S.: Comparison of BPMN2 Diagrams. In: Mendling, J., Weidlich, M. (eds.) Business Process Model and Notation. Springer, Berlin, Heidelberg (2012) Renn, O.: Concepts of Risk: An Interdisciplinary Review Part 1: Disciplinary Risk Concepts. GAIA - Ecological Perspectives for Science and Society 17(1), 50–66 (Mar 2008) Robson, C.: Real world research: a resource for social scientists and practitioner researchers. Blackwell, Oxford (1999) von Rosing, M., White, S., Cummins, F., de Man, H.: Business Process Model and Notation—BPMN. In: von Rosing, M., Scheer, A.W., von Scheel, H. (eds.) The Complete Business Process Handbook. Morgan Kaufmann, Boston (Jan 2015) Sato, D.M.V., De Freitas, S.C., Barddal, J.P., Scalabrin, E.E.: A Survey on Concept Drift in Process Mining. ACM Computing Surveys 54(9), 189:1–189:38 (Oct 2021) Satyal, S., Weber, I., young Paik, H., Di Ciccio, C., Mendling, J.: Business process improvement with the ab-bpm methodology. Information Systems 84, 283–298 (2019) Stiehl, V., Danei, M., Elliott, J., Heiler, M., Kerwien, T.: Effectively and Efficiently Implementing Complex Business Processes: A Case Study. In: Lubke, D., Pautasso, C. (eds.) Empirical Studies on the Development of Executable Business Processes. Springer International Publishing, Cham (2019) Sutton, R.S., Barto, A.G.: Reinforcement Learning, second edition: An Introduction. MIT Press (Nov 2018) Tsui, T.C.: Experience from the Anti-Monopoly Law Decision in China (Cost and Benefit of Rule of Law) (Apr 2013) Vassiliadis, P.: A survey of extract–transform–load technology. International Journal of Data Warehousing and Mining (IJDWM) 5(3), 1–27 (2009) Vom Brocke, J., Zelt, S., Schmiedel, T.: On the role of context in business process management. International Journal of Information Management 36(3), 486–495 (Jun 2016) Yan, Z., Dijkman, R., Grefen, P.: Business process model repositories – Framework and survey. Information and Software Technology (2012)

Aaron Friedrich Kurz, Technische Universität Berlin and SAP Signavio, Berlin, Germany; Timotheus Kampik, SAP Signavio, Berlin, Germany; Luise Pufahl, Technische Universität München, Munich, Germany; Ingo Weber, Technische Universität München and Fraunhofer Gesellschaft, Munich, Germany.

This paper is available on arxiv under CC BY 4.0 DEED license.

Based on reporting by hackernoon.com.