Policies over Procedures over Standard Work Instructions
This is the first part of a two-part article describing my personal journey and learnings as the first dedicated Scrum Master in a company working in a controlled regulated environment.
Let’s start with a confession: I made a mistake. Not a small mistake like doing a presentation in old corporate branding, or forgetting to loop someone important in CC. The scope was huge: I wrote Agile policies describing Agile processes.
Even years later — writing “I made a mistake” seems unnatural to me. In many of our highly regulated companies mistakes of others are a precious currency traded within management meetings. You can exchange this knowledge for mutual agreement on the success of one of your projects before having any reliable data on that. Knowing failure is worth getting your role praised with some required steps in a policy, and for knowing larger mistakes you could even increase the number of your directs.
In my defense — most of this already has been set in motion before my arrival as the first dedicated scrum master. There already had been policies, procedures, and work instructions galore. To an — maybe intended — extent every auditor couldn’t do differently than to approve with minor findings when having been tasked with a review. I once proposed to get a bunch of documents printed to show the sheer madness of having these kinds of processes in place. Our budget processes probably would have prevented us from spending this kind of money without having an NPI (New Product Introduction) planning years upfront and getting this approved would have just meant more documentation.
Policy Tracking System. Every month, every employee got his tailored share of approximately 30 changed and new policies to give a digital signature for a tracking system. It was a well-kept secret that only a few really read those. Some made an effort to give their signatures spread over the month — to maintain the appearance they at least could have read them. Some didn’t care much and did a signature marathon at the end or even on the first day of the required period and others didn’t care at all forever to be bugged by tracking systems notifications with pictures of red traffic lights annoying them and their managers.
Signature Workflows. Those policy documents produced other documents by demanding them to be in place and also digitally signed before something else could happen at all. Fixing a spelling error in a code comment by the process could easily require up to 8 SDLC (Software Development Life Cycle) documents newly being written and then each of those signed by up to 7 different managers. Those documents would be set to the state open and signature workflows would be started sending emails and notifications. Those workflows by an inherent logic — certainly not by tooling support — only some grasped, required a specific order in which those documents would need to be signed. The state signed/closed of some of those would be the precondition to close others. One signature missing on one of them — or heaven forbid — a finding, typo, or even comment of some kind would mean for everything to be rolled back.
Catch-22. Some months in I noticed having deadlocks in signature workflows. This left me wondering why none of my colleagues — all of which I regarded being more intelligent and knowledgeable than myself — did notice this prior and I decided to observe one project closely. It has been because cunning Project Managers and Product Owners knew ways to bypass our system by printing some of the documents, using their good connections in at least one of the departments responsible (among others: Multiple Regulatory Affairs divisions being placed around the globe, Quality Assurance, Validation, Requirements Engineering…) handed in a bunch of these paper-based documents to be signed simultaneously and scanned the results afterward. This loophole of course has been hegemonic knowledge, so using it to improve the workflows themselves didn’t cross anyone's mind.
“Scrum is sh*t — because of the impact assessment” a developer once complained to me in an outburst after a retrospective I facilitated. Feeling irritated by this, I asked whether he likes to specify. For each code change proposed, the developer told me, an extensive Impact Assessment document, also to be signed by management personnel for him to wait upon these signatures needs to be provided by him. Continuous deployment with small incremental changes implied, he couldn’t aggregate his code change proposals for some months (years?) as he had done previously to at least justify somehow the many days needed to produce this Assessment document. The developer had spent most of these previous months writing prose.
Bug Handling. As you might have guessed by now, Bugs didn’t result in a 0-bug-policy collaborative effort of a cross-functional, self-organized team to fix them — let alone a mere developer doing a push while touching the code of a flawed module anyways. In many cases, fixing the input documentation was the way to go. The original requirements on the SDLC level were modified to request the defect one year before finding it. This has not been a request to fix it, but for the bug to be in place, which was easiest, since it didn’t affect the output documentation as well. Even when a bug had been decided on to be fixed, it would still need to be traced to an original requirement describing explicitly the expected behaviour of the system without the bug. Since there is — and should be — a gap between code and requirements in many cases RD, FS and other documents known by two or three letter acronyms (MRD?, PRD? RRS? Risk Mitigation Documentation?, Testing Reports of different flavor?) would need to be modified after finding this defect. Should you be handed a Release Requirements Specification (on a side note: this has to be because you did something bad in a previous life) and wonder about, why in some cases it gets rather detailed: That’s why — a developer might have had a bad day, introducing new bugs in the required modules and the input documentation got detailed afterward — but only in these chapters.
One day a developer made the mistake to report a bug already having been reported. He arrived at my table complaining, that he can’t close (let alone “delete”) the duplicate in their ALM software since the original version had already been synchronized to another system. This one was under the sovereignty of the Testing and Requirements Engineering Departments. RE — having me employed originally — always has been supportive in these topics, but in that case, their hands also were tied. Testing — is measured by the number of bugs they processed — had workflows, where an additional number of employees would need to be involved, also deciding that this is a duplicate and providing the proper documentation on mitigation of the risks determined by having this duplicate bug. If you’re thinking about, whether the number of bugs is a good measure to improve development — the company also had dev. teams being measured by LoC (Lines of Code) at some point — with more lines really being regarded as better.
Policies of mixed complexity. Policies didn’t concern me till now, but it became clear that one would have to deal with them. I made an effort to gather, read, and understand all DLC and PLC policies. All of those have been written by competent colleagues, the whole set has been a construct for minds more beautiful than mine to be completely lost in. One of my learnings from that period has been, to look out for policies with a change of complexity and to tread carefully when digging deeper there. Policies having a simple part, maybe a simple part after and a labyrinth in-between usually meant: A colleague secured his position in our company in the complex part, by demanding a role only he fills to do a mouseclick now and then.
Further Process Steps introduced by ALM. Policies weren’t the only artifact needed to understand the processes in place. I had a closer look at the required steps in the supporting software. Only (a) the XY manager can generate this report to be signed and handed in with a change, only one of these other (three) roles may open this window, look at this sheet, and so on. I once tried to predict the cycle time of a digital document by correlation to the total number of directs to the managers being required to sign. Oh, just in case: It’s been Jira, TFS, Rally, and Siemens Polarion to rule them all.
Further Documents introduced by compliancy. Be it the FDA with their CFR 21 Part 11, the German BaFin, NHTSA, the regulations of 8th EU Directive, SOC2, ISO 26262, MiFID II — regulations and the authorities auditing and enforcing them are the killer argument against scrum in any controlled environment. Why can’t we have a microservice infrastructure? Bafin! Why couldn’t we show the tests running in a Jenkins pipeline instead to bridge the gaps between the different staging systems quickly and automated? FDA! Why is it not possible, that an engineer corrects a spelling error in a requirement? SOC2! Why do we also have this policy to demand this document with information already in those documents? NHTSA! To my best of knowledge, most of this simply is not true. Scrum is a sound framework to deploy working code in cross-functional teams frequently. Why should an auditor have anything against this? The real compliance rules in many cases are disgracefully few. The problem lies with the colleagues within your company who usually interpret those auditing rules. How to do this religious task is also hegemonic knowledge and in many cases, the result is requesting more documents to be provided.
Writing Agile SOPs and Agile SWIs. Obviously, you can’t have this “agile” which has been requested from somewhere unknown, without also putting the term in some documents. When I arrived, they already had one v0.x Agile SDLC policy for incremental iterative elaborative development in place. An effort of more than 50 exhausting Webex calls, each with about 60 invitees, and specialists, and external consultants doing the work after writing this. Me still knowing all version numbers of these policies speaks volumes. No one dared to call this scrum — they ultimately were aiming to have another framework that required even more roles and would prevent middle management and software architecture from feeling unrecognized. I then had endless discussions about a line in a BPM in a new agile Verification policy which implied a verification phase after development as a precondition for code deployment. Supported in writing Standard Operational Procedure on User Stories, smuggling in agile principles, scrum values, events, and collaborative practices mapping them to requested process steps to justify this. Wrote a Standard Work Instruction on Relative Estimation Practices, replacing a completely different document which has been owned by a colleague who hasn’t been working there anymore and wouldn’t find out about this. Got my own role included as ‘Agile Facilitator abbr. AF’ in some more policies; I knew ‘scrum masters’ can’t be in there. Wrote an Agile R&D Definition of Done as a rather generic new document with the most basic steps requested in there for the teams to further add upon. Even managed to deprecate a required document about Development Metrics by simply asking who the consumer of this would be and what — except some document workflows failing — would happen, should we not provide this. These policies found their way in the tracking systems and some months later developers had to sign some more documents.
To production and beyond. Well — no. We were not really deploying to production. Usually, this went in another staging system for another department like UX, HCI, Validation, or likewise to have a first look. She who now cries waterfall — you’re correct. Teams had multiple long-running projects in parallel, some in this new Agile SDLC and some in the Waterfallish SDLC. Yet both DLCs were valid and still constantly added to, both were requiring different sets of additional documentation to be handed in with different templates.
Regulatory Affairs (RA) disapproves. I made the mistake of using the word approve in the by then version-controlled Definition of Done policy. RA — already having been upset about this “agile” — had a colleague who — during my first on-boarding week by now 1 1/2 years ago — yelled at me saying that as far as he is concerned, I’m a Project Manager and nothing else. No wonder they now made the move declining any signatures under documents further down the workflow. They demanded that, by another policy unknown to me, only RA can formally approve anything and not our Product Owners. So nothing could be deployed anymore anywhere by any teams already having projects started with Agile SLDC document templates. Having written the policy in question, I acted on this, got into further discussions, and we decided to replace the word approve with accept in a new controlled version of this DoD. This change of words wasn’t as easy as one would imagine.
My manager approves, I don’t. My manager has been Senior Director of R&D, working in my company for over a decade and having quite some colleagues and departments reporting to him. I’ve been escalating all problems (Japanese would call these: Shouganai) to him every week for 1 1/2 years by now — in 95% copying my previous week's mail. He was benevolent, polite, had a university degree in computer science but — from what I determined by the sparse responses to my mails — didn’t care so much about what I have been doing. He gave his required signature for the policy change approve to accept instantly. His regretfully hasn’t been the only signature required and when having waited another deployment-less five months for the last of the six managers — who was working on another continent and probably never heard of the teams being impacted here — to sign, I stood at my bosses doorstep complaining. He told me to sit and started typing a request — done in the most polite English I’ve ever seen — to the American colleague asking to provide the one missing signature (fun fact: this still took one month after, so six in total). At that moment I noticed, that my boss had been caring the whole time. I felt gratitude, empathy, but this instant also knew I do not want to work there any longer.
