The Unicorn Project, Part 3

By Dick Stark

In case you missed my previous blog articles, Part 1 of the Unicorn Project reviewed the early failures of “Phoenix” from the Dev rather than the Ops perspective. (Remember the Phoenix Project is Gene Kim’s prior book about a fictional legacy automotive parts supply company, Parts Unlimited and their efforts to move from a traditional brick and mortar to digital business.)  In the Unicorn Project, the main character is Maxine, a very successful architect/developer who has just been “exiled” to the Phoenix Project. After just a short time coming up to speed on Phoenix, Maxine quickly understood why things have gone so far south, and now it is up to her and her team to try to fix things and save the company. In Part 2, Maxine, along with other members of the “rebellion” demonstrate that they can make a difference despite looming odds. In just a few short months, Maxine’s team pays down past “sins” (technical debt) to help begin Parts Unlimited’s digital transformation.

Part 3 opens with just several weeks to go before Black Friday, a day that will stress their computing infrastructure and the Phoenix code base. The Unicorn team is officially named with the understanding that “fast beats slow,” and that this team will power the customization and promotion efforts required for Black Friday. (It is critical that the Phoenix system be able to easily offer promotions to capture needed holiday revenue.)

On Tuesday before Black Friday a 1% simulation test is performed.  A serious bug is found, but thanks to improved processes and environment, a fix is made in about 10 minutes, an amazing accomplishment.

Dev and Ops team members go to work at 3:30 AM Black Friday morning. The launch goes off, but the orders are processing too slowly due to the heavy demand on the front-end servers. They offload the servers to a Content Distribution Network (CDN) which prevents everything from crashing. Success, but still a lot of work to prove that they aren’t a Borders, Blockbuster, Sears or ToysRUs.

The results of the holiday sales are significant–$35M in incremental revenue. Additionally, a new mobile store app eliminates much of the manual customer data collection process—a huge timesaver. Given, the perceived digital benefits, management is interested in investing $5M in a new innovation team and plans an innovation contest to pick the top three technology ideas.

Despite progress made, it is still not all “unicorns and rainbows” at Parts Unlimited. Their equity partner requires cuts of 150 people to ensure profitability. Although reductions in force are never easy, efficiency improvements make this less painful. Erik, Agile guru and Board advisor, discusses the concept of context v. core.  “Core” is Parts Unlimited’s core business, while context is what’s required internally to support that business. Cost cutting forces Parts Unlimited to determine what context can be unloaded, thus freeing up technical debt. An IT Manager remarks, “I’d pay anyone in duffel bags of unmarked bills to get rid of our helpdesk system…. combined the workload of managing all those things is easily three people…”

It’s now late January and the company reports profitability for the first time in 2.5 years. The CEO concludes, “Because of all the amazing groundwork that Phoenix laid down, the Unicorn teams were able to quickly create promotions capabilities to drive people to our mobile app, e-commerce site, and physical stores. It was an amazing combined effort that include in-store staff and the technology teams.”

Posted in Atlassian, Business Management, DevOps, ITSM | Tagged , , , , , , | Leave a comment

Report from Atlassian’s Virtual Summit

By Dick Stark and Alan Reynolds

Instead of a live Las Vegas Atlassian Summit, Atlassian hosted a virtual event April 1 and 2. Although it is not clear how many showed up, Atlassin reported that more than 25.000 registered for this event.  At any rate, I attended the keynote and Alan was able to attend most of the other sessions. Here is a short summary.

Keynote. Atlassian co-founder and co-CEO Scott Farquhar, pictured above, opened from his home office by wishing everyone well in a time of uncertainty. Given what’s going on in the world, now, Scott pointed out that success is dependent more on alignment between teams than the success of any individual team. Furthermore, if working from home, Atlassian is the hub of work for any team, no matter the location. Everything works together.

Next up was Kelly Drozd, Agile Delivery, Marketing, ALSAC/St. Jude. ALSAC is the charitable foundation associated with St Jude Children’s Research Hospital, and at $1.4B is one of the largest healthcare charities in the world. They are a huge Atlassian customer and raving fan. Even better, ALSAC also happens to be a RightStar BCM BCM customer. They have 175 projects going across the enterprise focused on enterprise ops, event management, gift planning, innovation, marketing, partnerships and HR.  Kelly explained that Atlassian tools have empowered teams to take control of work with Agile adoption scaled across all projects with the customer at the center of all they do. ALSAC has saved millions of dollars in software costs and “won’t stop until no child dies from cancer.”

Other takeaways. The focus on cloud was the central theme of the Summit. And for good reasons. Here is a short update.

Cloud Offering Tiers. Free, Standard, Premium, and Enterprise are all available. Premium (announced 2019 provides unlimited storage, guaranteed SLAs, and extended support. Enterprise (new) offers an unlimited horizontal scale, unified command center, enterprise grade marketplace apps and 24×7 customer support.

NextGen JSD Cloud Projects. What’s most exciting are the new templates not only for IT and External Support, but also purpose-built templates and workflows for HR, Legal and Facilities teams (available for both NextGen or classic projects). Change enhancements include a bulk change feature and a change risk assessment.

ITSM Early Access Program. (now available, atlassian.com/itsm-ea) EAP includes stronger integrations for Incident and Change Management. For example, submitting a pipeline request automatically submits a JSD change. Additionally, a Service Graph feature determines the impacted service by relating repositories to services and automatically attaches the relevant JSW tickets to the JSD ticket too, so the Change Management has full context. Other enhancements include personal accessibility settings for 508 compliance, and multi-lingual JSD customer portal.

Overall, despite the current situation, Atlassian is on a roll and is making excellent progress with new products, features and overall strategy.

Posted in Atlassian, Business Management, DevOps, Digital Engerprise Management (DEM, ITSM, RightStar | Tagged , , , , , | Leave a comment

The Unicorn Project, Part 2

By Dick Stark

To refresh, Part 1 of the Unicorn Project reviewed the early failures of “Phoenix” from the Dev rather than the Ops perspective. Here the main character is Maxine, a very successful architect/developer who has just been “exiled” to the Phoenix Project. After just a short time coming up to speed on Phoenix, Maxine quickly understood why things have gone so far south, and now it is up to her and her team to try to “join the digital age,” and save the company. Here’s a summary of Part 2.

Part 2 opens with Kurt being promoted to a Development Management position, taking over development of a legacy Data Hub system that interconnects and integrates many of Parts Unlimited’s mission critical systems. One of Kurt’s first tasks (with Maxine’s help) is to take down “technical debt” and automate things. This involves the third ideal, improvement of daily work. Parts Unlimited can’t get out of TWWADI syndrome, or “The Way We’ve Always Done It.”

For too long development teams at Parts Unlimited have tolerated work arounds, manual deployments and manual testing. Kurt does not want to end up like Nokia which fell from more than 50% of the worldwide smartphone business in 1998 to 0% in 2010. This fall is blamed in part on their failure to pay down technical debt.  For example, Nokia’s aging codebase slowed development—it took more than 48 hours for a simple build.

Next, software QA and testing is called out as another area for improvement. Maxine discovers that it may take weeks to get the Data Hub code tested due to the long backlog. To make matters worse, the urgent need to release the code reduces testing time. Additionally, QA uses a different ticketing system, is a location distant from the developers and has limited automated testing. Roy, the QA Manager does not like change and is afraid of automation.

Merging code at Parts Unlimited is like writing a screenplay with 50 writers: how to get everyone together in a room to merge their chapters in a short time, with no agreement on plot, characters, and the ending. Compounding this is the Change Management process in place that prohibits developers from pushing their own code into production. Operations must approve that change.

Maxime and Kurt look for quick wins and discover that containerization (i.e. Docker) allows code to work in both Dev and Test, making development and testing more efficient and much faster. By merging Dev and Test into one team, and using a CI server to do continuous builds and tests for every check-in, results in a significant improvement in the speed of the test and development process.

The next hurdle is deployment and how to get Ops ready and Data Hub into production. Parts Unlimited has a time-tested deployment approval review board, but after a great presentation, no approval. But, an exemption is granted allowing Data Hub to test, deploy and operate the code on their own. Another win is the movement of the Data Hub product manager to the Dev team. This eliminates a key bottleneck in the overall development process. Things start getting resolved quickly and the developers gain a better understanding of how the business works. All the retrospectives, learning, and hard work is paying off. And just in time for the Unicorn project roll-out in Part 3. Stay tuned….

Posted in Atlassian, Business Management, DevOps, Digital Engerprise Management (DEM, Uncategorized | Tagged , , | Leave a comment

The Unicorn Project, Part 1

By Dick Stark

At the end of last year and to much fanfare, Gene Kim released his follow-up book to the Phoenix Project, the Unicorn Project, a novel about “developers, digital disruption, and thriving in the age of data.” I encourage everyone involved in digital transformation projects, or just thinking about digital transformation to give it a read. The book is written in three parts and I’ll review one part at a time. Here is a brief summary of Part 1.

If you haven’t read the Phoenix Project, that’s OK. To summarize, the Phoenix Project takes place inside and around a fictional auto-parts company, Parts Unlimited, with retail outlets nationwide. The novel is about Bill Palmer, a recently promoted VP of IT and other company executives as they race to develop and bring on-line a new customer facing system, code named “Phoenix” which will attempt to close the gap between its arch-rival, that is “eating Parts Unlimited’s lunch.” Phoenix is a service catalog / retail site, closely integrated to its respective e-commerce channels. Along the way, nearly every IT calamity possible befalls the company. 

Part 1 of the Unicorn Project by contrast, reviews the early failures of “Phoenix” from the Dev rather than the Ops perspective. Here the main character is Maxine, a very successful architect/developer who has just been “exiled” to the Phoenix Project. After just a short time coming up to speed on Phoenix, Maxine quickly understands why things have gone so far south. For example:

  • Issues with help desk ticket system in trying to get approval to build a dev environment. Everything requires a ticket to be opened first. Lots of notifications spawn other tickets and soon Maxine gets hammered by a lack of disk space and no way to get new drives ordered quickly.
  • During a Dev and Ops meeting, no one can determine how many servers will be required. Getting a production environment requires 100s of tickets. The Firewall team alone needs four weeks to respond to all the tickets.
  • Kurt, one of the QA Managers is running a black market and gets everything that developers need like servers without having to submit tickets. (must be before AWS)
  • The QA Department doesn’t want to automate testing because of the cost savings which could reduce its next year budget. They feel developers should be kept away from testers anyway.

Later, Maxine joins Kurt’s band of “misfits” who plan surreptitiously on how to improve the development process. Along the way, they meet up with Erik, an efficiency expert/agile coach now working as a bartender. From Erik the team learns about the five ideals:

  1.  Locality and simplicity. Design locality in our systems and the organizations that build them. And we need simplicity in everything we do–it must be easy to do work.
  2. Focus, flow and joy. Work in small batches, ideally single piece flow, getting fast and continual feedback. These are the conditions that allow for focus and flow, challenge, learning, discovery, mastering our domain, and even joy.
  3. Improvement of daily work. Elevation of daily work over daily work itself. (aka continual improvement)
  4. Psychological safety. Where it is safe to talk about problems and mistakes.
  5. Customer focus. Ruthlessly question whether something actually matters to our customers and are they willing to pay us for it?

Stay tuned for Part 2 to see the five ideals in action…..

Posted in Uncategorized | Leave a comment

Report from the Atlassian Government Symposium

By Dick Stark

Last week, RightStar exhibited at the Atlassian Government Symposium, held at the JW Marriott in Washington, DC. This is one of a series of worldwide stops to promote Atlassian products, services and customer networking. Atlassian had a good turnout with more than 400 attendees.  We participted with Raytheon in a well attended session describing their DOD Success story. Here is a short summary of the Symposium.

Atlassian Government Update. Steve Elliott, head of product for Jira Align, opened the session with a short Atlassian update. According to Steve, Atlassian has 83% of the F500 and is in more than 1000 government agencies. And, the total worldwide customer count now stands at more than 150,000. Steve discussed the progress made with modernization and digital transformation projects using the Atlassian toolsets and highlighted two success stories—GSA and Sacramento County, California.

At GSA it is Jira, Confluence, and Trello that have made “hyper” collaboration possible, and the good news—Trello is currently going through the FedRAMP process with full certification expected later this year, and with Jira and Confluence to soon follow. Additionally, Sacramento County is using Jira Service Desk, Confluence, and Jira to scale ITSM and delivers seamless agile services and projects. The result is an improvement in efficiency, balanced workloads, and a freeing up of time and money to focus on the projects that matter most.

DevSecOps in the Air Force. Nicolas Chaillan, Chief Software Officer, USAF was up next as the keynote speaker. He described the journey to transform the modern Air Force into a software development powerhouse and the example he has set for the rest of the Department of Defense. What make the Air Force so special?

  • Containerization,
  • The DOD Centralized Artifact Repository (DCAR),
  • Baked in zero trust security,
  • Cloud and DevSec-Ops as managed services,
  • Standardized metrics,
  • Massive scale learning, and
  • Microservices architecture.

Nicolas concluded by highlighting some of the benefits of his Agile approach: consistent deployments, baked in security, auditability and compliance, everything as code, and declarative manifests and playbooks.

Atlassian Help Desk Replacement. Raytheon and RightStar rounded out the morning. We described the work involved to move a DOD agency off a legacy service desk to Jira Service Desk. The project was completed on-time and because the agency uses the DI2E hosted environment, there is no software cost. The JSD system provided far more functionality than the previous legacy system. We even wrote a script to archive all 88,000 old tickets and attachments into a read only Insight object schema. Other benefits included: extensive asset management, automated email, dedicated workflows for cyber security and integration testing, and tight integration between all systems and services.

Posted in Business Management, DevOps, Digital Engerprise Management (DEM, RightStar | Tagged , , , , , | Leave a comment

Report from the ATARC DevOps Working Group Launch

By Dick Stark

Last week I attended the ATARC DevOps Working Group Launch at the Department of the Interior. ATARC brings government practitioners together with Industry to discuss topics such as the role of security in DevOps, and cultural and organizational challenges to implementing DevOps. Attendance was excellent and so were the presentations. Here is a brief summary.

Dr. Brian Hermann, Services Development Executive, DISA opened the keynote by discussing the improvements that DISA has made in shortening the unreasonably long development cycle. Dr. Hermann relayed a story from not too long ago, about DISA’s typical dev cycle. “Back then, it took about three to four years, and what was delivered usually failed to meet the original requirements. Even if the needs were met, the reality was that it still took a long time.”

“DISA,” according to Dr. Hermann, “has made significant progress as a result of technical advances. Depending on the project it can take two to ten weeks to deliver software. Contractors, however, are being hamstrung by government cybersecurity practices. Security issues need to be overcome, or we will never get there.”

Dr. Hermann also mentioned the accelerated progress the Air Force is making with rapid deployments thanks in large part to Nick Chaillan, the Air Force’s Chief Software Officer, and his work at Kessel Run, where he has turned the Air Force into a software development powerhouse. The good news is that Nick is sharing his processes and tools, such as reusable containers with other DOD groups.

As a DISA contractor, RightStar is very familiar with DISA’s accreditation practices. RightStar’s DISA/Data Center Infrastructure Management (DCIM) project, is entering its fourth year. The system in now live and doing well, but it took more than two years for the Nlyte system to be granted an authority to operate (ATO) by DISA.

Interestingly a later panelist, Simmons Lough, Software Architect at the US Patent and Trademark Office discussed the progress he and his team have made in shortening the development life cycle.  At USPTO, Simmons has moved from two production deployments per year to two per day. And with $3.5B per year moving through PTO’s financial systems, rapid deployments are a very big deal. Simmons credits his success in part to the utilization of automated testing.

Derek Weeks, ended the session by promoting his DevOps Days events. Derek is the co-founder of All Day DevOps, an online community of 65,000 IT professionals. This year’s big event is planned for November 12 with more than 150 practitioner-led sessions across 5 tracks. Last year’s event drew more than 38,000 on-line practitioners.

Posted in Cybersecurity, DevOps | Tagged , , | Leave a comment

Agile Change Management

The Virtuous Cycle of Change

By Dick Stark

Last Week, RightStar emailed its customer base a Forrester Report, “Change Management: Let’s Get Back to Basics,” by Forrester Analyst Charles Betz. Given the focus on Agile and Continuous Deployment, Change is not the Change Management of old. According to Charles, “Manual documentation of changes, lengthy change-approvals delays, face-to-face change advisory boards, and the review of all changes are no longer essential or suitable to a modern change practice.”

Thanks largely to digital transformation projects over the last few years, organizations are realizing the value of rapid development projects. Change Management, which seemed like such a good idea ten years ago (and proven outage preventer), must be re-designed to fit the current agile environment. Robert Stroud, another Forrester analyst, told a story about a client’s traditional change process: “they’ve got a change that’s going to production, and as it goes into production, they’ve got 32 approvals, and Development developed the code in a week, and it takes 12 weeks to go through the change management approval cycle. And this is what’s annoying the living daylights out of people right now, it’s that the old change management is no longer relevant.” According to Forrester, here’s a summary of what to do:

Avoid the Vicious Cycle of Change. A slow change process leads to an increasing backlog of work and the temptation to clear the backlog through large and risky batches of change.

Adopt the Virtuous Cycle of Change (below). In the virtuous cycle, appropriate investment in change capabilities supports the desire for a faster change/release cadence with version control, traceability, rollback, automated testing, and continuous integration/deployment.

Keep the change process lean and focus on its intended outcome of reducing risk. Focus on difficult to reverse changes or changes involving systems with a history of instability.

Remember that communication and coordination is expensive. Consider the delay that might occur as more stakeholders are added to the approval process.

Love automation. Invest in modern test automation tools and processes.

Track the right change metrics. Target metrics that can result in avoiding the vicious cycle of change.

Separate changes from work orders. Use a service catalog portal for work orders. A Change Management request is the wrong place to request a work order.

Watch change management as a demand and risk signal at the aggregate level. Consider using a continuous delivery/ITSM dashboard to show the release calendar, the forward schedule of change, and major program milestones. This can display potential hot spots on the organization’s calendar, allowing the executive team to better spread demand across time.

Posted in Business Management, DevOps, Digital Engerprise Management (DEM, ITSM, Uncategorized | Tagged , , , | Leave a comment