This blog presents a case for Developers, Operation Engineers, Architects and other such roles to accelerate building up their application and Cloud Infrastructure using IaC. And we can proudly claim this because our parent consulting services team (Avyan Consulting Corp) has done it and have realized the benefits of it. For information about publicly available reference architecture is available here – Azure Blueprint – Payment processing solution for PCI DSS enablement.
This blog will not be going into the details of the Solution, but more on how the teams were structured, the collaboration that went on and the light-weight processes that got followed to build the solution from 3 different timezones. The RA solution that finally shipped is depicted below.
Defining Reference Architecture (RA)
A reference architecture in the field of software architecture or enterprise architecture provides a template solution for an architecture for a particular domain. It also provides a common vocabulary with which to discuss implementations, often with the aim to stress commonality. A software reference architecture is a software architecture where the structures and respective elements and relations provide templates for concrete architectures in a particular domain or in a family of software systems.
Infrastructure as code (IaC) is the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. Both physical equipment such as bare-metal servers as well as virtual machines and associated configuration resources are called “infrastructure”, although they have nothing to do with actual infrastructure. The definitions may be in a version control system. It can use either scripts or declarative definitions, rather than manual processes, but the term is more often used to promote declarative approaches.
Infrastructure as code approaches are promoted for cloud computing, which is sometimes marketed as infrastructure as a service (IaaS). IaC supports IaaS, but should not be confused with it.
Before we discuss about IaC, here are some high level bullet points that we thought were problematic if IaC was not the methodology that we had followed.
Need for Consistency in standing up multiple environments :
One of the first principles we had that we want to be consistent when we shipped our RA to our customers environment. Our team was developing in couple of Azure subscriptions and this entire infrastructure had to be tested into a subscription where we won’t have access. Repeat-ability and consistency were design constraints to us.
Delivering value in an Incremental Development Model :
We were following the Agile development methodologies where incremental value gets delivered on a weekly or bi-weekly basis. Our Internal Architects and Our Customers would review and provide us a feedback almost on a weekly basis.
Collaboration between team members is impossible without a Source Control based IaC :
Our pilot customer and our development teams were on 3 different continents. We needed a way to consistently keep high collaborations between the team members, but still meet the communication barriers due to multiple geographies.
Full Software Development Life Cycle would have not been possible:
Daily check-ins, daily testing and daily internal feedback was the stated goals from the very beginning. Our pilot customers were engaged on a daily basis as well. Our needs was to ensure that SDLC is carried out and it is still very Agile.
Impossible to document every single configuration and integration points at the outset :
Our Architects and our Pilot customers had difficulty to document all integration points and end-to-end configurations. We needed a way to store and manage configurations and integration from day-1. Source control mechanism and SDLC methodologies helped us to find the answers.
Time it takes to stand up an environment would have been in days :
Standing up and configuring an environment with more than 15+ Azure resources would take days even if proper documentation was provided (and documentation becomes stale after sometime). We wanted to stand up environments within few hours.
Not possible to verify :
After deploying such a complex environment, we wanted for our pilot customers to be able to verify the configurations for each resources. Our pilot customers, Architects and Internal Operations teams needed a mechanism to run their tests and verify if very quickly. Azure ARM templates provides Idempotency that helps a lot to meet the goal.
Supportability is a constraint:
Our pilot customers had difficulty to report the issues they were facing during the deployment. Open sourcing IaC gave the ability for stakeholders to pin point the issue they were having, many cases they could support themselves (self-service) by fixing the issue and sending us the Pull Request (PR)
Recommendations / Solutions
All the problem listed above and more will be solved using IaC. IaC helped us achieve consistency irrespective of the environment to which they were deployed. It also allowed us to make incremental changes and additions to our existing reference architecture due to its idempotent nature. Idempotent means templates can be deployed over and over again without affecting the current state of resources. This reduced lots of efforts for our Architects to fix bugs or add any new configuration to an environment.
Implementing a Source Control system like GIT to manage the codes helped us collaborate people around different regions. This not only enabled us to track the code version but also allowed our Architects and Operations team to work on different resources and configuration simultaneously thus allowing us to reduce the time for building this reference architecture.
With Infrastructure now managed as Code, we need not had to document every configuration but only steps to deploy those codes on a particular environment. Due to which our Architects did not had to document the configuration in detail and our Pilot customers were saved to go through lengthy manuals to stand up such environments.
Our team has shipped very complex reference architecture in collaboration with multiple teams distributed across 3 different continents using the above recommendations and principles. Our verification mechanisms are also automated and the verification rules have also been shipped into our product Cloudneeti available in Azure marketplace.