Research has shown that there is a correlation between the environmental effects from climate change and the development of cancer. In response, a growing number of Flatiron employees are finding ways to apply their time, talents, and creativity at work to the fight against climate change, as part of an employee-led initiative called the Green Task Force. Through grassroots efforts to reduce waste, and Flatiron's carbon footprint, these employees embody Flatiron's value to "do the right thing" - and push forward Flatiron's commitment to corporate sustainability.
In March 2022, Flatiron held a climate change focused hackathon, and a group of employees set out to reduce energy usage by cleaning up Flatiron's cloud computing infrastructure. As computing power moves increasingly to the cloud, it can be less obvious when a machine is running unnecessarily than in the days when all servers were physically hosted in a company's office - but out of sight needn't mean out of mind.
Flatiron holds 3 of these company-wide hackathons per year, where for three days at a time, employees are encouraged to set aside their regular projects and teams, and come together to work on something that excites them. Each hackathon is themed and judged, with participants encouraged to submit projects in categories like "Patient Advocacy and Experience", "Flatiron @ Scale", or the rotating category for the current theme.
Led by Angus Glover Wilson, Senior Director of Corporate Strategic Operations, and Matt McNaughton, Principal Software Engineer, the hackathon team set out to find $75,000 of annual cloud infrastructure savings, motivated by a commitment from Flatiron leadership to donate 20% of those savings to Clean Air Task Force (CATF) - an organization working to address climate change on a global scale by advocating for emissions regulation, system-wide policy change and the promotion of low-carbon technologies.
The team reached their goal within the first few hours of the hackathon, prompting Flatiron leadership to raise the donation-matching target from 20% of 75,000 to 20% of $300,000. Over the three days, the team exceeded their increased goal, resulting in just over $1,000,000 of annual cloud services savings and a $60,000 donation to CATF.
This incredible achievement built on years of groundwork from Matt, Angus, and the Green Task Force (GTF) initiative. In 2018, a diverse group of folks from different teams across the business j oined efforts to create the GTF, with an initial focus on reducing the carbon footprint of Flatiron's New York office. In 2021, when the office began reopening after the COVID-19 pandemic, the GTF came back with renewed enthusiasm and an announcement of what they had used the time away to focus on: certified carbon neutrality for 2021 (and a plan to continue this for each year going forward, starting with 2022, which was just finalized in December).
Behind the scenes, the GTF had proposed and developed a partnership with Green Places , enabling accurate estimates of the NY office's carbon footprint, identifying ways to reduce it, and purchasing carbon offsets to counteract it.
In partnership with Green Places, the GTF identified Flatiron's cloud usage as being just over 1% of Flatiron's overall 2021 carbon footprint. While this is a small percentage, there's an important place for it in the task force's portfolio of carbon reduction projects; carbon reduction is a marathon, not a sprint, and the GTF aims for a balanced mix of projects.
In order to provide a starting point for other individuals or organizations building efforts to reduce their carbon footprint, the GTF shared the factors they use to evaluate projects:
-
Direct carbon reduction impact
-
Timeline (sustained effort vs a quick win)
-
Company-wide visibility
-
Alignment with other corporate goals
-
Secondary impact (through committed donations, or other flow-on effects)
The cloud services hackathon project would be evaluated as a quick win, with high visibility, strong alignment with other corporate goals, and high secondary impact - the kind of project that generates enthusiasm and keeps momentum going in order to support projects that have a higher reduction impact, but a slower timeline.
Jason Cannavale, VP of Software Engineering, expressed it best: "I cannot take my eyes off this [slack] channel, this is an amazing effort and progress in just a couple of days". Gabriel Raines, Network Engineer, highlighted some of the team-building benefits: "one of the things I've loved most about watching this hack unfold is seeing so many different teams working together to make it happen" .
Research increasingly shows the direct relationship between the environmental effects from climate change and the risk they pose for the development of cancer, disruptions to care, and survival. We believe Flatiron's commitment to the environment is vital not only because it is "the right thing", but because taking action on climate change furthers our mission to improve and extend lives. Finding ways to honor this commitment to sustainability, sustainably, gives employees opportunities to collaborate in new ways as we work towards another of Flatiron's core values - "solving problems that matter".
Allison Candido, VP of Software Engineering, had this takeaway from the hackathon: "One of the coolest things about seeing the spreadsheet is that there isn't one silver bullet of something to delete that contributes all the savings...it's more of a collection of waste being cleaned up. To me it's a great example of 'eating the elephant one bite at a time".
For those looking to eat their own elephant, read on for some technical insights and tips that the team put together from their experience with this project.
When looking to reduce Flatiron's cloud usage, there were 2 primary categories the team identified:
-
Unused resources (e.g. machine images, compute instances, database volumes, etc) that could be deleted
-
Resources that could be optimized or shrunk
Two members of the team, Sofia Nieves, Senior Site Reliability Engineer; and Marc Budofsky, Senior Director of Security Engineering, investigated the following signs that signaled a resource might be unused:
-
No logs showing usage activity for a resource, both in Splunk (Flatiron's monitoring system of choice), and on the resource directly
-
Low usage signals in CloudWatch graphs for the resource
-
The team assigned to the resource no longer exists, and the team that took over their mandate is unaware of the resource
-
Resource age indicates it hasn't been cycled in a long time
-
A large number of resources that indicate they are temporary or for debugging coming from the same service/team
-
There are no associated running instances
Andrea Bolivar, Senior Software Engineer; Sudhir Koduri, Database Reliability Engineer; and former employee, John McCall, focused on the second category – identifying these signs that a resource might be under-utilized:
-
CPU and Memory usage are low
-
Resource age/type pre-dates a newer more efficient option (e.g. gp2 vs gp3)
After the hackathon team identified a resource that looked like a promising target, they reached out to the owning team at Flatiron to validate their findings. Once they were confident the resource could be safely removed or modified, the hackathon team sent out changes to Flatiron's infrastructure code (usually Ansible or Terraform) for peer review.
The hackathon team was committed to making sure this was not a one-time effort, but rather a sustainable change that would result in ongoing savings. Through the hackathon, they identified the following ways to build resource management into the development lifecycle:
-
Sofia's team, Enterprise Engineering, is planning an overhaul of their AMI creation process to include lifecycle management.
-
Marc identified a potential future hackathon project: modifying AMI Cleaner to run on a regular cadence and alert teams to actions they could take.
-
Andrea created a monthly cleanup job for her team, HC Platform, to automatically clean up their older AMIs.
Although early in the journey, the Green Task Force is buoyed by increased participation after the Green Hackathon in March, and has a number of initiatives underway to continue driving down Flatiron's carbon footprint:
-
a greener corporate flights policy
-
reducing the environmental impact of conference marketing materials
-
sustainability knowledge sharing sessions
-
a grassroots initiative to reduce food waste in the New York office
-
purchasing carbon offsets to achieve neutrality for 2022, following a 2022 carbon footprint assessment
-
offering charitable donations as an alternative option to physical items for employee swag
It's been energizing and inspiring to see the time and creativity our coworkers are investing to make Flatiron a more sustainable organization, and we're excited to share more updates this year!
