It would be impossible for today’s software engineer to learn many facets of these technologies in detail without abstractions.  View in browser »
The New Stack Update

ISSUE 256: Have Abstractions Gone Too Far?

Talk Talk Talk

“Military hyper-masculine self-sacrificing [culture] not only leads to burnout but also just crowds a lot of people out of the space.”

Add It Up
ClickHouse Has Rapidly Rivaled Other Open Source Databases in Active Contributors

ClickHouse has come out of seemingly nowhere to rival Elasticsearch as the database-related open source software project with the most active contributors. Runa Capital’s Konstantin Vinogradov continued his ongoing work with open source metrics in a new blog post that analyzed GitHub data for 358 of the 735 databases listed on In his analysis, an active contributor is defined as anyone who makes a commit within a 12-month period of time.

ClickHouse originated out of Russia’s Yandex and its commercialization is being led by Altinity. ClickHouse is column-oriented and allows for analytics reports to be generated using SQL queries in real time. ClickHouse’s rise in popularity began in 2016, which happens to be when Apache Spark peaked. TiDB also had more than 200 active contributors in 2020. CockroachDBPrometheusMongoDB, and TrinoDB were in the second group of contenders with 150–170 active contributors.

Skeptics will complain that some of the aforementioned data sources are no longer governed by open source licenses, so shouldn’t be called open source. Let’s put that debate aside for the moment. Perhaps a recent controversy is partly responsible for the drop in contributors seen in the chart. A closer look at the graphic shows that other databases have also seen drops in active contributors after they made similar changes in their licenses. However, those drops were short term in nature.

Having a lot of developers regularly committing changes to a project shows that people and probably several companies are investing in a technology. It is not a proxy for actual adoption, nor does it mean there is deep community engagement in the development effort. That said, an assessment of the data shows the robust number of developers working on projects at key junctures of the new data pipeline.

What's Happening

The world has certainly changed during the past year. The paradigm shift in how we live and work that started with the pandemic has brought an even stronger push by numerous organizations to scale operations across cloud native and distributed environments. This has largely been in response to the surge in remote workers and developers whose distributed data and application needs have scaled accordingly.

In this The New Stack Makers podcast, hosted by Alex Williams, founder and publisher of The New Stack, Vaibhav Kamra, chief technology officer of Kasten by Veeam, discussed the changes he has observed, and ultimately, the lessons learned during the past year. During this time, Kasten has provided the necessary platforms for application and data management that organizations rely on to scale across Kubernetes applications.

Vaibhav Kamra CTO of Kasten on Cloud Native Lessons Learned During these Pandemic Days

Have Abstractions Gone Too Far?

Abstractions have been with us since the birth of computer science. They provide a way to simplify the complexities of working with computers. The first assembly languages simplified how programmers fed instructions to the machine. Today we are awash with them: Kubernetes, React, and the Linux Kernel are all abstractions. It would be impossible for today’s software engineer to learn many facets of these technologies in detail without abstractions.  

But have we gone too far in developing abstractions, making their use more difficult than the things they were designed to simplify?

This was the question raised by Venmo Systems Engineer André Henry at the Usenix SRECon last December. Abstractions took us from a one-to-one relationship with software to a “multilayered configuration and deployment” model. 

“It’s almost like we’re all starship captains now. We have become managers of ecosystems of complexity,” he told the virtual crowd. “If there is a problem with the application, we now have to start peeling back all these layers to figure out what’s causing the problem. Is the abstraction broken? Is the layer it's abstracting broken? Is the other layer broken? An individual software might now understand all the layers covered by the abstractions.”

How much of this do we all need, he asked rhetorically, pointing to the infamous Cloud Native Computing Foundation’s voluminous landscape chart of cloud native technologies. For instance, is the container so popular because it solved the problem of dependency management? Perhaps more effort should have been applied to improving the packaging management tools themselves. “If your abstraction is compensating for a problem in another tool, should you write an abstraction for that, or should you solve that in the other tool?”

And these fancy abstractions may not even be the best approach to solving the problem at hand. Take Kubernetes, for example, which offers the true benefit of fast scaling of resources. But do all web services require such rapid scaling? He offered an example from a former job at an educational company. There, the company needed to ramp up its services each morning for the school day, and then ramp it down at the end of the day when the students go home. While this could have been handled easily by Kubernetes, Kubernetes wasn’t really needed at all, given that the workload was predictable. An easier approach would be to pre-scale, or pre-schedule, a set of pre-baked images for these times. No K8s required. 

“Think about the goal of the problem you're solving, like what are you trying to accomplish and if this is the most efficient way and sustainable way. All abstractions are going to have a life. They have to be managed right,” he advised.

Google Grapples with Ethical AI

When Google fired Timnit Gebru, technical co-lead of its ethical artificial intelligence team last year, amid controversy over a paper she was co-authoring with academics and other Google researchers, the company unleashed a firestorm of criticism of the bias apparently shown by its own management, its apparent defensiveness about potential flaws in the AI underpinning its core search service.

Gatsby v3.0 Offers Incremental Builds to Speed Web Deployment Times

Gatsby, the creators of the open source JavaScript-centric website framework GatsbyJS, held its user conference GatsbyConf 2021 this week, where it launched Gatsby v3.0, which now comes with some highly desired features such as incremental builds and faster image handling.

Shameless Developers: How to Manage Imposter Syndrome within Your Team

We see a lot of Imposter Syndrome in the IT industry, where highly skilled professionals still are plagued with self-doubt regarding their personal and professional accomplishments, where they fear being exposed as a fraud, despite actual success. In this contributed post, Truss’ Everett Harper and Peter Karman revealed some useful strategies for identifying feelings of shame within your dev team, and mitigating those issues before they start to impact your teams’ performance.

The New Stack Makers podcast is available on: — Pocket CastsStitcher — Apple PodcastsOvercastSpotifyTuneIn

Technologists building and managing new stack architectures join us for short conversations at conferences out on the tech conference circuit. These are the people defining how applications are developed and managed at scale.
Best of DevSecOps: Trends in Cloud Native Security Practices

This is the first in a new series of anthologies that assemble some of our best articles on a trending subject, paired with our editors’ insightful analysis to frame the bigger picture. These exclusive ebooks help developers, architects, operators and management go in-depth, quickly, on hot topics in at-scale development and management.

In this ebook, we explore how security practices are now being integrated into the development process, as well as the build pipeline and runtime operations of cloud native applications. You’ll learn more about:

  • How DevSecOps enables faster deployment cycles.
  • Why DevSecOps is necessary for cloud native architectures.
  • The challenges and benefits of DevSecOps practices.
  • The new role of developers and operators in security.
  • How to measure DevSecOps success.
  • Tools and best practices for adoption.
  • Emerging trends to pay attention to.
Download Ebook
We are grateful for the support of our ebook sponsor:

Copyright © 2021 The New Stack, All rights reserved.

Want to change how you receive these emails?
You can update your preferences or unsubscribe from this list

Email Marketing Powered by Mailchimp