Archive for category Database Reliability Engineering

What does DevOps mean in the DBA World?

What does DevOps mean in the DBA World?

Since April 2015, I had been working as a database engineer in various DevOps-oriented team. I firmly believe that we should all transform from being a DBA to a Database Engineer. We can no longer be just the administrators of our databases, we need to be part of the entire deployment lifecycle. Being an engineer means we need to be able to code and automate our database deployments. We need to be able to read application code and understand system internals. Gone are the days where we were part of a massive 25 man DBA team and we care only about databases. Anything above or below the database tier is not our problem.

With DevOps/Microservices, we will see teams being structured around features/services. For example, for a search-box feature, you have 4 developers, 2 DevOps Sysadmin and 1 DevOps DBA. With companies like Facebook/Google adopting such team structure, more companies should follow suit.

I has always been curious about how other DBAs adopt DevOps and have tried to understand from others during conferences or meet-ups. There are my observations from DBAs across London and Singapore:

  1. Chef/Puppet/Ansible are the most popular Configuration Management(CM) tools
  2. Most engineers uses CM to manage the database configuration files like postgresql.conf or my.cnf
  3. Very few engineers uses CM to manage database schema and objects
  4. Very few engineers uses CM to manage database users and roles
  5. Very few engineers automate or semi-automate any form of database minor version upgrades or rolling restarts
  6. Even fewer engineers perform automated database recovery testing

Personally, I’m still on the journey to accomplish the last 4 points. They can be achieved having more logic in your CM tool, rundeck and a combination of both. However, all the above points are the technical aspects.

What I agree fully with Chris Dodds, is the change of mentality. Automation and processes should be good enough to prevent an engineer from bringing down a service. If an engineer is able to delete a production database directory accidentally, then the process has a problem, not the engineer. I have worked in or with larger MNCs that has a very strong blaming culture. Any form of failure is not accepted and you will get the wrath during post-mortem meetings.

This kind of culture has a saying in Singapore: The more you do, the more mistakes you make. Thus, people start to do the bare minimum. This is not ideal in DevOps, it promotes mediocrity.

It’s a paradigm shift for us in the new era of DevOps. Our job should not be creating database objects or managing users. We should be spending most of our time in automating/improving the processes, testing/benchmarking new database version or benchmarking different database technologies. There should be as little manual work as possible (Google call it TOIL). Fault blaming should be thrown out of the window.

In the heart of DevOps, among Microservices, Docker, Containers, Kubernetes, Chef, Vagrant and other hundreds of other technologies/buzzwords, it is a mindset change that’s the most important. Without a proper DevOps culture, the technology doesn’t matter, it will only beautify your resume.

This blog post was inspired by this and this.

Regards,
Wei Shan

Leave a comment