Archive for category DevOps

What does DevOps mean in the DBA World?

What does DevOps mean in the DBA World?

Since April 2015, I had been working as a database engineer in various DevOps-oriented team. I firmly believe that we should all transform from being a DBA to a Database Engineer. We can no longer be just the administrators of our databases, we need to be part of the entire deployment lifecycle. Being an engineer means we need to be able to code and automate our database deployments. We need to be able to read application code and understand system internals. Gone are the days where we were part of a massive 25 man DBA team and we care only about databases. Anything above or below the database tier is not our problem.

With DevOps/Microservices, we will see teams being structured around features/services. For example, for a search-box feature, you have 4 developers, 2 DevOps Sysadmin and 1 DevOps DBA. With companies like Facebook/Google adopting such team structure, more companies should follow suit.

I has always been curious about how other DBAs adopt DevOps and have tried to understand from others during conferences or meet-ups. There are my observations from DBAs across London and Singapore:

  1. Chef/Puppet/Ansible are the most popular Configuration Management(CM) tools
  2. Most engineers uses CM to manage the database configuration files like postgresql.conf or my.cnf
  3. Very few engineers uses CM to manage database schema and objects
  4. Very few engineers uses CM to manage database users and roles
  5. Very few engineers automate or semi-automate any form of database minor version upgrades or rolling restarts
  6. Even fewer engineers perform automated database recovery testing

Personally, I’m still on the journey to accomplish the last 4 points. They can be achieved having more logic in your CM tool, rundeck and a combination of both. However, all the above points are the technical aspects.

What I agree fully with Chris Dodds, is the change of mentality. Automation and processes should be good enough to prevent an engineer from bringing down a service. If an engineer is able to delete a production database directory accidentally, then the process has a problem, not the engineer. I have worked in or with larger MNCs that has a very strong blaming culture. Any form of failure is not accepted and you will get the wrath during post-mortem meetings.

This kind of culture has a saying in Singapore: The more you do, the more mistakes you make. Thus, people start to do the bare minimum. This is not ideal in DevOps, it promotes mediocrity.

It’s a paradigm shift for us in the new era of DevOps. Our job should not be creating database objects or managing users. We should be spending most of our time in automating/improving the processes, testing/benchmarking new database version or benchmarking different database technologies. There should be as little manual work as possible (Google call it TOIL). Fault blaming should be thrown out of the window.

In the heart of DevOps, among Microservices, Docker, Containers, Kubernetes, Chef, Vagrant and other hundreds of other technologies/buzzwords, it is a mindset change that’s the most important. Without a proper DevOps culture, the technology doesn’t matter, it will only beautify your resume.

This blog post was inspired by this and this.

Wei Shan

Leave a comment

Puppet, a tool in the DevOps engineer’s Arsenal

Puppet, a tool in the DevOps engineer’s Arsenal

What is Puppet?

Puppet is a open-source configuration management tool. Adoption of DevOps means that you will need some sort of configuration management tool, else you can’t automate your software delivery. Puppet is one of the many configuration management tools out there. There are many others like Chef, Ansible or Salt. There are already a HUGE number of discussion online regarding Puppet vs Chef vs Ansible vs Salt so I’m not going to talk about them over here.

What is the value of Configuration Management tool?

The power of configuration management tool will be obvious when you have hundreds and thousands of nodes to manage. When you have less than 50 nodes, you can decide the purpose of each node and manually install the required packages on it. If there’s a patch, you can just install the new fix easily. If there’s a need to disable SSLv2, you can just login to each of the server to disable them.

  • What happens if you have 1000 nodes, and you need to disable SSLv2 on all of them?
  • What if you need to update the gcc packages on 40/1000 nodes?
  • What if the annoying IT auditors want to audit your security patches in your environment?

Why do I like Puppet?

I prefer using Puppet because the following reasons:

  • Non-programmer friendly. You don’t have to write any OO code. Puppet DSL is a declarative language. You just have to specify the end state.
  • Gentle learning curve. The Puppet DSL is quite intuitive.
  • Largest community around. This is important because you can get help easily via Google
  • Additional plugin modules are available if you want to provision any software using Puppet. (For etc, Oracle)

How is using Puppet going to help me as a Database Administrator?

The typical databases to DBA ratio is about 250:1. The volume and complexity of the databases you manage is growing. On top of that, you’ve got demands from the business to do more and faster, with the same amount of resources and manpower. Now, with additional Puppet modules, you can actually manage the database level configurations using Puppet.

For example;

tablespace {'PIO_DATA': 
ensure => 'present', 
bigfile => 'yes', 
datafile => 'pio_data.dbf',
size => '200G', 
logging => 'yes',
autoextend => 'on', 
next => '100M', 
max_size => '12288M',
extent_management => 'local', 
segment_space_management => 'auto', }


I believe that using configuration management tool to deploy databases is a new trend for DBAs. It allow us to manage thousands of databases easily. However, there certain caveats to it. The Puppet oradb module is still quite new, there are still certain bugs that needs to be iron out. That said, I still believe that it is another alternative to to building your own DBaaS using Oracle OEM.

It is something I believe DBAs should explore and determine if it’s suitable for your own environment.

Wei Shan