Q&A with our Head of Web Operations, Chris Kendall.
The term WebOps stands for a suite of technical tasks and prerequisite skillsets applied to keep a site performing at its highest potential.
As a cross functional team, Entermedia has front end specialists, back end specialists, design and content specialists, and so on. Like spokes on a wheel, they make your project well rounded.
But the operator in the middle—the hub that turns your wheel—that’s our WebOps specialist, Chris Kendall. Responsible for performance tuning, host optimization, infrastructure updating, automated testing, bug fixing, and more, Chris’ speciality is understanding how the whole system fits together, and smoothing over any friction points.
“Certain personality traits help, I think. I like to solve puzzles and find efficient solutions. I learn new stuff every day.”Chris Kendall, on being a web developer.
How much of your day involves web operations?
There isn’t a typical day for me at Entermedia, which is what keeps it interesting – not mundane or routine.
I wear many hats (owner/operator, team lead, developer, DevOps, QA lead, etc). I have to evaluate each day’s tasks and plan specific slots of time for each set of tasks and stick to the schedule.
The key to managing it is having sprints planned far enough ahead in advance.
How did you get into web development and WebOps?
My introduction to web / internet started in college in the early 1990s. I was a biology major and spent a lot of time in the labs. At first, we were using Gopher protocol to share results with other universities. Then HTTP took over in our labs, and that became the future of the internet. I gravitated to the computers and the networking. I seemed to be one of a few of us who had the patience to reboot after every change we made to get them communicating.
My career in technology started in QA. I started as a beta game tester. We developed and tested games on Sony and Nintendo consoles and PC. Our bug tracking process and software was rudimentary. Getting paid to play video games sounds like the ideal job, right? Walking into the same dark room, hearing the same title screen audio loop because you wrapped a rubber band around the controller so the ‘x’ button repeatedly registers just to try to reproduce a possible bug you found before you left the night before gets old… fast. I preferred the maintenance of the testing and reporting apps. Working with developers and testers to streamline the testing, reporting, and bug/feature lifecycle suited me well. I transitioned to doing more level design and intranet development/maintenance.
Sounds like WebOps found you, versus you looking for it?
That’s right. Eventually, I got recruited to be a QA Engineer at an IT start-up. This was bridging the gap between online news content and the original print newspaper. Our QA environment consisted of every possible physical machine with every possible browser configuration. We used GHOST to re-image each box after each testing iteration and/or bug fix. Sometimes several times a day. We could only dream of what VM-ware would become. Technology such as Docker would have made our lives so much easier.
I moved from QA Engineer to Sr. QA Engineer to Coldfusion and PHP Developer. I developed their Flash/ActionScript NewsReader. Wearing so many hats and working in so many areas of each company was a good training ground. I learned how to coordinate and automate between departments. Coming up with processes to improve productivity, I kept adding knowledge and experience that I use today.
How would you define ‘WebOps’? How is it different from ‘Dev Ops’?
I think Pantheon defines it best:
“WebOps is a set of practices that facilitates collaboration and automates processes to improve the productivity of the whole web team from developers and designers to content editors, stakeholders, and more. The result is cross-functional web teams empowered to develop, test, and release website changes faster and more reliably.”
The line between WebOps and DevOps gets blurry fast. I don’t worry about the definitions. The work requires being able to zoom in to the finest detail, and then zoom out to to see the big picture.
What does a WebOps person need to have in their ‘bag’?
You need a pretty big bag. And the contents will constantly change.
You’ll need to understand the entire development life cycle of the particular projects they are working with.
You’ve got to understand the entire stack of languages and environments they use to build, deploy, and host/deliver. As well as a myriad of third party tools such as Jenkins, Docker, K8s that monitor and aid in the build, test, and deliver workflow.
These are the tools I’m using these day:
What common mistakes do you find when we take over an existing site?
The most common mistake or bad practice I find when taking over both WP and Drupal sites is decent PHP developers re-inventing the wheel because they don’t know the CMS they are working in. They take just enough time to figure out the templating overrides and go no further.
I find many cases where they write custom templates with tons of code for every page in the site. In most cases, whatever they were trying to do could have been done via configuration using core functionality.
Other times, the functionality typically could have been achieved by writing a simple plugin or module. One might even exist, ready to hook into existing core or plugin/module functionality. Instead, their PHP is typically deprecated when the next php version comes along. Or breaks with the next WP or Drupal core update.
Being open source, are Drupal and WordPress more or less prone to problems?
The most common problem we come across is a neglected CMS.
Drupal and WordPress are attractive to developers and site builders because they give us a jump-start. The basic needs are available ‘out of the box’ with core functionality and the addition of user contributed modules or plugins. They are open source and extendable to meet unique requirements, which is why they are popular.
Vulnerabilities in open source core and plugins are not unusual, though. Also, PHP functions in core and plugins/modules become deprecated as PHP matures. The community reports and fixes bugs and security issues all the time.
Thus, WordPress and Drupal, while easy to work with, are not ‘set it and forget it’.
You must stay up to date with core CMS code and contributed modules and plugins. If you don’t, deprecated functions and poor code in plugins will return errors and break your site. WordPress sees a lot of hacker activity. Automated hacker scripts scour the internet for sites with outdated WordPress sites. They are looking to exploit known vulnerabilities and turn you site into zombies for their own nefarious purposes.
Talk about what’s involved in performance tuning.
Performance tuning is not only about analyzing performance and fixing bad code. Experience teaches us sensible configuration ‘recipes’. Caching strategies, quality of hosting, and hosting features are part of it.
WordPress and Drupal come with a multitude of site building features and code hooks for developers. Unfortunately, these great tools are just enough enough rope to hang yourself with. You can build a view in Drupal that will bring the entire site and host to its knees. A developer needs to know enough about WebOps to avoid these things.
Walk us through a typical WebOps performance tuning process.
- First, enable logging. The first thing I do is turn on MySQL slow query log and turn php logging up for a couple days. You’ll isolate the obvious issues through normal site use.
- Next, review New Relic. I highly recommend using New Relic’s platform to help you understand your entire environment.
- Then, begin load testing. With MySQL and PHP logging on, it’s time to load test. I use Locust. Locust is an open source load-testing tool written in Python. It lets you write tests against your web application which mimic your user’s behavior. Then run the tests at scale to help find bottlenecks or other performance issues. Start with a normal ‘swarm’ of locusts to mimic ordinary user behavior. Then increase the load to simulate unreasonable traffic.
- Note: you may need to let your host know you are doing load testing! Don’t end up locking your IP out and then be unable to access your site.
- Using New Relic and PHP/MySQL logs, isolate the problem queries, code, environment settings such as memory and time limits, caching settings, and so on. Fix and repeat testing.
- Finally, implement Varnish Cache and CDN if possible and perform load testing again.
Why does Entermedia recommend Pantheon for Drupal and WordPress hosting?
Pantheon delivers industry leading WebOps platform features built for Drupal and WordPress environments. Developers, site builders, and site owners can benefit.
- Security Updates – Pantheon manages WordPress and Drupal core updates. And non-security updates, as well.
- SSL – Pantheon includes fully managed and free TLS 1.2 HTTPS encryption with all hosting plans. No need to hassle the client to generate CSRs, renew and install certificates.
- Caching – Pantheon distributes everything across their Global Content Delivery Network (CDN). By default, they provide page caching, asset caching, and HTTPS certificates. If caching gets in the way, they allow you to bypass caching at all levels.
- Development Workflow – Every Pantheon site comes with three environments: Dev, Test, and Live. Each environment runs a version of the site on its own container.
- Multidev Environments – Multidevs are a branch off of the main Dev > Test > Live workflow. Clone them from any of the other Environments. Spin up a multidev with the click of a button. Use them to develop new features, fix bugs, or allow site-builders to experiment. Obviously, this is much better than experimenting in the live environment. Merge the changes into the main workflow via the interface or Git command line.
- CI/CD – Hook into Quicksilver platform workflows to automate your Pantheon WebOps workflow.
- Repo Options – You can use Pantheon’s repository exclusively or you can use GITHUB or Bitbucket.
If a site isn’t using Drupal or WordPress, what hosting setup might you recommend?
You can’t answer that, without knowing what the site does, what technologies it uses, what resources it will need now or in the future, etc.
Quick aside: For simple sites, such as landing pages, WordPress makes sense. It’s often quicker to spin up and deploy a WordPress site and create one page as the homepage than it is to write out the HTML by hand. For sites where there won’t be any custom plugin development or integrations, I recommend clients use Bluehost. It has managed WordPress, so you won’t have to worry about keeping your Core WP up to date.
But for very simple static—as well as complex sites/applications—I recommend AWS. You can spin up an entire complex application using AWS free tier and scale up to any level as demand increases. Or you can serve a static site using s3 and Cloudfront.
Are there differences between how Drupal and WordPress roll out updates?
Not really, as far as the way core updates roll out. I would point out the differences with frequency and reliability of updates between Drupal and WordPress.
For one thing, Drupal has stricter testing protocols. Each contributed module must pass these before they become available. These tests ensure that the module adheres to naming conventions, security protocols, and the like. They run each time code gets committed to a module.
On the other hand, WordPress plugins do not get the same attention and rigorous testing before they are available to the general public. So be careful which plugins—free or premium—that you use on your site.
As vulnerabilities in WP or Drupal core are exposed and fixed and security patches released, modules and plugins must also be updated. If not, the vulnerabilities will still exist and your module/plugin may break your site.
We have automated nightly checks for core and contributed modules and plugin. If they detect updates are available, these get applied, tested, and deployed.
Talk about the importance of automated testing.
Automated Testing is an integral part of Continuous Integration and Continuous Delivery (CI/CD).
We have implemented different levels of automated testing in different environments at Pantheon. Pantheon’s workflow (Dev > Test > LIVE) and Quicksilver hooks helps us achieve this.
- In MultiDev environments, when code is committed, basic visual and functional regression tests run.
- These can be tuned to test only the specific areas of the site you may be working on.
- Before merging into the main workflow, the entire test suite can be manually triggered.
- When code updates advance into the TEST environment, another round of visual, functional, and performance tests will run.
Our automated WP and Drupal update process at Pantheon leans heavily on automated testing. If any visual regression, functional regression, or performance tests fail, the updates are not merged into the main workflow. Pass/fail and status alerts send to our slack channel. Only if they all pass, do they roll into the main workflow.
Automated testing is not foolproof. The functional test results and reliability are only as good as the tests themselves. You must create reliable tests and update them as features change. Add tests for any new features added to your application.
What automated testing tools does Entermedia use?
- BackstopJS for visual regression testing
- Lighthouse for performance comparison
- Jenkins for orchestrating tests
- Custom BASH scripts
What common sense advice do you have for site owners on the following topics?
- Spend the time and money up front or you’ll spend more later. Take the time to prevent hacks because it takes longer to clean them up.
- Automate updates, tests, and deployment. This will ensure that vulnerabilities get fixed as soon as they are available.
- If not automated, apply updates to core or contributed module/plugins manually. As soon as they are available.
- Do the research on contributed modules and plugins. They are not all created equal.
- SEO is leaning on performance more and more. If your site does not perform well on all devices and screen widths, your SEO mojo drops drastically.
- An expensive, great looking website is a waste if no one can find it. If they do find it, but it doesn’t look good, doesn’t load quickly, or isn’t accessible they’ll leave.
How to report a bug
- Too much is better than not enough information
- What did you do before and after you got to the page or functionality where you found the bug?
- Have you been making any other changes lately? Tell us about them, whether you think they could connect with the bug, or not.
- What browser were you using? Try it in every browser you can before reporting.
- Give us lots of url links and screenshots
Planning new features
- Include developers early and often when you plan new website features or sections.
- Don’t make assumptions that something in the design will be ‘super easy’ or ‘too difficult’. We won’t let you go down the wrong path if we’re part of the conversation.
- It’s much easier to ask us for advice before you spend weeks creating and approving an approach that might be flawed to begin with.
Any parting thoughts for aspiring WebOps folks?
Don’t get too used to any one tool or set of tools.
A better tool for the job may be out there, or may come out tomorrow. It does the same job, but in a better, more efficient, or simpler way.
Don’t be afraid to experiment with new processes or tools.