Real World Benchmarking with dotCMSJun 10, 2015
Quick Summary (TL;DR)
As you know, benchmarking performance is dependent on the page that is being tested. To jump straight to the bottom line in our testing: Using commodity hardware and testing a real world page (29k) a single dotCMS server can dynamically build and serve more than 1,500 page requests per second. And if you enable “Page Caching” and turn off on the same page, dotCMS can serve more than 15,000 page requests per second (turn off clickstreams and dotCMS can do 19,000+ requests), which is more than 3x the amount of data a Gigabit Ethernet connection can handle. dotCMS RESTful apis are just as performant - serving content at more than 11,000 requests per second.
What You Test Matters
At dotCMS we are often asked for performance numbers and sizing information regarding dotCMS installations. The inevitable answer to questions about performance is well, "it depends on your implementation." Though it is the correct answer, this never seems to satisfy anyone. This blog is intended to show 1) reasons why you should be skeptical of benchmarks not performed against your implementation and 2) how well an out-of-the-box dotCMS installation can perform on commodity hardware.
There are a lot of benchmarks for different CMSs that purport to test performance, all with various degrees of reproducibility and applicability, What will separate this exercise from some other performance tests is the real world nature of what is being tested. For example, below are two screenshots: The first is the dotCMS starter homepage that we will use for our performance numbers, the second image is a screenshot of the types of pages that are commonly used to demonstrate performance numbers. Both are legitimate pages to test, but the difference is striking. The dotCMS page that is being tested has:
Responsive / Bootstrap 3 based Design
Dynamic, multi-level menus
11 different editable content areas
16 different content objects being formatted/displayed
A video widget that plays in an overlay
A banner carousel
A dynamic New widget that it querying the “News” content type for the latest News items that have been tagged as “investment” News.
A twitter feed widget
The dotCMS test page is a “Real World” example of a CMS driven page that could be part of any actual implementation. It is dynamic, built on the fly, AND EVERY content object on it is being checked and displayed based on visitor permissions. This means, for example, that if the second banner in the carousel does not allow for anonymous access, it will not display on the page for an anonymous visitor (but it will show for a logged in visitor). The HTML for the whole page weighs in at a real and acceptable 29KB.
The HTML for this page is 29 KB
Compare this to pages that others use for benchmarking. Now, the page below can be a fair and accurate representation of performance, but it is easy to understand that testing performance against such a page is a very different test than testing against the page above. The test page below literally has a single menu and a single content object on it. It weighs in at a wee 2 KB.
The HTML for this page is 2 KB
And so here we come to the dirty secret about benchmarking a CMS: the magic “requests per second” number is directly correlated to the complexity of the page tested, the size of the source of the page (in Kb) and the amount of dynamic content that is on the page. And this is exactly why we, as vendors, have to answer performance questions with a “Well, it depends on your implementation”. But enough meta info about CMS benchmarks, let’s get to the testing.
So my son needed a home computer for high school. In my house, when a 15 year old says he needs a “computer” for school what he is really saying is that he does not want a Chromebook, he wants a computer so he can play video games. I get it, I was young once too. Instead of buying him a disposable laptop, I thought it would be a good father-son project to build a family computer from parts. We ended up putting together a nice, though normal, commodity box whose parts came in at ~ $1,000. Here are some of what we put together:
Intel i7 4770k (Haswell, 4-core 3.5ghz)
Samsung 256GB SSD
Other parts that matter less for our discussion
The good news is that Steam is available for Linux so my son happily installed a default Ubuntu 14 installation-
cat /etc/issue Ubuntu 14.04.2 LTS \n \l
First we need to have java installed. In this case, we will be using the default OpenJDK 7 version available in Ubuntu’s package manager.
$ sudo apt-get install openjdk-7-jdk $ java -version java version "1.7.0_75" OpenJDK Runtime Environment (IcedTea 2.5.4) (7u75-2.5.4-1~trusty1) OpenJDK 64-Bit Server VM (build 24.75-b04, mixed mode)
Second, we will be using Apache Benchmark. Apache Benchmark, or ab for short, is a great, simple benchmarking tool that is very useful to spot check a system's performance. I’ve relied on it for more than 15 years to provide raw performance numbers on dotCMS sites and other web based applications. It is part of the Apache Util package and can be installed like this:
$ sudo apt-get install apache2-utils
Download the 3.2.1 dotCMS Release
Finally, we download the latest standard distribution of dotCMS, untar it and start it up. On its initial startup, dotCMS will automatically install what we call the “Starter site”, which loads a site for a fictitious financial company called “Quest Financial”. This site, while fully featured, is just intended as a demonstration for dotCMS and its capabilities.
$ mkdir dotcms $ cd dotcms $ wget http://dotcms.com/physical_downloads/release_builds/dotcms_3.2.1.tar.gz $ tar -zxvf dotcms_3.2.1.tar.gz $ ./bin/startup.sh $ tail -f dotserver/tomcat-8.0.18/logs/catalina.out …… …… (dotCMS initial setup, importing database, content) …… …… 30-Apr-2015 16:02:45.890 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /home/will/dotcms/dotserver/tomcat-8.0.18/webapps/ROOT has finished in 122,232 ms 30-Apr-2015 16:02:45.892 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"] 30-Apr-2015 16:02:45.896 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 122257 ms
When you see the message, INFO: server startup it means your dotCMS is ready to go. By default, dotCMS will use an embedded local database, H2, to store content and metadata, though in production, we always recommend using a standalone database. Because dotCMS caches pages and objects in memory, the database we use for these tests does not make a difference.
For this exercise we will benchmark our starter site’s home page. You can see this page on our demo site: http://demo.dotcms.com. If you login to our demo site and browse to the home page and switch to “Edit Mode”, you will see how dynamic it is. Everything is editable (with the right permissions), navigation is dynamically generated, content is included dynamically, etc. The dotCMS starter site home page is is very much a real world example page. Really, when we were building the demo home page, the last thing we were concerned about was how the page would perform during benchmarking. And this is exactly what makes is a great benchmark testing target.
Out of the box, dotCMS caches all objects in memory. dotCMS initially loads these objects from the database and into memory lazily, when a page is first requested. After that, when the page is requested again, dotCMS assembles these in-memory objects, queries elasticsearch (not the db) index for any dynamic content listings and checks permissions (again in memory) on everything at the Object level. The fact that dotCMS uses the in-memory object cache and the elasticsearch content index rather than the database gives dotCMS tremendous scalability.
Page Perfomance: Without Page Cache
Once dotCMS is up and running, you should visit the home page to make sure everything loads properly. We can then do a few ab “warm up” runs, which insure the cache is warmed up and allows the jvm to JIT compile and optimize methods that are often used. The command we use to benchmark looks like this:
$ ab -n 10000 -c 50 http://localhost:8080/
This tells Apache Benchmark to request the homepage 10,000 times with a concurrency of 50. Once dotCMS is warmed up, we are ready to benchmark. Here are some typical results I got after a few runs:
$ ab -n 10000 -c 50 http://127.0.0.1:8080/ This is ApacheBench, Version 2.3 <$Revision: 1528965 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient) Server Software: Apache-Coyote/1.1 Server Hostname: 127.0.0.1 Server Port: 8080 Document Path: / Document Length: 29974 bytes Concurrency Level: 50 Time taken for tests: 6.466 seconds Complete requests: 10000 Failed requests: 0 Total transferred: 302920075 bytes HTML transferred: 299740000 bytes Requests per second: 1546.59 [#/sec] (mean) Time per request: 32.329 [ms] (mean) Time per request: 0.647 [ms] (mean, across all concurrent requests) Transfer rate: 45751.37 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.4 0 10 Processing: 3 32 23.2 26 217 Waiting: 2 31 22.9 25 217 Total: 3 32 23.2 26 217
There are a few things to note about the above numbers. First, you can see that dotCMS is serving more than 1,500 requests per second, where every piece of content is dynamically generated, permissioned and rendered. Second, look at that transfer rate/second - dotCMS is pumping some serious content. A transfer rate of 45,751 KBps translates to > 366 Mbps, which is a lot of data. And again, keep in mind that this page and content rendering is completely dynamic.
Page Performance: Now with more Page Cache
A dotCMS Enterprise license unlocks the ability to use what we call the “Page Cache”. Once a page is loaded into the Page Cache, the cache serves subsequent requests for that page statically. The result of using the Page Cache is about 10x greater page performance. And the dotCMS Page Cache is more intelligent than external caching mechanisms such as CDNs or a reverse proxy. For example, a page being served from cache still respects page permissions- if a user does not have permission to a page, they will be redirected to login or sent a 403 error. Additionally, the dotCMS Page Cache allows a user to specify different TTLs for different pages and will automatically invalidate a page when a change is made to the page or its template. It will also cache the page based on users, or different url parameters or other variables that might affect a page’s output.
Even though the Page Cache is doing all of the permissions and invalidation work, we still see significantly faster performance. Here are the results of running Apache Benchmark against a dotCMS server with the page cache enabled:
$ ab -n 10000 -c 50 http://127.0.0.1:8080/ This is ApacheBench, Version 2.3 <$Revision: 1528965 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient) Server Software: Apache-Coyote/1.1 Server Hostname: 127.0.0.1 Server Port: 8080 Document Path: / Document Length: 29974 bytes Concurrency Level: 50 Time taken for tests: 0.662 seconds Complete requests: 10000 Failed requests: 0 Total transferred: 302920000 bytes HTML transferred: 299740000 bytes Requests per second: 15115.44 [#/sec] (mean) Time per request: 3.308 [ms] (mean) Time per request: 0.066 [ms] (mean, across all concurrent requests) Transfer rate: 447145.54 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.2 0 2 Processing: 0 3 3.3 2 36 Waiting: 0 3 3.2 2 36 Total: 0 3 3.3 2 36
15,000+ requests/sec for a fully realized and permissioned page, right out of the box, is pretty amazing performance. It translates to 3,577+ Mbps which is more than enough to saturate Gigabit Ethernet networks (which is why I tested against the localhost vs. a remote client).
Wait, there is more: Turning off Sessions
By default, dotCMS ships with clickstream tracking on. This forces a new session to be created for each visitor. You can turn off this behavior by setting the configuration property ENABLE_CLICKSTREAM_TRACKING=false. If you do this on the latest release, the numbers get even more impressive:
$ ab -n 100000 -c 50 http://localhost:8080/ This is ApacheBench, Version 2.3 <$Revision: 1528965 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Server Software: Apache-Coyote/1.1 Server Hostname: localhost Server Port: 8080 Document Path: / Document Length: 29961 bytes Concurrency Level: 50 Time taken for tests: 5.125 seconds Complete requests: 100000 Failed requests: 0 Total transferred: 3020400000 bytes HTML transferred: 2996100000 bytes Requests per second: 19512.50 [#/sec] (mean) Time per request: 2.562 [ms] (mean) Time per request: 0.051 [ms] (mean, across all concurrent requests) Transfer rate: 575542.63 [Kbytes/sec] received
19,000 Requests per second. That is some serious performance from a dynamic system.
API Performance: Content as JSON
dotCMS offers a robust Content API that allows dotCMS to act as a content store. This API has a java interface and a RESTful interface that can be accessed via normal url endpoints. Like all our content access, the Content API respects permissions for all requests. For more information about using this Content API, see: http://dotcms.com/docs/latest/content-api.
You can see an example of the URL I used to test the content API (this points to the current demo site)
What this url is doing is calling the content with id 767509b1…. which is the id of the content found on the “About Us” page of the demo site, and returning that in a JSON representation as seen below:
Here is the command and output from my API test:
$ ab -n 10000 -c 50 http://127.0.0.1:8080/api/content/id/767509b1-2392-4661-a16b-e0e31ce27719 This is ApacheBench, Version 2.3 <$Revision: 1528965 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient) Server Software: Apache-Coyote/1.1 Server Hostname: 127.0.0.1 Server Port: 8080 Document Path: /api/content/id/767509b1-2392-4661-a16b-e0e31ce27719 Document Length: 2473 bytes Concurrency Level: 50 Time taken for tests: 0.887 seconds Complete requests: 10000 Failed requests: 0 Total transferred: 26010000 bytes HTML transferred: 24730000 bytes Requests per second: 11268.13 [#/sec] (mean) Time per request: 4.437 [ms] (mean) Time per request: 0.089 [ms] (mean, across all concurrent requests) Transfer rate: 28621.48 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.2 0 4 Processing: 0 4 2.7 4 48 Waiting: 0 4 2.2 4 37 Total: 0 4 2.6 4 48
Again, these are great numbers. The test represents more than 11,000 content API requests per second, all dynamic.
As an Enterprise Content Management System, performance is very important to dotCMS. As these performance numbers indicate, dotCMS has been architected to scale, even when testing against real world pages and use cases. In a clustered configuration, dotCMS’ performance numbers become even more impressive as each node of dotCMS can render pages without any resource contention. We look forward to hearing any questions or comments you might have regarding these tests.