Replace Your Slow Prometheus Configuration With Watchdog’s Zero-Config, Automatic Alerts
–Dr. Frederick Ryckbosch, Department of Applied Mathematics, Computer Science and Statistics, University of Ghent, Belgium
Watchdog includes Essential Alerts™ out of the box and takes less than a minute to install, while Prometheus can’t even offer anomaly detection. See how Watchdog stacks up.
System admins don’t have time to investigate every little problem. But they need to be aware of issues that lead to downtime. Why spend precious time on something as common as server monitoring?
Let Watchdog guard your hardware — automatically — by installing our Smart Agent and get proactive alerts on:
VM steal time
–Andrew Hurd, Sr. Software Engineer, ZipBooks
Small and lightweight, Watchdog can be set up on all your servers in minutes thanks to its 23-second install
As a startup, you know you need to monitor your servers and apps, but you are also trying to bootstrap and trim costs where possible. Watchdog installs on all your systems for free, giving you zero-config Essential Alerts™ (CPU, memory, swap space, etc.) and Custom Alerts in minutes.
When your business is scaling to meet the demands of your customers, you need to ensure high uptime and availability for retention and loyalty. Watchdog takes only 23 seconds to install, and Custom Alerts are easy to configure on the fly.
Setting up Watchdog on existing large installations is easy because there is no config required to get Essential Alerts on even hundreds of servers.
Part 1: Installing Prometheus Daemons
STEP 1: Download the source archive
Go to the Prometheus download page and download the source archive. Once downloaded, extract the sources, by running:
$ tar -zxvf Prometheus-3.4.0.tar.gz
Enter the correct Prometheus version in the command. It must match the name of the downloaded archive.
STEP 2: Create user account
For all of the Prometheus daemon processes, an unprivileged user is required. If a Prometheus daemon is started from an unprivileged user account, it will run as that user.
However, if a daemon is started from a 'root' account, it will switch to a 'Prometheus' user account, which must be present. To create such a user account (in its own group, “Prometheus”) on Linux systems, run:
useradd -g Prometheus Prometheus
Running Prometheus as root, bin, or any other account with special rights is a security risk.
STEP 3: Create Prometheus database
For Prometheus server and proxy daemons, as well as Prometheus frontend, a database is required. It is not needed to run Prometheus agent.
SQL scripts are provided for creating database schema and inserting the dataset. Prometheus proxy database needs only the schema while Prometheus server database requires also the dataset on top of the schema.
Having created a Prometheus database, proceed to the following steps of compiling Prometheus.
STEP 4: Configure the Sources
When configuring the sources for a Prometheus server or proxy, you must specify the database type to be used. Only one database type can be compiled with a server or proxy process at a time.
To see all of the supported configuration options, inside the extracted Prometheus source directory run:
To configure the sources for a Prometheus server and agent, you may run something like:
./configure --enable-server --enable-agent --with-mysql --enable-ipv6 --with-net-snmp --with-libcurl --with-libxml2
--with-libcurl configuration option with cURL 7.20.0 or higher is required for SMTP authentication, supported since Prometheus 3.0.0.
--with-libcurl and --with-libxml2 configuration options are required for virtual machine monitoring, supported since Prometheus 2.2.0.
To configure the sources for a Prometheus server (with PostgreSQL etc.), you may run:
./configure --enable-server --with-postgresql --with-net-snmp
To configure the sources for a Prometheus proxy (with SQLite etc.), you may run:
./configure --prefix=/usr --enable-proxy --with-net-snmp --with-sqlite3 --with-ssh2
To configure the sources for a Prometheus agent, you may run:
You may use the --enable-static flag to statically link libraries. If you plan to distribute compiled binaries among different servers, you must use this flag to make these binaries work without required libraries. Note that --enable-static does not work under Solaris.
Using --enable-static option is not recommended when building server.
In order to build the server statically you must have a static version of every external library needed. There is no strict check for that in configure script.
Command-line utilities Prometheus_get and Prometheus_sender are compiled if --enable-agent option is used.
Add optional path to the MySQL configuration file --with-mysql=/<path_to_the_file>/mysql_config to select the desired MySQL client library when there is a need to use one that is not located in the default location.
It is useful when there are several versions of MySQL installed or MariaDB installed alongside MySQL on the same system.
Use --with-ibm-db2 flag to specify location of the CLI API.
Use --with-oracle flag to specify location of the OCI API.
STEP 5: Make and install everything
If installing from SVN, it is required to run first:
$ make dbschema
$ make install
This step should be run as a user with sufficient permissions (commonly 'root', or by using sudo).
Running make install will by default install the daemon binaries (Prometheus_server, Prometheus_agentd, Prometheus_proxy) in /usr/local/sbin and the client binaries (Prometheus_get, Prometheus_sender) in /usr/local/bin.
To specify a different location than /usr/local, use a --prefix key in the previous step of configuring sources, for example --prefix=/home/Prometheus. In this case daemon binaries will be installed under <prefix>/sbin, while utilities under <prefix>/bin. Man pages will be installed under <prefix>/share.
STEP 6: Review and edit configuration files
You need to configure this file for every host with Prometheus_agentd installed.
You must specify the Prometheus server IP address in the file. Connections from other hosts will be denied.
You must specify the database name, user and password (if using any).
The rest of the parameters will suit you with their defaults if you have a small installation (up to ten monitored hosts). You should change the default parameters if you want to maximize the performance of Prometheus server (or proxy) though. See the performance tuning section for more details.
You must specify the server IP address and proxy hostname (must be known to the server), as well as the database name, user and password (if using any).
With SQLite the full path to database file must be specified; DB user and password are not required.
STEP 7: Start up the daemons
Run Prometheus_server on the server side.
Make sure that your system allows allocation of 36MB (or a bit more) of shared memory, otherwise the server may not start and you will see “Cannot allocate shared memory for
See the "See also" section at the bottom of this page to find out how to configure shared memory.
Run Prometheus_agentd on all the monitored machines.
Make sure that your system allows allocation of 2MB of shared memory, otherwise the agent may not start and you will see “Cannot allocate shared memory for collector.” in the agent log file. This may happen on Solaris 8.
If you have installed Prometheus proxy, run Prometheus_proxy.
PART 2: Installing Prometheus web interface
Copying PHP files (4 commands)
Prometheus frontend is written in PHP, so to run it a PHP supported webserver is needed. Installation is done by simply copying the PHP files from frontends/php to the webserver HTML documents directory.
Common locations of HTML documents directories for Apache web servers include:
It is suggested to use a subdirectory instead of the HTML root. To create a subdirectory and copy Prometheus frontend files into it, execute the following commands, replacing the actual directory:
cp -a . <htdocs>/Prometheus
If installing from SVN and planning to use any other language than English, you must generate translation files. To do so, run:
msgfmt utility from gettext package is required.
Additionally, to use any other language than English, its locale should be installed on the web server. See the "See also" section in the “User profile” page to find out how to install it if required.
Installing the Frontend
In your browser, open Prometheus URL: http://<server_ip_or_name>/Prometheus
You should see the first screen of the frontend installation wizard.
Make sure that all software prerequisites are met.
Enter details for connecting to the database. Prometheus database must already be created.
Enter Prometheus server details.
Review a summary of settings.
Download the configuration file and place it under conf/ in the webserver HTML documents subdirectory where you copied Prometheus PHP files to.
Providing the webserver user has write access to conf/ directory the configuration file would be saved automatically and it would be possible to proceed to the next step right away.
Finish the installation.
The Prometheus frontend is finally ready after all of these steps. The default username is Admin, password Prometheus.
As open source software, Prometheus is offered for free once you make it through the installation. Hosted Prometheus is a third-party option. You can skip the painful installation process detailed above, but then you have to budget for your needs
Add on inevitable support contracts, a developer salary to maintain your configuration, and paid courses to learn it, and you can see why Prometheus isn’t really free.
Create a free Blue Matador account
You can get started by creating your account here
Install the Blue Matador Smart Agent, which is automatically configured with Watchdog alerts.
On Linux, just enter:
curl -L -s "https://app.bluematador.com/install?os=linux" | sudo ACCOUNT="a67acd6e68f872b807a1c9afe3a2453564e67877" API_KEY="f80d0ca101e967b50b730ddf8e8aca0de85e8df6" AGENT_TAGS="env:prod" bash
Receive Essential Alerts™ on your servers without any config on your part. Set up Custom Alerts for any arbitrary alert you need for your setup.
$0 - Free Forever
We aren’t kidding when we say it’s free.
We get the frustration of fighting fires all the time, always balancing the need to fix production, add more monitoring, and do more of the real DevOps work you were hired to do.
Watchdog is free forever because we want to help system admins get ahead of the firefighting curve. Our founders spent years doing on-call DevOps monitoring, day and night.
–Aaron Mefford, VP of Technology & Development, DMG
You can download and install Watchdog on your own just by creating a Blue Matador account instantly. There’s no waiting around for permission or approval.
Watchdog really is free. Not convinced? Use our pricing calculator below to doublecheck.
Watchdog is fully featured for as many servers as you want. There are no user limitations, feature paywalls, data restrictions, update restraints, or alert throttling.
Your monthly bill would be
Did you think we were kidding?
Watchdog really is free forever.GET FREE SERVER MONITORING
Watchdog runs entirely on your server. In fact, after installing the Blue Matador Smart Agent, your server detects hardware anomalies and generates the report metrics itself. And it does this with the amazingly low overhead that our Smart Agent provides (<2% CPU overhead, <2% memory usage, and 90% gzip compression).