Virtual Graffiti, Inc - Your Source for Technology Solutions

VG Tuesday Tips: How to Deploy a Graylog SIEM Server in AWS and Integrate with Imperva Cloud WAF

  Virtual Graffiti

Follow us for your more blog posts today!


Security Information and Event Management (SIEM) products provide real-time analysis of security alerts generated by security solutions such as Imperva Cloud Web Application Firewall (WAF). Many organizations implement a SIEM solution to bring visibility of all security events from various solutions and to have the ability to search them or create their own dashboard.

Note that a simpler alternative to SIEM is Imperva Attack Analytics, which reduces the burden of integrating a SIEM logs solution and provides a condensed view of all security events into comprehensive narratives of events rated by severity. A demo of Imperva Attack Analytics is available here.

This article will take you step-by-step through the process of deploying a Graylog server that can ingest Imperva SIEM logs and let you review your data. They are:

  • Step 1: Deploy a new Ubuntu server on AWS
  • Step 2: Install java, Mongodb, elasticsearch
  • Step 3: Install Graylog
  • Step 4: Configure the SFTP server on the AWS server
  • Step 5: Start pushing SIEM logs from Imperva Incapsula

The steps apply to the following scenario:

  • Deployment as a stand-alone EC2 on AWS
  • Installation from scratch, from a clean Ubuntu machine (not a graylog AMI in AWS)
  • Single server setup, where the logs are located in the same server as Graylog
  • Push of the logs from Imperva using SFTP

Most of the steps below also apply to any setup or cloud platforms besides AWS. Note that in AWS, a Graylog AMI image does exist, but only with Ubuntu 14 at the time of writing. Also, I will publish future blogs on how to parse your Imperva SIEM logs and how to create a dashboard to read the logs.

Step 1: Deploy an Ubuntu Server on AWS

As a first step, let’s deploy an Ubuntu machine in AWS with the 4GB RAM required to deploy Graylog.

  1. Sign in to the AWS console and click on EC2
  2. Launch an instance and select.
  3. Select Ubuntu server 16.04, with no other software pre-installed.
  4. It is recommended to use Ubuntu 16.04 and above, as some repo are already pre-included such as MongoDB and Java openjdk-8-jre, which simplifies the installation process. The command lines below apply for Ubuntu 16.04 (systemctl command, for instance, is not applicable for Ubuntu 14).

  5. Select the Ubuntu server with 4GB RAM.
  6. 4GB is the minimum for Graylog, but you might consider more RAM depending on the volume of the data that you plan to gather.

  7. Optional: increase the disk storage.
  8. Since we will be collecting logs, we will need more storage than the default space. The storage volume will depend a lot on the site traffic and the type of logs you will retrieve (all traffic logs or only security events logs).

    Note that you will likely require much more than 40GB. If you are deploying on AWS, you can easily increase the capacity of your EC2 server anytime.

  9. Select an existing key pair so you can connect to your AWS server via SSH later.
  10. If you do not have an existing SSH key pair in your AWS account, you can create it using the ssh-keygen tool, which is part of the standard openSSH installation or using puttygen on Windows. Here’s a guide to creating and uploading your SSH key pairs.

  11. Give your EC2 server a clear name and identify its public DNS and IPv4 addresses.
  12. Configure the server security group in AWS.
  13. Let’s make sure that port 9000 in particular is open. You might need to open other ports if logs are forwarded from another log collector, such as port 514 or 5044.

    It is best practice that you open port 22 only from Cloud WAF IP (this link) or from your IP. Prevent from opening port 22 to the world.

    You can also consider locking the UI access to your public IP only.

  14. SSH to your AWS server with the Ubuntu user, after uploading your key in Putty and putting the AWS public DNS entry.
  15. Update your Ubuntu system to the latest versions and updates.
  16. sudo apt-get update

    sudo apt-get upgrade

    Select “y” when prompted or the default options offered.

    Step 2: Install Java, MongoDB and Elasticsearch

  17. Install additional packages including Java JDK.
  18. sudo apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen

    Check that Java is properly installed by running:

    java -version

    And check the version installed. If all is working properly, you should see a response like:

  19. Install MongoDB.
  20. Graylog uses MongoDB to store the Graylog configuration data

    MongoDB is included in the repos of Ubuntu 16.04 and works with Graylog 2.3 and above.

    sudo apt-get install mongodb-server

    Start mongoDB and make sure it starts with the server:

    sudo systemctl start mongod

    sudo systemctl enable mongod

    And we can check that it is properly running by:

    sudo systemctl status mongod

  21. Install and configure Elasticsearch
  22. Graylog 2.5.x can be used with Elasticsearch 5.x. You can find more instructions in the Elasticsearch installation guide:

    wget -qO – https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –

    echo “deb https://artifacts.elastic.co/packages/5.x/apt stable main” | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

    sudo apt-get update && sudo apt-get install elasticsearch

    Now modify the Elasticsearch configuration file located at /etc/elasticsearch/elasticsearch.yml and set the cluster name to graylog.

    sudo nano /etc/elasticsearch/elasticsearch.yml

    Additionally you need to uncomment (remove the # as first character) the line:

    cluster.name: graylog

    Now, you can start Elasticsearch with the following commands:

    sudo systemctl daemon-reload

    sudo systemctl enable elasticsearch.service

    sudo systemctl restart elasticsearch.service

    By running sudo systemctl status elasticsearch.service you should see Elasticsearch up and running as below:

    Step 3: Install Graylog

  23. We can now install Graylog repository and Graylog itself with the following commands:
  24. wget https://packages.graylog2.org/repo/packages/graylog-2.5-repository_latest.deb sudo dpkg -i graylog-2.5-repository_latest.deb

    sudo apt-get update && sudo apt-get install graylog-server

  25. Configure Graylog
  26. First, create a password of at least 64 characters by running the following command:

    pwgen -N 1 -s 96

    And copy the result referenced below as password

    Let’s create its sha256 checksum as required in the Graylog configuration file:

    echo -n password | sha256sum

    Now you can open the Graylog configuration file:

    sudo nano /etc/graylog/server/server.conf

    And replace password_secret and root_password_sha2 with the values you created above.

    The configuration file should look as below (replace with your own generated password):

    Now replace the following entries with your AWS CNAME that was given when creating your EC2 instance. Note that also, depending on your setup, you can replace the alias below with your internal IP.

    REST API:

    web:

  27. Optional: Configure HTTPS for the Graylog web interface
  28. Although not mandatory, it is recommended that you configure https to your Graylog server.

    Please find the steps to setup https in the following link: http://docs.graylog.org/en/2.3/pages/configuration/web_interface.html#configuring-webif-nginx

  29. Start the Graylog service and enable it on system startup
  30. Run the following commands to restart Graylog and enforce it on the server startup:

    sudo systemctl daemon-reload

    sudo systemctl enable graylog-server.service

    sudo systemctl start graylog-server.service

    Now we can check that Graylog has properly started:

    sudo systemctl status graylog-server.service

  31. Login to the Graylog console
  32. You should now be able to login to the console.

    If the page is not loading at all, check if you have properly configured the security group of your instance and that port 9000 is open.

    You can login with username ‘admin’ and the password you set as your secret password.

    Step 4: Configure SFTP on your server and Imperva Cloud WAF SFTP push

  33. Create a new user and its group
  34. Let’s create a directory where the logs will be sent to Incapsula to send logs.

    sudo.adduser incapsula

    incapsula is the user name created in this example. You can replace it to the name of your choice. You will be prompted to choose a password.

    Let’s create a new group:

    sudo groupadd incapsulagroup

    And associate the incapsula user to this group

    sudo usermod -a -G incapsulagroup incapsula

  35. Let’s create a directory where the logs will be sent to
  36. In this example, we will send all log to /home/incapsula/logs

    cd /home

    sudo mkdir incapsula

    cd incapsula

    sudo mkdir logs

  37. Now let’s set strict permissions restrictions to that folder
  38. For security purposes, we want to restrict access of this user strictly to the folder where the logs will be sent. The home and incapsula folders can be owned by root while logs will be owned by our newly created user.

    sudo chmod 755 /home/incapsula

    sudo chown root:root /home/incapsula

    Now let’s assign our new user (incapsula in our example) as the owner of the logs directory:

    sudo chown -R incapsula:incapsulagroup /home/incapsula/logs

    The folder is now owned by incapsula and belongs to incapsulagroup.

    And you can see that the incapsula folder is restricted to root, so the newly created incapsula user can only access the /home/incapsula/logs folder, to send its logs.

  39. Now let’s configure open-ssh SFTP server and set the appropriate security restrictions.
  40. sudo nano /etc/ssh/sshd_config

    Comment out this section:

    #Subsystem sftp /usr/lib/openssh/sftp-server

    And add this line right below:

    subsystem sftp internal-sftp

    Change the authentication to allow password authentication so Incapsula can send logs using username / password authentication:

    PasswordAuthentication yes

    And add the following lines at the bottom of the document:

    match group incapsulagroup

    chrootDirectory /home/incapsula

    X11Forwarding no

    AllowTcpForwarding no

    ForceCommand internal-sftp

    PasswordAuthentication yes

    Save the file and exit.

    Let’s now restart the SSH server:

    sudo service sshd restart

  41. Now let’s check that we can send files using SFTP
  42. For that, let’s open use Filezilla and try to upload a file. If everything worked properly, you should be able to:

    • Connect successfully
    • See the logs folder, and be unable to navigate
    • Copy a file to the remote server

    Step 5: Push the logs from Imperva Incapsula to the Graylog SFTP folder

  43. Configure the Logs in Imperva Cloud WAF SIEM logs tab
    • Log into your my.incapsula.com account.
    • On the sidebar, click Logs > Log Setup
      • Make sure you have SIEM logs license enabled.
    • Select the SFTP option
    • In the host section enter your public facing AWS hostname. Note that your security group should be open to Incapsula IPs as described in the Security Group section earlier.
    • Update the path to log
    • For the first testing, let’s disable encryption and compression.
    • Select CEF as log format
    • Click Save

    See below an example of the settings. Click Test Connection and ensure it is successful. Click Save.

  44. Make sure the logs are enabled for the relevant sites as below
  45. You can select either security logs or all access logs on a site-per-site basis.

    Selecting All Logs will retrieve all access logs, while Security Logs will push only logs where security events were raised.

    • Note that selecting All Logs will have a significant impact on the volume of logs.

    You can find more details on the various settings of the SIEM logs integration in Imperva documentation in this link.

  46. Verify that logs are getting pushed from Incapsula servers to your FTP folder
  47. The first logs might take some time to reach your server, depending on the volume of traffic on the site, in particular for a site with little traffic. Generate some traffic and events.

  48. Enhance performance and security
  49. To improve the security and performance of your SIEM integration project, you can consider enforcing https in Graylog. You can find a guide to configure https on Graylog here.

    That’s it!