Upgrade Elasticsearch from 2.4.6 to 5.6.8 for Graylog 2.4.3

Nicholas Siakotos
graylog

I know this is a bit messy but I wanted to get this onto somewhere before I forget.

So there are some new features in Graylog 2.4 that I wanted to use so I finally broke down and decided to upgrade my elasticsearch cluster to 5.6 the latest support version for graylog.

  1. Make sure you are up to date with all other updates before you start OS and Graylog
  2. I prefer to do everything through repos so on centos:
    1. nano /etc/yum.repos.d/elasticsearch.repo
    2. Past the following
    3. [elasticsearch-5.x] name=Elasticsearch repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
    4. Save and exit
  3. yum clean all and reboot
  4. Do this to all of your elasticsearch nodes
  5. Shutdown your elasticsearch cluster
  6. Timing is key with this if you do not want to lose logs.  This is why I always give myself a good size on the graylog journal so that if I have to do something to the cluster it will write everything to the journal space if it can't contact the elasticsearch cluster
  7. So when you are ready shutdown your elasticsearch cluster.
    1. You can do this through curl commands or just simply stop the service on all nodes
    2. systemctl stop elasticsearch
  8. Now run updates
    1. yum update
    2. Hit yes when you see your shiny new elasticsearch version
    3. wait for the update to finish
    4. Since this is an update it will bring over your existing yml files.  The problem is some of your settings in 2 will prevent elasticsearch from starting and the logs will be useless
  9. Before continuing elasticsearch has some changes from 2 to 5 that will cause the cluster to fail to start if you set them
  10. In particular you need to comment out:
    1. #discovery.zen.ping.multicast.enabled: false
    2. #bootstrap.memory_lock: true
    3. #script.inline: false #script.indexed: false #script.file: false #indices.store.throttle.max_bytes_per_sec: 150mb
  11. Now set your jvm options
    1. nano /etc/elasticsearch/jvm.options
    2. You really only need to set whatever you had before for memory for -xmx and xmx, leave everything else alone
  12.  
  13. Now systemctl start elasticsearch
  14. It will take some time to start up
  15. To watch it happen
    1. tail -50 /mnt/elasticsearch/logs/graylog.log
    2. /mnt/elasticsearch is a seperate disk where I have elasticsearch data and logs stored.  If you went with a default install it should be in /var/log/elasticsearch
  16. You can also curl "bindaddress":9200/_cat/health once you are green you are good to go.
    1. graylog green 3 3 760 760 0 0 0 0 - 100.0%