Friday, November 8, 2019

Streaming Spring boot logs to ELK stack

In my previous blog, we have done ELK installation on windows 10 and we have even tried to push messages from input console to Elastic Search and finally viewed on Kibana Server.

I will write a separate blog on why do we need ELK?

In this blog, I’ll show you how can we push spring boot application log directly to Elastic search using Logstash which we can analyze on Kibana and If you don’t know how to install ELK on windows 10 then you can refer my previous blog and start Elastic Search and Kibana server.

Prerequisite


  • Elastic Search and Kibana running on your machine
  • Basic knowledge of Spring boot application


If you don’t want to start your application from scratch then you can download one spring boot application from my GitHub repository as well.

I am assuming that the Elastic Search and Kibana server are running on your machine and you have a fair idea of how to start the Logstash server and what is Logstash conf file.

So, to push spring boot logs continuously to Elastic Server, We have to open one TCP port in Logstash server and for that we have to create one Logstash config file (say elklogstash.conf) under ${LOGSTASH_HOME}/conf directory mentioning on which port TCP port should be listening under input filter and where to push the data once we received under Output filter.

For simplicity, I am skipping the filter tag as it is optional.

elklogstash.conf




Now start the Logstash server bypassing newly created conf file.
   bin\logstash -f .\config\elklogstash.conf



Cool! Now Logstash server is also up and running and if you observe the log, you will realize that it is also listening on port 4560 as mentioned in the conf file. Configure the newly created index (elkbootlogs) on Kibana as we have done during the ELK setup.

Now let's do some changes to spring boot application so that it can push all the logs to 4056 TCP port.

For this tutorial, I am using spring-logger project from my Github repository.

Add below dependency to the pom.xml file. We need Logstash encoder to encode messages.

<!-- Added for logstash Encoder-->
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>6.2</version>

</dependency>

Open logback-spring.xml file which is under the resource folder and create new appender (say elk). The task of this appender is to push logs to the destination TCP socket and under this appender, compulsory use LogstashEncoder.

<appender name="elk" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
    <destination>localhost:4560</destination>
    <!-- encoder is required -->
    <encoder class="net.logstash.logback.encoder.LogstashEncoder" />

</appender>

Add new appender to root level

<!-- LOGGING everything at INFO level -->
<root level="info">
<appender-ref ref="RollingFile" />
<appender-ref ref="Console" />
<appender-ref ref="elk" />
</root>

Save all files and start your application. So, we are done with all the setup. Its time to check whether all the changes are done properly or not.

Open Kibana on your browser (http://localhost:5601) and select your index under the Discover tab. You will see all logs are populating on Kibana as well.



Congratulations! Our configuration is working absolutely fine and it is pushing logs to Elastic Search. 

You can download the source code from here, ELK code chnages are under elkstack branch.






12 comments:

  1. Thank you for such useful insights. Visit here for more on advanced tech courseselk stack training

    ReplyDelete
  2. You completed certain reliable points there. I did a search on the subject and found nearly all persons will agree with your blog. Voir film

    ReplyDelete
  3. Wow, super make regarded. i'd once to draft in the back of this too - taking period and valid difficult feat to make a large article. This declaration has encouraged me to write some posts that i'm going to jot down quickly. totalsportek.news/f1-streaming

    ReplyDelete
  4. All guides online are "how to hardcode Logstash URL in logback-spring.xml", but in production use logstash host would be coming from Spring configuration (via configserver) in runtime, how to configure this correctly?

    ReplyDelete
  5. Live stream video and broadcast your event live to audiences on the web and mobile devices using our live streaming platform and services and share your event live corporate live streaming services

    ReplyDelete
  6. Nice to be visiting your blog again, it has been months for me. Well this article that i’ve been waited for so long. I need this article to complete my assignment in the college, and it has same topic with your article. Thanks, great share. Video Streaming Server

    ReplyDelete
  7. This shows that the Contivity VPN Switch never reacted to the association endeavor. best vpn service for streaming

    ReplyDelete
  8. On 24 April 2014, contender Netflix declared that it had contracted with three little link organizations to give endorsers admittance to its substance through TiVo DVRs, while on 28 April 2014 it reported an arrangement with Verizon to give Netflix supporters high velocity online admittance to streaming substance, the second such arrangement Netflix has made with an Internet specialist co-op ("ISP"). https://www.buyyoutubesubscribers.in/

    ReplyDelete
  9. Playcasino | 100% up to €1000 + 300 FS
    Join and 바카라 검증 get up to €1000+ in bonus 부들 이 벗방 codes for new 스핀토토 players at Playcasino! Discover amazing 바카라 사이트 주소 promotions, 업소 사이트 generous bonuses & fast payouts.

    ReplyDelete
  10.  I'm going to give you 3 very valuable methods to improve you Manga skills. kissanime

    ReplyDelete
  11. https://www.floraindia.com/flower-delivery/delhi.html Car games do occupy an important place in the world of video games. These games can be found in abundance in a lot of websites. There are various types of car racing games, like drag racing games, Chevy racing games, Formula 1 racing games or simply road racing games.

    ReplyDelete
  12. https://www.visualaidscentre.com/lasik-surgery-in-gurgaon/ Every LASIK patient needs some help finding a doctor to perform their LASIK surgery in Michigan. This article will provide fact that some surgeons don't share with their patients!

    ReplyDelete

How TOPT Works: Generating OTPs Without Internet Connection

Introduction Have you ever wondered how authentication apps like RSA Authenticator generate One-Time Passwords (OTPs) without requiring an i...