Apache DevLake is an open source Big Data development platform for analyzing and visualizing fragmented data from DevOps tools in order to derive specific insights for engineering productivity. It compiles all DevOps data into a single dashboard that is customizable, versatile, and extendable. I've already gone over Apache DevLake in-depth, so check out the prior article for additional information on this particular topic.

In this article, we will integrate Apache Devlake with Grafana to visualize and display all of our repository data. To begin, we will deploy Apache Devlake locally using Docker, then connect it to our GitLab account and fetch and collect all of the data from a single repository, and finally, create a pipeline to trigger data collection. So, without further ado, let’s get started.

Setting up Devlake Server using Docker

Before we can begin integrating Grafana into our Devlake server, we must first run that server. For this article, we will use Docker to run our server locally, so make sure your Docker desktop is up and running.

After you have configured and set up your Docker desktop, we can start configuring our Docker compose and environment variable files. To do so, navigate to the empty directory and create a file called docker-compose.yml, into which you copy the following code from the below-mentioned snippets.

https://gitlab.com/-/snippets/2351019

https://gitlab.com/-/snippets/2351019

Your file and folder structure should resemble something like this.

Following that, you must configure your environment variables. Make a file called .env and paste the code from the below-mentioned snippets into it.

https://gitlab.com/-/snippets/2351020

https://gitlab.com/-/snippets/2351020

This is how your folder structure should look after you've added the environment variables.

Let's get Apache Devlake up and running. To begin, open the folder in VScode, then open the integrated terminal within it, and then proceed to type the following command to start and launch the Devlake server.

docker-compose up -d

It is important to note that pulling the images for the first time will take some time, but if you have done it before, it will not take as long.

Head back to your Docker desktop, and your Apache Devlake server should be up and running, as shown in the screenshot below. You should be able to see four servers running concurrently: one for MySQL, one for Grafana, one for devlake, and one for the config-ui.

Configure Data connection

We must first create our data connection before proceeding any further. To access your config-ui, open your browser and head over to the following URL.

http://localhost:4000/

For the sake of this tutorial, we will be using and fetching all data from our GitLab account.

Once you've opened the above-mentioned URL in your browser, After that, you should see something similar to this.

Configuring GitLab as a Data connection

You must first link the data connection, which you may accomplish by clicking on the GitLab icon, as we will be fetching data from GitLab.

When you click on that GitLab icon, you should be redirected to another page where you can add and update the connections.

You should be able to adjust the GitLab settings after clicking on the connection. To connect to the GitLab API, the connection section of the settings screen requires the following essential fields.

Connection Name: Connection name is set to GitLab by default and cannot be modified. Because GitLab is now a single-source data provider, the connection name is read-only because there is only one instance to maintain.
Endpoint URL (REST URL, starts with https:// or http:// The endpoint must be a legitimate REST API Endpoint, such as “https://gitlab.example.xn--com/api/v4/-ck3f and the URL must end with /.
Personal Access Token: To obtain a Personal Access Token, log in to your GitLab account and generate one in order to log in with the API using HTTP Basic Authentication.

Setting up a Personal Access Token in GitLab

Before we go any further, we'll need to create a personal access token that we'll use in Devlake's Auth Section to log in with the API using HTTP Basic Authentication. To do so, go to your GitLab account and click on the avatar in the upper right corner.

Go to the Access Tokens section on the left sidebar of your profile page, as shown in the screenshot below.

Finally, you've arrived at the Token settings page, so go ahead and give the token a name, don't forget to include the expiry date, and then select the scope for that token before clicking Create a personal access token.

After you've created your own access token, it should look something like this, so copy it and keep it somewhere secure because you won't be able to access it once you leave the page.

Finally, return to your DevLake GitLab settings page and input that copied Access Token obtained from GitLab into the Auth Token choices, remember that the rest of it can be kept as default for now.

We need to test the connection first to make sure it's working before we go any further. Select the Test connection option to do so, and a toast should appear at the bottom of your screen. \n

It's time to save our connection now that it's been established. Simply press the Save Connection button and proceed to the next step where we will start creating a pipeline.

It's time to set up a pipeline after successfully configuring and setting up our GitLab connection. To do so, select create a pipeline from the pipeline option located on the left sidebar of the screen.

To get started, we'll need to create a repository on GitLab. To do so, log into your account and create a repository; however, I have already done so and will choose that one. After you've completed that, copy the project id as shown in the screenshot below.

After you've successfully copied that repo project id, you'll need to enter it into that GitLab project id field, but before you do, double-check that you have the GitLab Pipeline provider toggled on. \n
To begin gathering data and information from that specific repository, click Run Pipeline located at the bottom of the page. This operation may take some time depending on the number of issues, commits, and overall traffic of that repo.

When it's finished fetching the data and information, you should be able to see something similar to this.

Finally, Grafana may be used to monitor data and information from your GitLab repository, as well as to create visualizations of it. To access the Grafana dashboard, simply click on the grafana button next to the pipeline you just created, or go directly to View dashboard button located at the top of the sidebar.

You can now access the Grafana dashboard, but keep in mind that the default login credentials are admin for the username and admin for the password.

Let's define Grafana before we go any further. Grafana is a web-based analytics and interactive visualization program that runs on a variety of platforms. It generates charts, graphs, and alerts for the web when connected to compatible data sources. Grafana is a common monitoring stack component that is frequently used with time series databases such as InfluxDB, Prometheus, and Graphite; monitoring platforms such as Sensu, Icinga, Checkmk, Zabbix, Netdata, and PRTG; SIEMs such as Elasticsearch and Splunk; and other data sources.

Grafana was first launched in 2014 as an extension of an Orbitz project by Torkel Ödegaard. It was originally designed to support time-series databases like InfluxDB, OpenTSDB, and Prometheus, but it has now expanded to include relational databases like MySQL, PostgreSQL, and Microsoft SQL Server.

You'll be able to view the Grafana dashboard in no time. Simply follow the procedures below to see your repo's beautiful visualization and chart.

Go to your home page and click on the square dashboard menu on the left side of the screen, then click on the manage option to see your repo's detailed visualization and chart.

You'll see a plethora of options once you reach the manage page. For the time being, skip them all and click Demo commit count by author.

As you can see, only one commit history appears to be appearing at the moment, and everything is taking place in real-time.

You may now personalize it any way you want; Grafana has a variety of visualizations to select from; but, before you go any further, make sure to apply the modifications and save it.

And with that, you've converted the graph from a graph chart to a gauge chart, and you can also experiment with your own query by adding your own custom query and displaying your unique visualization of choice.

Conclusion

Congratulations! You've effectively combined and leveraged Grafana's unrivalled capability with Apache Devlake. To summarize what we learned, we learned how to install Apache Devlake locally using Docker, then we connected it to GitLab and fetched and collected all of the commit history data from a single repository, and finally, we built a pipeline to trigger data collection and used Grafana to visualize and display all of our commit history data from that particular repository, and then we experimented with other visualization charts that you can utilize it in Grafana.

How to configure and integrate Grafana in DevLake?