Apache DevLake is an open source Big Data development platform for analyzing and visualizing fragmented data from DevOps tools so that relevant insights can be extracted from it. If you'd want to learn more about Apache devlake in depth, visit our previous article.

In this article, we'll establish a pipeline in Advanced mode, then use it to collect multiple GitLab and GitHub repositories simultaneously. To begin, we'll use Docker to install Apache Devlake locally, then connect it to our GitLab account to fetch and collect data from multiple repositories, and finally, create an advanced mode pipeline to trigger data collection. So, without further ado, let’s get started.

Launching Devlake in Docker

We must first launch our Devlake server before we can begin integrating the pipeline in advanced mode. For the purposes of this article, we will utilize Docker to host our server locally, so make sure your Docker desktop is running. If you're new to Docker, make sure to read the official documentation first.

However, because I've previously configured and installed Docker on my PC already, I'll simply launch my Docker desktop environment.

We may begin configuring our Docker compose and environment variable files once you've established and set up your Docker desktop. To do so, go to the empty directory and create a file named docker-compose.yml, into which you'll paste the code mentioned below.

version: "3"
services:
  mysql:
    image: mysql:8.0.26
    platform: linux/x86_64
    volumes:
      - mysql-storage:/var/lib/mysql
    restart: always
    ports:
      - 127.0.0.1:3306:3306
    environment:
      MYSQL_ROOT_PASSWORD: admin
      MYSQL_DATABASE: lake
      MYSQL_USER: merico
      MYSQL_PASSWORD: merico

  grafana:
    image: mericodev/grafana:v0.10.1
    ports:
      - 3002:3000
    volumes:
      - grafana-storage:/var/lib/grafana
    environment:
      GF_USERS_ALLOW_SIGN_UP: 'false'
      GF_DASHBOARDS_JSON_ENABLED: 'true'
      GF_INSTALL_PLUGINS: grafana-piechart-panel
      GF_LIVE_ALLOWED_ORIGINS: '*'
      MYSQL_URL: mysql:3306
      MYSQL_DATABASE: lake
      MYSQL_USER: merico
      MYSQL_PASSWORD: merico
    restart: always
    depends_on:
      - mysql

  devlake:
    image: mericodev/lake:v0.10.1
    ports:
      - 127.0.0.1:8080:8080
    restart: always
    volumes:
      - ./.env:/app/.env
    depends_on:
      - mysql

  config-ui:
    image: mericodev/config-ui:v0.10.1
    ports:
      - 127.0.0.1:4000:80
    env_file:
      - ./.env
    environment:
      DEVLAKE_ENDPOINT: devlake:8080
      GRAFANA_ENDPOINT: http://localhost:3002
      #ADMIN_USER: devlake
      #ADMIN_PASS: merico
    depends_on:
      - devlake

volumes:
  mysql-storage:
  grafana-storage:

This is how your file and folder structure should look like.

Once you set up & configure your docker-compose file, you must configure your environment variables. To do so, create a file called .env and paste the code mentioned below.

#############
# Lake core #
#############
# Lake plugin dir, absolute path or relative path
PLUGIN_DIR=bin/plugins
# Lake Database Connection String
DB_URL=merico:merico@tcp(mysql:3306)/lake?charset=utf8mb4&parseTime=True
# Silent Error Warn Info
DB_LOGGING_LEVEL=Error
# Lake REST API
PORT=:8080
MODE=release

NOTIFICATION_ENDPOINT=
NOTIFICATION_SECRET=

API_TIMEOUT=10s
API_RETRY=3
API_REQUESTS_PER_HOUR=10000
#TEMPORAL_URL=temporal:7233
TEMPORAL_URL=
TEMPORAL_TASK_QUEUE=
# Debug Info Warn Error
LOGGING_LEVEL=

########################
# Docker configuration #
########################

COMPOSE_PROFILES=user
######################
# Jira configuration #
######################

# Jira configuration has been migrated into MySQL #
########################
# Gitlab configuration #
########################

GITLAB_ENDPOINT=https://gitlab.com/api/v4/
GITLAB_AUTH=
GITLAB_PROXY=
GITLAB_API_REQUESTS_PER_HOUR=
##########################
# Jira <> Gitlab mapping #
##########################

# Lake maintains a many-to-many relationship between Jira boards and Gitlab projects
# Format:
#   JIRA_BOARD_GITLAB_PROJECTS=JIRA_BOARD_ID1:GITLAB_PROJECT_ID1,PROJECT_ID2;JIRA_BOARD_ID2:...
JIRA_BOARD_GITLAB_PROJECTS=
#########################
# Jenkins configuration #
#########################
JENKINS_ENDPOINT=
JENKINS_USERNAME=
JENKINS_PASSWORD=
JENKINS_PROXY=
JENKINS_API_REQUESTS_PER_HOUR=
#########################
# Feishu configuration #
#########################

FEISHU_APPID=
FEISHU_APPSCRECT=
FEISHU_ENDPOINT=https://open.feishu.cn/open-apis/vc/v1/

########################
# GitHub configuration #
########################

GITHUB_ENDPOINT=https://api.github.com/
GITHUB_AUTH=***
GITHUB_PROXY=
GITHUB_API_REQUESTS_PER_HOUR=
# GITHUB_PR_TYPE=type/(.*)$ the program will extract the value in (), in this example, you will get "refactor" from "type/refactor"
GITHUB_PR_TYPE='type/(.*)$'
# GITHUB_PR_COMPONENT=component/(.*)$ the program will extract the value in (), in this example, you will get "plugins" from "component/plugins"
GITHUB_PR_COMPONENT='component/(.*)$'
# GITHUB_ISSUE_SEVERITY=severity/(.*)$ the program will extract the value in (), in this example, you will get "refactor" from "type/refactor"
GITHUB_ISSUE_SEVERITY='severity/(.*)$'
# GITHUB_ISSUE_COMPONENT=component/(.*)$ the program will extract the value in (), in this example, you will get "refactor" from "type/refactor"
GITHUB_ISSUE_COMPONENT='component/(.*)$'
GITHUB_ISSUE_PRIORITY='^(highest|high|medium|low)$'
GITHUB_ISSUE_TYPE_BUG='^(bug|failure|error)$'
GITHUB_ISSUE_TYPE_REQUIREMENT='^(feat|feature|proposal|requirement)$'
GITHUB_ISSUE_TYPE_INCIDENT=
# GITHUB_PR_BODY_CLOSE_PATTERN='(?mi)(fix|close|resolve|fixes|closes|resolves|fixed|closed|resolved)[\s]*.*(((and )?(#|https:\/\/github.com\/%s\/%s\/issues\/)\d+[ ]*)+)'
GITHUB_PR_BODY_CLOSE_PATTERN='(?mi)(fix|close|resolve|fixes|closes|resolves|fixed|closed|resolved)[\s]*.*(((and )?(#|https:\/\/github.com\/%s\/%s\/issues\/)\d+[ ]*)+)'
# GITHUB_PR_TITLE_PATTERN='.*\(#(\d+)\)'
GITHUB_PR_TITLE_PATTERN='.*\(#(\d+)\)'

##########################
# AE (Merico-Analysis-Engine) configuration #
##########################
AE_APP_ID=
AE_SECRET_KEY=
AE_ENDPOINT=
AE_RATE_LIMIT=100
AE_PROXY=

##########################
# Sensitive information encryption key
##########################
ENCODE_KEY=

After you've added the environment variables, this is how your folder structure should look.

Let's launch our Apache Devlake. To start and launch Devlake, open the folder in VScode(It should be in the same directory as the docker-compose file and environment variables), then proceed to open the integrated terminal and type the command mentioned below.

docker-compose up -d

If you're executing this command for the first time, it may take some time to pull the images, so grab a cup of coffee while you wait.

However, if you have previously pulled all of the images, it will not take as long as shown in the screenshot below.

Return back to your Docker desktop, and your Apache Devlake server should now be running, as shown below.

Setup Data connection/integration with GitLab

Let us first establish our data connection before proceeding any further. To do so, first, open your browser and navigate to the following URL to access your config-ui dashboard.

http://localhost:4000/

We will be using and retrieving all data from our GitLab account for the purpose of this article tutorial. When you open the aforementioned URL in your browser, you should see something similar to this.

The next step is to connect the data connection, which you can do by clicking on the GitLab logo, as we will be fetching all of our data solely from the GitLab account.

After you click the GitLab logo, you should be routed to another page where you can add and update your connections.

After clicking on the connection, you should be able to modify the GitLab settings. The following fields are required in the connection section of the settings screen to connect to the GitLab for fetching Data from it.

Connection Name: The connection name is GitLab by default and cannot be changed.
Endpoint URL: The endpoint must be a legitimate REST API Endpoint and the URL must end with /.
Personal Access Token: To obtain a Personal Access Token, log in to your GitLab account and generate one in order to authenticate in using the API using HTTP Basic Authentication.

Configuring Personal Access Token in GitLab.

We have now successfully established a link with GitLab and Devlake. All that remains is to produce a personal access token and configure it within our Apache Devlake server, which we'll use in Devlake's Auth Section to login to the API using HTTP Basic Authentication. To do so, head over to the GitLab account and login into it.

Now, proceed to click on the avatar located in the upper right corner.

As illustrated in the screenshot below, navigate to the Access Tokens section located on the left sidebar of your profile page.

You've now reached the Token settings page, so give the token a name, including the expiry date, and then select the scope for that token before hitting Create a personal access token.

Note: Only select the read ones; the write_repository and write_history can be completely ignored.

When you click Create Personal Access Token, a token should appear, similar to this.

Copy this token and store it somewhere safe because you won't be able to access it once you exit the page.

Go back to your Devlake GitLab settings page and paste the copied Access Token from GitLab into the Auth Token options; the rest can be left default for now.

Before we proceed any further, we must first test the connection to ensure that it is fully operational. To do so, choose Test connection and a toast should display at the bottom of your screen.

Now that our connection has been created, it's time to save it. Simply press the Save Connection button located at the bottom of the page and move on to the next phase, where we'll begin to connect GitHub as well.

Configuring a GitHub Connection

Select the GitHub icon that appears on the right side of your dashboard, and now proceed with the same exact steps just like we did while connecting our GitLab account.

Once you've reached the Github Integration section, select the default connection Github from the list and proceed to the next stage.

You should be able to see something like this.

we must first establish a personal access token, which we will utilize in Devlake's Auth Section. To do so, go to your GitHub account and select Settings from the drop-down menu or you can read this article, which goes through everything in great depth.

Go to the developer setting options once you're on the settings page to configure your personal access token in GitHub.

After that, head over to the top of the screen and click the Generate New Token menu button.

Choose the choices you want to give this token repo privileges and scopes. It's totally up to you whether you use the options shown below or create your very own.

Click the Generate Token button and a token will get generated for you, which you may now use inside Devlake as an Authentication Token.

Head back to your devalake Github settings and enter that very token into the Auth Token options.

Note: rest of the options can be left as default for the time being.

We need to test the connection first to make sure it's working. To do so, pick the Test connection option, and a toast should appear at the bottom of the screen.

Now that our connection has been made, we must save it. To do so, repeat the process as before, but this time simply click the Save Connectionbutton and proceed to create a pipeline in Advanced mode. \n

Creating a Pipeline in Advanced mode

It's time to set up a pipeline now that we've successfully configured and set up both our GitHub and GitLab connections in our Apache Devlake server. To do so, select create a pipeline from the pipeline menu located on the left sidebar of the dashboard.

Once you're on the create pipeline page, scroll down to the bottom of that page and turn on Advanced Mode.

What is Advanced Mode in Apache Devlake?

In devlake's advanced mode, users can write JSON to design any pipeline. This is useful for users who want to collect multiple GitHub/GitLab repositories or Jira projects within a single pipeline, as well as for users who want fine-grained control over which entities to collect or which subtasks to run for each plugin, as well as for users who want to orchestrate a complex pipeline with multiple stages of plugins. Advanced mode also allows customers the maximum flexibility by exposing the entire JSON API.

So, after you toggle on that Advanced mode, you'll notice a Tasks Editor appear on the screen.

You can rest assured that Apache Devlake offers a significant number of templates to choose from. To access a template, simply click on the settings gear icon located next to the format option of the Task Editor, from which you may play with any type of integration.

We're going to choose and integrate our Github and Gitlab accounts in this article, so we'll need two things before we start writing the script: first, we'll need a project id from our GitLab repo, and second, we'll need a GitHub account name and repo name. Getting these details is really simple, just follow the steps below.

For Github

For GitLab

The following is an example of successively collecting 1 GitHub data and 3 GitLab repos. It is divided into four sections, each of which includes a GitLab and Github task. So, make sure that you followed the steps outlined above and added the project id and repo name to each plugin.

[
  [
    {
      "Plugin": "github",
      "Options": {
        "repo": "devlake-demo",
        "owner": "pramit-marattha"
      }
    }
  ],
  [
    {
      "Plugin": "gitlab",
      "Options": {
        "projectId": 37050144
      }
    }
  ],
  [
    {
      "Plugin": "gitlab",
      "Options": {
        "projectId": 23368216
      }
    }
  ],
  [
    {
      "Plugin": "gitlab",
      "Options": {
        "projectId": 37009301
      }
    }
  ]
]

After that, select Valid option to see if the task is valid or not, and then click on the Create Pipeline option.

You should be redirected to the Pipeline activity stage, where you should be able to monitor the activity of each task as it is being fetched in real-time by the devlake.

As you can see in the screenshot below, the Github data was successfully fetched, and the fetching process was 100% complete.

The remainder of the stages are the same as the process above, where it will fetch all of the repo data from our GitLab account. It's important to note that it'll only pull data from the repo where its project id was provided.

Visualizing data in Grafana

Let's use Grafana to monitor data and information from your GitLab and Github repositories, as well as build visualizations from them. Simply click the Grafana button next to the pipeline you just created, or go straight to the View dashboard button at the top of the sidebar, to enter the Grafana dashboard.

Grafana dashboard is now accessible, however keep in mind that the default login credentials are as follows:

Username:

admin

Password

admin

To access your repo's detailed visualization and chart, head over to your home page and click on the square dashboard menu on the left side of the screen, then click on the manage option. (Where you can find a bunch of pre-defined dashboards for us made by Devlake teams)

You'll find a variety of dashboard options to pick from, all of which were created by the Devlake team to help you create a quick visualization. Skip them all for now and instead pick Demo commit count by author and Github Metrics.

Your whole commit history data from GitLab is displayed and neatly rendered in gorgeous format inside your Demo_commit_count_by_author dashboard, as you can see, and it is real-time.

You can now choose the All option in Github_Basic_Metrics to see the visualization of your numerous repo data, but as we've integrated four of them, we'll stick with one for the time being, which will be Github one. As you can see, our Github repository(that particular repository we selected) only contains one issue, hence it is fetching that particular one in the real-time.

You may also now customize it any way you want; Grafana offers a choice of visualizations to choose from; but, before you go any further, make sure to save any sorts of changes you make. You may also play with your own query by creating a custom query and displaying your preferred visualization.

Conclusion

Congratulation! We successfully set up a pipeline in Advanced mode, then used it to simultaneously collect data from our multiple GitLab and GitHub repositories. To summarize what we did, we first used Docker to install Apache Devlake locally, then connected it to our GitLab and GitHub accounts and fetched and collected data from our multiple repositories, and finally, we created a pipeline in advanced mode and created a custom Tasks to trigger data collection, and finally, we used Grafana to visualize our data from each individual repository located across multiple platforms (i.e Github and Gitlab)

Configuring an integration pipeline in Advanced mode | DevLake