Microsoft Fabric Updates Blog

From RabbitMQ to PowerBI reports with Microsoft Fabric Real-Time Analytics

In this blog I’ll show you an End-to-End scenario, where you can send humidity and temperature telemetry data from RabbitMQ to a KQL Database in Microsoft Fabric. You will be using an Eventstream to orchestrate the data streaming. Once your data is in a KQL Database, you’ll be able to visualize it in PowerBI to gain insights from it.

Here’s the full demo at work:

Sending data to Microsoft Fabric from RabbitMQ

Creating a KQL Database

In Microsoft Fabric, select the Synapse Real-Time Analytics experience. Then create a new KQL Database that will hold the streamed data. Call it telemetry.

Once the database has been created, go back to the Synapse Real-Time Analytics experience, and create a new Eventstream. Call it from-rmq.

Then you will add a new Custom App source. This will let you receive data from RabbitMQ. Call the Custom App rmq-ca.

Configuring a RabbitMQ Shovel

Once the Custom App is created, it’s time to get the AMQP URI. In the Custom App details pane, select the AMQP tab, and copy the Connection string-primary key.

Then you can go to RabbitMQ management and create a new shovel. A RabbitMQ shovel will take care of grabbing messages from your RabbitMQ telemetry queue, and ship them to Fabric. You need the RabbitMQ Shovel plugin enabled for this to work.

Call the shovel to-fabric. In this example, the telemetry queue in RabbitMQ is called rmq-fabric. The shovel destination will be the AMQP 1.0 Custom App you just created. In the URI field paste the AMQP URI you copied from Fabric. Make sure to add the query parameter ?verify=verify_none. (This is needed since Erlang/OTP Version 26.0).

Go back to your Custom App and copy the Entity Name, this will be used as the Address in the shovel configuration. Select Add Shovel to finish the shovel setup.

Sending telemetry data to RabbitMQ

In your dotnet project, add the RabbitMQ.Client NuGet package:

dotnet add package RabbitMQ.Client

Then you can use the following code to send telemetry data to RabbitMQ:

using System.Text;
using RabbitMQ.Client;

var factory = new ConnectionFactory() { HostName = "localhost" };
using (var connection = factory.CreateConnection())
using (var channel = connection.CreateModel())
    channel.QueueDeclare(queue: "rmq-fabric",
                        durable: true,
                        exclusive: false,
                        autoDelete: false,
                        arguments: null);

    int i = 0;
    while (true)
        var message = GenerateRandomMessage();
        var body = Encoding.UTF8.GetBytes(message);

        channel.BasicPublish(exchange: "",
                            routingKey: "rmq-fabric",
                            basicProperties: null,
                            body: body);
        if (i++ % 100 == 0)
            Console.WriteLine(" [x] Sent {0}", message);

string GenerateRandomMessage()
    var random = new Random();
    var temperature = random.Next(0, 50);
    var humidity = random.Next(0, 100);
    var timestamp = DateTime.UtcNow.ToString("o");
    return $"{{\"Timestamp\": \"{timestamp}\", \"Temperature\": {temperature}, \"Humidity\": {humidity}, \"Source\": \"rabbitmq\"}}";

Save the code in a file called `Program.cs`. Then you can run it with:

dotnet run

This will start sending telemetry data to RabbitMQ.

Creating a KQL Table

Back in your Eventstream, now you will configure the destination. Select New destination -> KQL Database. Call it rmq-dest, then choose My Workspace, and specify the telemetry database from the KQL Database dropdown. Select Create and configure.

Here go through the steps in the wizard. Call the table sensor-telemetry, and make sure to select JSON as the Data format. Fabric will then parse the data sent from RabbitMQ and create a table with the appropriate columns: Timestamp, Temperature, Humidity, and Source.

Querying the data

Once the Eventstream has been fully setup, you will be able to go to the KQL Database, and query the data. Select the `telemetry` database, and then select the `sensor-telemetry` table.

In the query editor we will run the following query:

| summarize ptemperature = avg(Temperature), phumidity avg(Humidity) by bin(Timestamp, 1s)

This will give you the average temperature and humidity per second. Select Run to execute the query. Then select Build PowerBI report to create a new PowerBI report.

Creating a PowerBI report

In the PowerBI editor, you will add two line charts. The first one will display the phumidity values, while the second one will display the ptemperature values.

Then you can select File -> Save to publish the report to PowerBI. Once the report is published, you can open it in PowerBI. There you will be able to share the dashboard with your colleagues. Congratulations, you just consumed data from RabbitMQ into PowerBI using Microsoft Fabric Real-Time Analytics.

Learn More

Microsoft Fabric Real-Time Analytics

Kusto Query Language

RabbitMQ Shovel

Related blog posts

From RabbitMQ to PowerBI reports with Microsoft Fabric Real-Time Analytics

November 22, 2023 by Alex Lin

At Microsoft Ignite this year, we were thrilled to announce the General Availability of Eventstream. One of the exciting features is Eventstream’s real-time processing tailored for seamless integration with the KQL Database. This feature ensures a smooth experience for ingesting and transforming data streams into your KQL table. In this blog post, we’ll dive into … Continue reading “Explore Data Transformation in Eventstream for KQL Database Integration”

November 20, 2023 by Anshul Sharma

As part of the One logical copy effort, we’re excited to announce that you can now enable availability of KQL Database in Delta Lake format. Delta Lake  is the unified data lake table format chosen to achieve seamless data access across all compute engines in Microsoft Fabric. The data streamed into KQL Database is stored … Continue reading “Announcing Delta Lake support in Real-Time Analytics KQL Database”