Trigger Azure Functions on Event Hub
Azure Event Hub is one of a suite of products offered within Microsoft Azure. This product allows you to create event-based solutions, handling millions of events per second. Ideal for IoT (internet of things) devices and is geared towards big data scenarios, such as banking, manufacturing, and many other scenarios.
Event Hub decouples the produces of the events from the consumers reading the events. It can store events for up to 7 days. This helps you scale the application to fit with different load patterns that your customers might generate. Perhaps there is more activity during the weekend. Event hub support automatic scaling out and in.
Each event can be up to 256kb in size(this may have changed since writing, review the Azure documentation for latest limitations).
So, what is an Event?
An event is simply:
a type of message
Only contains information about what happened, not what triggered it. The sender and receiver are not dependant on each other.
There are no expectations from the publisher as to how the event will be handled. The consumer will decide how it handles the event.
Events Hubs can be easily integrated with both Azure and non-Azure services.
You can automatically capture events to Azure blob storage or data lake as events enter the hub.
An event can be consumed by multiple consumers from Event Hubs by using Consumer groups
Components of Event Hubs
Namespace - Container for the event hubs
** Event produces** - Sends data to event hubs
Partitions - Buckets of messages (1-32 partitions)
Consumer groups - View of an event hub
Subscribers - The applications that read events contained within an event hub
Create an Event Hub Client
Microsoft has amazing documentation and samples, so you can easily follow what’s shown in this article: Send events to and receive events from Azure Event Hubs - .NET (Azure.Messaging.EventHubs)
The client will look something like this:
using Microsoft.Azure.EventHubs;
using System;
using System.Text;
using System.Threading.Tasks;
namespace AzureEventHubClient
{
public class Program
{
private static EventHubClient eventHubClient;
private const string EventHubConnectionString = "{Event Hubs connection string}";
private const string EventHubName = "{Event Hub path/name}";
public static void Main(string[] args)
{
MainAsync(args).GetAwaiter().GetResult();
}
private static async Task MainAsync(string[] args)
{
// Creates an EventHubsConnectionStringBuilder object from the connection string, and sets the EntityPath.
// Typically, the connection string should have the entity path in it, but for the sake of this simple scenario
// we are using the connection string from the namespace.
var connectionStringBuilder = new EventHubsConnectionStringBuilder(EventHubConnectionString)
{
EntityPath = EventHubName
};
eventHubClient = EventHubClient.CreateFromConnectionString(connectionStringBuilder.ToString());
await SendMessagesToEventHub(100);
await eventHubClient.CloseAsync();
Console.WriteLine("Press ENTER to exit.");
Console.ReadLine();
}
// Creates an event hub client and sends 100 messages to the event hub.
private static async Task SendMessagesToEventHub(int numMessagesToSend)
{
for (var i = 0; i < numMessagesToSend; i++)
{
try
{
var message = $"Message {i}";
Console.WriteLine($"Sending message: {message}");
await eventHubClient.SendAsync(new EventData(Encoding.UTF8.GetBytes(message)));
}
catch (Exception exception)
{
Console.WriteLine($"{DateTime.Now} > Exception: {exception.Message}");
}
await Task.Delay(10);
}
Console.WriteLine($"{numMessagesToSend} messages sent.");
}
}Create an Event Hub
Rather than walking you through this step by step, I’m going to defer to Microsoft who’ve put together this excellent article
To create an event hub, you’ll need to:
Create a resource group
Create an event hub namespace
Create an event hub within the new or existing namespace
Set your partition count (max 32), and message retention (max 7 days)
Once your event hub has been created, create a new shared access policy, I suggest you set send/list as permissions. Once created, copy the primary connection string. You’ll need this for deploying and testing.
Create an Event Hub .NET Core client
Again, Microsoft has amazing documentation and samples, so you can easily follow what’s shown in this article:
Send events to and receive events from Azure Event Hubs - .NET (Azure.Messaging.EventHubs)
You’ll end up with something that looks like this:
Create a function app trigger
Just create the default Azure function app trigger template for an event hub. This is shipped with visual studio.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.EventHubs;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
namespace MyFirstFunctions
{
public static class EventHubHelloWorld
{
[FunctionName("EventHubHelloWorld")]
public static async Task Run([EventHubTrigger("helloworld", Connection = "AzureEventHubConnectionString", ConsumerGroup = "$Default")] EventData[] events, ILogger log)
{
var exceptions = new List<Exception>();
foreach (EventData eventData in events)
{
try
{
string messageBody = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
// Replace these two lines with your processing logic.
log.LogInformation($"C# Event Hub trigger function processed a message: {messageBody}");
await Task.Yield();
}
catch (Exception e)
{
// We need to keep processing the rest of the batch - capture this exception and continue.
// Also, consider capturing details of the message that failed processing so it can be processed again later.
exceptions.Add(e);
}
}
// Once processing of the batch is complete, if any messages in the batch failed processing throw an exception so that there is a record of the failure.
if (exceptions.Count > 1)
throw new AggregateException(exceptions);
if (exceptions.Count == 1)
throw exceptions.Single();
}
}
}When I went to build, this is had two problems. Firstly the using Microsoft.Azure.EventHubs was showing as missing. I had to go and reference that NuGet package manually.
Secondly, the EventHubTrigger attribute was shown as unknown… (.net core), so it wouldn’t build. After some digging, I found this GitHub issue, which mentions that you’ll need to reference Microsoft. Azure.Web jobs.extensions.eventhubs. This is echoed in this [GitHub issue] (https://github.com/Azure/azure-webjobs-sdk/issues/1558)
You may also find that when you start your local Azure function emulator that you get an error like method not found microsoft.azure.eventhubs.eventhubclient to fix this you’ll need to reference a the following nuget package Microsoft.Azure.EventHubs.Processor
Make sure to add the hub connection strings to your local.settings.json file:
When you deploy for the first time, you may see an error like this: Microsoft.Azure.WebJobs.Host: Error indexing method (..) Microsoft.Azure.WebJobs.EventHubs: Value cannot be null. Parameter name: receiverConnectionString
If you do, simply go back to the publish screen and hit Edit Azure App Service Settings Edit the settings and set the connection string to that of your event hub (the one we copied earlier)
This should now deploy correctly.
Wrap Up
Azure functions are a fantastic tool for integrating with event hub.