Need help to understand if azure sentinel data connection solution is being built correctly

Question

My requirement is to develop and publish a solution. Workbooks, hunting queries, analytic rules, data connectors and more will be part of the solution.

Overall, customers who use this solution should be able to provide an AWS S3 bucket as input and allow this solution to ingest data from that bucket into custom tables defined in their log analytics workspace.

For the data connector part:

1. It has to talk to AWS S3 buckets and ingest data into custom tables defined in log analytics workspace.
2. Custom tables are built based on DCR.
3. An Azure Function will be used to trigger a script
4. Script is written in python that connects to the bucket that customer provides when they deploy this solution. Once connected, script reads data from the bucket and sends events in a batch over to sentinel using log ingestion api. Some instructions are here: https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-logs-ingestion-api?source=recommendations

My question is, is this the right direction for building the data connector part of this solution.

Answer 1

You can use the below given code to send the custom logs using timer trigger function.

import json
import azure.functions as func
from datetime import datetime
import requests

app = func.FunctionApp()

@app.schedule(schedule="0 * * * * *", arg_name="myTimer", run_on_startup=True,
              use_monitor=False) 
def timer_trigger(myTimer: func.TimerRequest) -> None:
    
    time_generated = datetime.now().strftime('%Y-%m-%d %H:%M:%S') 
    logCombined = [
        {
        "TimeGenerated": time_generated,
        "Name": "Ikhtesam",
        "Computer": "Computer1",
        "AdditionalContext": "context-1"
      },
      {
        "TimeGenerated": time_generated,
        "Name": "Afreen",
        "Computer": "Computer2",
        "AdditionalContext": "context-2"
      }
    ]
    payload = json.dumps(logCombined)
    tenantId = "{tenantId}"
    clientId = "{clientId}"
    clientSecret = "{clientSecret}"
    scope = "https://monitor.azure.com/.default"
    dceUri = "https://******.eastus-1.ingest.monitor.azure.com"
    dcrImmutableId = "dcr-2e7e*******2de1"
    table = "DCR_Data_CL"

    body = f"client_id={clientId}&scope={scope}&client_secret={clientSecret}&grant_type=client_credentials"
    headers = {"Content-Type": "application/x-www-form-urlencoded"}
    uri = f"https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token"
    response = requests.post(uri, data=body, headers=headers)
    bearerToken = response.json().get("access_token")
    
    headers2 = {"Authorization": f"Bearer {bearerToken}", "Content-Type": "application/json"}
    uri = f"{dceUri}/dataCollectionRules/{dcrImmutableId}/streams/Custom-{table}?api-version=2023-01-01"
    uploadResponse = requests.post(uri, data=payload, headers=headers2)
    print("Response: ", uploadResponse.status_code) 

requirement.txt-

azure-functions
requests

While executing, I am getting the expected output.