Azure Log Analytics: How to send custom python pandas DataFrames into LAW

Question

I have a Python script that I want to run daily that produces a Pandas dataframe.

I want this dataframe to be added daily to my log analytics workspace.

I have a Windows server that I can use to run my Python script.

What do I need to do to make this work? Is there a way to go from DataFrames to push to a Syslog server?

Answer 1

Azure Log Analytics: How to send custom python pandas DataFrames into LAW

I have reproduced in my environment and below are my expected results:

I have taken below code from Microsoft-Document and modified a little:

import json
import requests
import datetime
import hashlib
import hmac
import base64

logtype2="RithwikLogs1"
def build_signature(customer_id, shared_key, date, content_length, method, content_type, resource):
    x_headers = 'x-ms-date:' + date
    string_to_hash = method + "\n" + str(content_length) + "\n" + content_type + "\n" + x_headers + "\n" + resource
    bytes_to_hash = bytes(string_to_hash, encoding="utf-8")  
    decoded_key = base64.b64decode(shared_key)
    encoded_hash = base64.b64encode(hmac.new(decoded_key, bytes_to_hash, digestmod=hashlib.sha256).digest()).decode()
    authorization = "SharedKey {}:{}".format(customer_id,encoded_hash)
    return authorization

def post_data1(customer_id, shared_key, body, log_type):
    method = 'POST'
    content_type = 'application/json'
    resource = '/api/logs'
    rfc1123date = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
    content_length = len(body)
    signature = build_signature(customer_id, shared_key, rfc1123date, content_length, method, content_type, resource)
    uri = 'https://' + customer_id + '.ods.opinsights.azure.com' + resource + '?api-version=2016-04-01'

    headers = {
        'content-type': content_type,
        'Authorization': signature,
        'Log-Type': log_type,
        'x-ms-date': rfc1123date
    }

    response = requests.post(uri,data=body, headers=headers)
    if (response.status_code >= 200 and response.status_code <= 299):
        print('Rithwik Data Frame is sent to Log Analytics Worksapce ')
    else:
        print("Response code: {}".format(response.status_code))
rithwik_data1 = pd.DataFrame({
    'Name': ['Rithwik', 'Bojja', 'Chotu'],
    'Age': [23, 23, 20]
})

post_data1('310603f7', 'a3m2jEErIT6HONEdrAhgIGBrT9L78AK0wk0H8HJKkEdTva4nmw==',rithwik_data1.to_json(orient="records")
, logtype2)

Output:

After running the python code, it takes some time for the logs to be generated in Log Analytics(Wait for around 5-10 min).

Note:

In calling of function:

post_data1('yourworkspaceid', 'Primarykey(sharedkey)',body(json of df), logtype2)

You can get above values by following below: