- deployed using helm
datadog:
apiKeyExistingSecret: datadog-secret
appKeyExistingSecret: datadog-secret
remote_configuration:
enabled: true
clusterName: "localhost"
forwarder_storage_max_size_in_bytes: 500000000
dogstatsd_metrics_stats_enable: true
site: datadoghq.com
kubelet:
tlsVerify: false
apm:
# enabled: true
portEnabled: true
port: 8126 # default
instrumentation:
enabled: true
logLevel: TRACE
logs:
enabled: true
containerCollectAll: true
kubeStateMetricsEnabled: true
networkMonitoring:
enabled: true
orchestratorExplorer:
enabled: true
clusterAgent:
enabled: true
replicas: 2
createPodDisruptionBudget: true
- value.yml file contains the Pod is used for .net application.
# Default values for weatherforecast.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
repository: weatherforecast
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: "main"
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
# Specifies whether a service account should be created
create: true
# Automatically mount a ServiceAccount's API credentials?
automount: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
podAnnotations: {
"ad.datadoghq.com/weatherforecast.logs": '[{"source":"weatherforecast","service":"weatherforecast2"}]'
}
podLabels: {}
podSecurityContext: {}
# fsGroup: 2000
securityContext: {}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000
service:
type: NodePort
port: 80
# service:
# type: LoadBalancer
# port: 80
ingress:
enabled: false
className: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- host: localhost
paths:
- path: /
pathType: ImplementationSpecific
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
livenessProbe:
httpGet:
path: /
port: http
readinessProbe:
httpGet:
path: /
port: http
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
# targetMemoryUtilizationPercentage: 80
# Additional volumes on the output Deployment definition.
volumes: []
# - name: foo
# secret:
# secretName: mysecret
# optional: false
# Additional volumeMounts on the output Deployment definition.
volumeMounts: []
# - name: foo
# mountPath: "/etc/foo"
# readOnly: true
nodeSelector: {}
tolerations: []
affinity: {}
deployment file
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "weatherforecast.fullname" . }}
labels:
{{- include "weatherforecast.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "weatherforecast.selectorLabels" . | nindent 6 }}
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "weatherforecast.labels" . | nindent 8 }}
{{- with .Values.podLabels }}
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "weatherforecast.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: {{ .Values.service.port }}
protocol: TCP
# livenessProbe:
# {{- toYaml .Values.livenessProbe | nindent 12 }}
# readinessProbe:
# {{- toYaml .Values.readinessProbe | nindent 12 }}
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.volumeMounts }}
volumeMounts:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.volumes }}
volumes:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
Issues
- Not able to see the log of pod on datadog
- Network is not configured
- System.Probe is failed on datadog agent
- Orchestrator is partial.
- Datadog agent is failing for System.Probe only
I have been given permission to access the API as a remote configuration. Additionally, I want to push logs to DataDog from my pod. I am using the 7.51.0 version of agent and cluster. Dataagent push the details of clustes,pod,node,namespace etc details to datadog but after some time they also disappears.
orchestrator_pod
----------------
Instance ID: orchestrator_pod:888ebc42a3817b00 [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/orchestrator_pod.d/conf.yaml.default
Total Runs: 51
Metric Samples: Last Run: 0, Total: 0
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 0, Total: 0
Average Execution Time : 6ms
Last Execution Date : 2024-03-15 01:24:13 UTC (1710465853000)
Last Successful Execution Date : 2024-03-15 01:24:13 UTC (1710465853000)
telemetry
---------
Instance ID: telemetry [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/telemetry.d/conf.yaml.default
Total Runs: 51
Metric Samples: Last Run: 4, Total: 204
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 0, Total: 0
Average Execution Time : 0s
Last Execution Date : 2024-03-15 01:24:20 UTC (1710465860000)
Last Successful Execution Date : 2024-03-15 01:24:20 UTC (1710465860000)
uptime
------
Instance ID: uptime [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/uptime.d/conf.yaml.default
Total Runs: 51
Metric Samples: Last Run: 1, Total: 51
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 0, Total: 0
Average Execution Time : 0s
Last Execution Date : 2024-03-15 01:24:12 UTC (1710465852000)
Last Successful Execution Date : 2024-03-15 01:24:12 UTC (1710465852000)
Check Initialization Errors
===========================
etcd (6.1.0)
------------
instance 0:
could not invoke 'etcd' python check constructor. New constructor API returned:
Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/etcd/etcd.py", line 74, in __init__
super(Etcd, self).__init__(
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 126, in __init__
raise CheckException(
datadog_checks.base.errors.CheckException: The agent could not connect to any of the following URLs: ['https://192.168.65.3:2379/metrics', 'http://192.168.65.3:2379/metrics'].
Deprecated constructor API returned:
Etcd.__init__() got an unexpected keyword argument 'agentConfig'
kube_controller_manager (5.0.0)
-------------------------------
instance 0:
could not invoke 'kube_controller_manager' python check constructor. New constructor API returned:
Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_controller_manager/kube_controller_manager.py", line 115, in __init__
super(KubeControllerManagerCheck, self).__init__(
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_controller_manager/sli_metrics.py", line 25, in __init__
super(SliMetricsScraperMixin, self).__init__(*args, **kwargs)
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 126, in __init__
raise CheckException(
datadog_checks.base.errors.CheckException: The agent could not connect to any of the following URLs: ['https://192.168.65.3:10257/metrics', 'https://localhost:10257/metrics', 'http://192.168.65.3:10252/metrics', 'http://localhost:10252/metrics'].
Deprecated constructor API returned:
KubeControllerManagerCheck.__init__() got an unexpected keyword argument 'agentConfig'
kube_scheduler (4.8.0)
----------------------
instance 0:
could not invoke 'kube_scheduler' python check constructor. New constructor API returned:
Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_scheduler/kube_scheduler.py", line 150, in __init__
super(KubeSchedulerCheck, self).__init__(
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_scheduler/sli_metrics.py", line 25, in __init__
super(SliMetricsScraperMixin, self).__init__(*args, **kwargs)
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 126, in __init__
raise CheckException(
datadog_checks.base.errors.CheckException: The agent could not connect to any of the following URLs: ['https://192.168.65.3:10259/metrics', 'https://localhost:10259/metrics', 'http://192.168.65.3:10251/metrics', 'http://localhost:10251/metrics'].
Deprecated constructor API returned:
KubeSchedulerCheck.__init__() got an unexpected keyword argument 'agentConfig'
Loading Errors
==============
etcd
----
Core Check Loader:
Check etcd not found in Catalog
JMX Check Loader:
check is not a jmx check, or unable to determine if it's so
Python Check Loader:
could not configure check instance for python check etcd: could not invoke 'etcd' python check constructor. New constructor API returned:
Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/etcd/etcd.py", line 74, in __init__
super(Etcd, self).__init__(
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 126, in __init__
raise CheckException(
datadog_checks.base.errors.CheckException: The agent could not connect to any of the following URLs: ['https://192.168.65.3:2379/metrics', 'http://192.168.65.3:2379/metrics'].
Deprecated constructor API returned:
Etcd.__init__() got an unexpected keyword argument 'agentConfig'
kube_controller_manager
-----------------------
Core Check Loader:
Check kube_controller_manager not found in Catalog
JMX Check Loader:
check is not a jmx check, or unable to determine if it's so
Python Check Loader:
could not configure check instance for python check kube_controller_manager: could not invoke 'kube_controller_manager' python check constructor. New constructor API returned:
Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_controller_manager/kube_controller_manager.py", line 115, in __init__
super(KubeControllerManagerCheck, self).__init__(
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_controller_manager/sli_metrics.py", line 25, in __init__
super(SliMetricsScraperMixin, self).__init__(*args, **kwargs)
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 126, in __init__
raise CheckException(
datadog_checks.base.errors.CheckException: The agent could not connect to any of the following URLs: ['https://192.168.65.3:10257/metrics', 'https://localhost:10257/metrics', 'http://192.168.65.3:10252/metrics', 'http://localhost:10252/metrics'].
Deprecated constructor API returned:
KubeControllerManagerCheck.__init__() got an unexpected keyword argument 'agentConfig'
kube_scheduler
--------------
Core Check Loader:
Check kube_scheduler not found in Catalog
JMX Check Loader:
check is not a jmx check, or unable to determine if it's so
Python Check Loader:
could not configure check instance for python check kube_scheduler: could not invoke 'kube_scheduler' python check constructor. New constructor API returned:
Traceback (most recent call last):
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_scheduler/kube_scheduler.py", line 150, in __init__
super(KubeSchedulerCheck, self).__init__(
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/kube_scheduler/sli_metrics.py", line 25, in __init__
super(SliMetricsScraperMixin, self).__init__(*args, **kwargs)
File "/opt/datadog-agent/embedded/lib/python3.11/site-packages/datadog_checks/base/checks/openmetrics/base_check.py", line 126, in __init__
raise CheckException(
datadog_checks.base.errors.CheckException: The agent could not connect to any of the following URLs: ['https://192.168.65.3:10259/metrics', 'https://localhost:10259/metrics', 'http://192.168.65.3:10251/metrics', 'http://localhost:10251/metrics'].
Deprecated constructor API returned:
KubeSchedulerCheck.__init__() got an unexpected keyword argument 'agentConfig'
