SUSE Support

Here When You Need Us

How to test rancher-logging Flows and Outputs with local file output

This document (000021317) is provided subject to the disclaimer at the end of this document.

Environment

  • A Rancher v2.5+ managed Kubernetes clusters
  • rancher-logging installed in the cluster

 

Situation

When configuring rancher-logging on a cluster, it may be necessary to troubleshoot Flows and Outputs. This guide provides a way to test these without any third-party tools involved (e.g. an external logging destination such as Elasticsearch).

Resolution

  1. The first step will be to create an Output on the cluster-logging-system namespace. You can do so by exploring the cluster within the Rancher UI, navigating to Logging > Outputs in the resources menu on the left side, and then clicking Create (alternatively, you can apply the manifest via the kubectl CLI). The following example manifest will create a basic Output that you can customize later. It will send the resulting logs to the /tmp/logs/ folder on the FluentD pod (rancher-logging-root-fluentd-0) in the cluster-logging-system Namespace:
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: Output
    metadata:
     name: test-output
     namespace: cattle-logging-system
    spec:
     file:
        path: /tmp/logs/${tag}/%Y/%m/%d/%H.%M
        buffer:
          timekey: 1m
          timekey_wait: 10s
          timekey_use_utc: true
    
  2. The next step is to create a Flow that targets the pod created in step 3, using a label selector, and routes logs to the Output created in step 1. Again, this can be done from the UI by clicking on the Logging > Flows tab and clicking Create (or via kubectl):
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: Flow
    metadata:
      name: test-flow
      namespace: cattle-logging-system
    spec:
      localOutputRefs:
        - test-output
      match:
        - select:
            labels:
              logging-test: 'true'
    
  3. Next, a dummy pod will be deployed to generate log messages with which to test the Flow and Output. In this example the swiss-army-knife debug image created by Rancher Support is used, but you can replace this as required by your testing. You can create this Pod via the Import YAML function at the top-right of Rancher the UI or via the kubectl CLI:
    apiVersion: v1
    kind: Pod
    metadata:
      name: test-logging-pod
      namespace: cattle-logging-system
      labels:
        logging-test: "true"
    spec:
      containers:
      - name: sak
        image: rancherlabs/swiss-army-knife:latest
    
  4. With the Output, Flow, and Pod in place, you can now test the setup by executing a shell in the test pod and sending any desired test log messages to STDOUT (before doing this, be sure that the fluentd-configcheck pod in the cattle-logging-system Namespace is in status Completed):
    $ kubectl -n cattle-logging-system exec -it test-logging-pod -- /bin/bash
    # echo 'Hello World from Rancher Logging!' > /proc/1/fd/1
    
  5. To view the results, you have to go to the /tmp/logs folder that will be created in the Fluentd pod (you might have to wait up to a few minutes for the log file to be populated, due to container log scraping and buffering by Fluent Bit). Inside the /tmp/logs directory will be a directory for the test Pod, and the log file will be in a child directory under the day, month and year:
  6. $ kubectl -n cattle-logging-system exec -it rancher-logging-root-fluentd-0 -- /bin/sh
    
    $ ls -lrt /tmp/logs
    total 4
    drwxr-xr-x    3 fluent   fluent        4096 Feb 28 15:06 kubernetes.var.log.containers.test-logging-pod_cattle-logging-system_sak-8611b0ee3869dc245a4921aac63242901b104f77c86c7d58a32caafafe59aaaa.log
    
    $ cat 15.05_0.log 
    2024-02-28T15:05:50+00:00       kubernetes.var.log.containers.test-logging-pod_cattle-logging-system_sak-8611b0ee3869dc245a4921aac63242901b104f77c86c7d58a32caafafe59aaaa.log   {"log":"Hello World from Rancher Logging!\n","stream":"stdout","time":"2024-02-28T15:05:50.766535534Z","kubernetes":{"pod_name":"test-logging-pod","namespace_name":"cattle-logging-system","pod_id":"1d16a19e-f565-447b-b9f0-39bdf559eaf6","labels".....
    
  7. You can now freely change your Output and Flow configuration to test and troubleshoot configurations, filters and parsers etc. The results will appear in the /tmp/logs/ folder of the Fluentd pod.

Additional Information

Provided you followed the steps above, you can now change the created Output and Flow to test any custom configurations that you need. You should be able to see the results right away and quickly spot any problems. Remember that you can use the following KB to guide you in troubleshooting issues: How to troubleshoot rancher-logging | Support | SUSE

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

  • Document ID:000021317
  • Creation Date: 02-Jan-2024
  • Modified Date:08-Mar-2024
    • SUSE Rancher

< Back to Support Search

For questions or concerns with the SUSE Knowledgebase please contact: tidfeedback[at]suse.com

tick icon

SUSE Support Forums

Get your questions answered by experienced Sys Ops or interact with other SUSE community experts.

tick icon

Support Resources

Learn how to get the most from the technical support you receive with your SUSE Subscription, Premium Support, Academic Program, or Partner Program.

tick icon

Open an Incident

Open an incident with SUSE Technical Support, manage your subscriptions, download patches, or manage user access.