Create a custom search command of type EventingCommand for Splunk. In the article below, we created a GeneratingCommand.
This time, create an EventingCommand.
What is "EventingCommand"? For those who say, "[Four types of custom search commands](https://qiita.com/msi/items/02d029d655e1f9285806#%E3%82%AB%E3%82%B9%E3%82%BF] % E3% 83% A0% E3% 82% B5% E3% 83% BC% E3% 83% 81% E3% 82% B3% E3% 83% 9E% E3% 83% B3% E3% 83% 89% E3 % 81% AE4% E3% 81% A4% E3% 81% AE% E7% A8% AE% E9% A1% 9E) ". Regarding the development environment, the explanation of SDK etc. is omitted.
Let's create a command toupper that takes the previous output" hello, world" and converts all characters to uppercase.
SPL Example
| generatehello2 | toupper greeting ← "greeting"Is the field name
The equivalent of eval's upper () is converted by field specification.
Before we get started, let's write about the search command protocols Version 1 and Version 2.
There are various introductory articles about creating custom search commands for Python, but I mainly use the following techniques.
run command.splunk.InterSplunk module internally instead of using run.There are differences in how they handle inputs and outputs, but there are no fundamental differences.
The first can be any form that can be called from splunk.
The second uses splunk.InterSplunk for input and output.
These uses are called in protocol Version 1 or in the traditional way called the legacy protocol.
text:commands.conf.spec
chunked = [true|false]
* If set to "true", this command supports the new "chunked" custom
search command protocol.
* If set to "true", the only other commands.conf settings supported are
'is_risky', 'maxwait', 'maxchunksize', 'filename', 'command.arg.<N>', and
'run_in_preview'.
* If set to "false", this command uses the legacy custom search command
protocol supported by Intersplunk.py.
* Default: false
Quoted from "commands.conf --Splunk Documentation"
If you use Version 1, set chuncked = false in the commands.conf setting, or leave it to the default and leave it as false.
On the other hand, if you want to use the new version 2 custom search command, use the class ʻEventingCommand TransformingComannd` `GeneratingCommand StreamingCommand` instead of using splunk.InterSplunk.
The differences between the Version 1 protocol and the Version 2 protocol are as follows:
Quoted from "[About writing custom search commands --Splunk Documentation](https://docs.splunk.com/Documentation/Splunk/latest/Search/Aboutcustomsearchcommands)"About the protocols
Version 2 protocol
There are significant advantages to using the Version 2 of the Custom Search Command protocol.
Version 1 protocol
The Version 1 of the Custom Search Command protocol processes events in groups, using 50,000 events for each group. The external program is invoked and terminated multiple times for large result sets.
With v2, more languages will be supported besides Python, but the biggest difference is that the program will not be called every 50,000 events. Conversely, ** v1 launches a custom search command every 50,000 events **, which makes a difference in performance.
It makes no difference if you handle less than 50,000 events, but Splunk will soon have more than 50,000 events, so if you're going to create one, don't use splunk.InterSPlunk and v2. You should use.
In this article, we will create a custom search command using the Version 2 protocol.
HelloWorld-work directory to$ SPLUNK_HOME. Copy it to / etc / apps / HelloWorld /and restart Splunk.$ SPLUNK_HOME / etc / apps / HelloWorld / lib / and under it splunklib in the SDK Copy.
(You can also install the SDK library with pip3 install -t lib / splunk-sdk.)The structure of directories and files is as follows.
${SPLUNK_HOME}/etc/apps/
HelloWorld/
├─ bin
│ ├─ HelloWorld.py
│ ├─ HelloWorld2.py
│ ├─ filter.py
│ ├─ generate.py
│ ├─ report.py
│ └─ stream.py
├─ default
│ ├─ app.conf
│ ├─ commands.conf
│ └─ data
│ └─ ui
│ └─ nav
│ └─ default.xml
├─ lib
│ ├─ splunk_sdk-1.6.12.dist-info
│ │ (This directory exists if you installed with pip.)
│ └─ splunklib
│ ├─ __init__.py
│ ... (abridgement)
└─ metadata
└─ default.meta
Create ToUpper.py by copying filter.py under bin /.
cd bin/
cp filter.py ToUpper.py
The original filter.py is a template for EventingCommand, but it doesn't seem to be maintained and can't be used as is. (SDK v1.6.12)
ToUpper.py
#!/usr/bin/env python
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lib"))
from splunklib.searchcommands import \
dispatch, StreamingCommand, Configuration, Option, validators
@Configuration()
class %(command.title())Command(EventingCommand):
""" %(synopsis)
##Syntax
%(syntax)
##Description
%(description)
"""
def transform(self, events):
# Put your event transformation code here
pass
dispatch(%(command.title())Command, sys.argv, sys.stdin, sys.stdout, __name__)
Modify the part that is StreamingCommand in import to ʻEventingCommand`.
diff
--- filter.py
+++ ToUpper.py
@@ -5,8 +5,8 @@
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lib"))
from splunklib.searchcommands import \
- dispatch, StreamingCommand, Configuration, Option, validators
+ dispatch, EventingCommand, Configuration, Option, validators
ToUpper.py Revised
#!/usr/bin/env python
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lib"))
from splunklib.searchcommands import \
dispatch, EventingCommand, Configuration, Option, validators
@Configuration()
class %(command.title())Command(EventingCommand):
""" %(synopsis)
##Syntax
%(syntax)
##Description
%(description)
"""
def transform(self, events):
# Put your event transformation code here
pass
dispatch(%(command.title())Command, sys.argv, sys.stdin, sys.stdout, __name__)
% (command.title ()) parts to ToUpper.transform () is a required body, so add processing to this part. (Explanation will be described later)diff
--- ToUpper.py.orig
+++ ToUpper.py
@@ -9,7 +9,7 @@
@Configuration()
-class %(command.title())Command(EventingCommand):
+class ToUpperCommand(EventingCommand):
""" %(synopsis)
##Syntax
@@ -23,6 +23,11 @@
"""
def transform(self, events):
# Put your event transformation code here
- pass
+ for event in events:
+ for field in self.fieldnames:
+ if field in event:
+ event[field] = event[field].upper()
+ yield event
+ return
-dispatch(%(command.title())Command, sys.argv, sys.stdin, sys.stdout, __name__)
+dispatch(ToUpperCommand, sys.argv, sys.stdin, sys.stdout, __name__)
ToUpper.py After update
#!/usr/bin/env python
import sys
import os
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "lib"))
from splunklib.searchcommands import \
dispatch, EventingCommand, Configuration, Option, validators
@Configuration()
class ToUpperCommand(EventingCommand):
""" %(synopsis)
##Syntax
%(syntax)
##Description
%(description)
"""
def transform(self, events):
# Put your event transformation code here
for event in events:
for field in self.fieldnames:
if field in event:
event[field] = event[field].upper()
yield event
return
dispatch(ToUpperCommand, sys.argv, sys.stdin, sys.stdout, __name__)
transform () is a required method of ʻEventingCommand and is called by dispatcher`.transform () (here ʻevents`) is a list of each event.) are passed as a list to self.fieldnames`.That is the element of EventingCommand.
In the above example, for each event (for event in events:), for each specified field (for field in self.fieldnames:), convert it to uppercase and reassign it (ʻevent). [field] = event [field] .upper () `).
Check the operation on the command line separated from Splunk.
To run it locally, specify __EXECUTE__ and run it.
python3 ToUpper.py __EXECUTE__ </dev/nul
If there are no errors, it is successful.
Since / dev / null is specified for the input, there is no output, but in fact, nothing is displayed when I input.
Try it with the previously created HelloWorld2.py.
$ python3 HelloWorld2.py __EXECUTE__ count=3 < /dev/null # HelloWorld2.Operation check with py alone
_time,__mv__time,greeting,__mv_greeting,_raw,__mv__raw
1589160289.7283704,,"hello, world",,"hello, world",
1589160289.7284214,,"hello, world",,"hello, world",
1589160289.7284305,,"hello, world",,"hello, world",
$ python3 HelloWorld2.py __EXECUTE__ count=3 < /dev/null | python3 ToUpper.py __EXECUTE__ greeting
$← Return to the command prompt without displaying anything.
Note: I couldn't find a way to display it on the command line. If anyone knows about it, I would appreciate it if you could teach it.
For the time being, you can check if there is an error.
Check the operation as a Splunk search command.
If you want to modify what is provided, create a file under the local / directory and set it, but this time, from the standpoint of "provider", change the file under default / ..
commands.conf
Edit default / commands.conf.
--- commands.conf.orig
+++ commands.conf
@@ -10,3 +10,8 @@
filename = HelloWorld2.py
chunked = true
python.version = python3
+
+[toupper]
+filename = ToUpper.py
+chunked = true
+python.version = python3
chunked = true.default/commands.conf
# [commands.conf]($SPLUNK_HOME/etc/system/README/commands.conf.spec)
# Configuration for Search Commands Protocol version 2
[generatehello]
filename = HelloWorld.py
chunked = true
python.version = python3
[generatehello2]
filename = HelloWorld2.py
chunked = true
python.version = python3
[toupper]
filename = ToUpper.py
chunked = true
python.version = python3
This time, set searchbnf.conf and default.meta as Tips.
Create a new searchbnf.conf.
searchbnf.conf
[toupper-command]
syntax = toupper <field-list>
description = toupper command to transfer into upper cases by the specify fields.\
This is a long description.
shortdesc = make fields contents to upper cases.
usage = public
See also: searchbnf.conf --Splunk Documentation
[toupper-command], the toupper part is the search command name set in commands.conf, and the -command part is fixed [^ 1].[^ 1]: As an aside, UNIX roff convention is to write variable parts in italics. The bad news about MarkDown notation and HTML is that you can't write italics in \ quote \ to indicate variable parts. It's hard to tell which is a variable character and which is a fixed character, so I have to explain it every time. On the contrary, it may be difficult to understand from the document alone without explanation.
Make additional settings for searchbnf to metadata / default.meta.
--- default.meta.orig
+++ default.meta
@@ -1,2 +1,5 @@
[]
access = read: [ * ], write : [ admin ]
+
+[searchbnf]
+export = system
metadata/default.meta
[]
access = read: [ * ], write : [ admin ]
[searchbnf]
export = system
Restart splunkd [^ 2].
[^ 2]: Not sure if it will be enabled by debug / refresh.
sudo ${SPLUNK_HOME}/bin/splunk restart
Or, if set with systemctl
sudo systemctl restart Splunkd
Check the operation in the search window.
SPL statement
| generatehello2 count=3
| table greeting
Execution result
greeting
hello, world
hello, world
hello, world
SPL statement
| generatehello2 count=3
| table greeting
| toupper greeting
Execution result
greeting
HELLO, WORLD
HELLO, WORLD
HELLO, WORLD
The lowercase letters have been successfully converted to uppercase.
Recommended Posts