The road to updating Splunkbase with your own Splunk app for Python v2 / v3

This article is one of the articles in Splunk Advent Calendar 2019. Please also enjoy Other Articles.

This time, it's good to publish your own App on Splunkbase, It's about time (?!) I have to support Python 3 ... Splunk also Migrate to Python 3!

This is for Splunk App developers who are busy with the deadline.

To briefly introduce myself, In my daily work, I'm mainly in charge of building and troubleshooting Splunk. My favorite feature is the Metrics type index.

Briefly about the background of the article.

What is Python 3 migration in the first place?

Python, an interpreted programming language, The v2 and v3 series have been used for a long time, Maintenance of v2 series (v2.7) by the development community Ended on 2020/1/1 It has been announced 12 years ago (2008).

Now, 2019 is finally over a little. It's time to raise your heavy hips.

By the way, just after the maintenance is finished After 2020/1, it doesn't stop suddenly. Just to be sure.

In the first place, it should be written for Python 3.

Yes. You're right.

However,

The library you are using does not yet support v3.
I'm not sure about the repair points (Is it just unstudied ...).
It's not good because it's moving.

I think that there are many cases where it is postponed for some reason.

Splunk's Python affairs.

What's more, there are some special circumstances with Splunk.

Software called Splunk ** Python runtime environment is included **. Therefore, without installing Python separately on the OS side, It was available as a plugin (App, Add-on).

This bundled Python will be available until just the other day (October 22, 2019) I was in a situation where I could only choose Python 2.7.

Python 3.7, a v3 family, began to be included in Splunk v8.0.0.

Now I can't use the environment as an excuse (it's a misleading nuance).

By the way, v8.0.0 includes Python 2.7 and Python 3.7. Both environments are included and For each function used in App and Add-on You can choose which environment to run in. By default, it works with the traditional v2.7 series (However, with some exceptions such as web application functions).

As a future roadmap, By mid-2020, the default Python will change from v2.7 to v3.7 (v2.7 can be selected). In the second half of 2020, it seems that v2.7 will be deprecated and only v3.7 will be included.

Just rewrite to v3.7?

As a point to note when migrating ** v2 / v3 compatible notation is highly recommended **.

Well, you said that v2 will not be maintained by EOL, The opinion is reasonable, but ...

If you're familiar with Splunk, you might be familiar with it, There is a function called distributed search.

Splunk can be used all-in-one in a single instance, or it can be used. It is also possible to separate instances for each major function to improve search efficiency and availability. As the main actor,

Search head (SH) that accepts searches and displays the results
Indexer (IDX) that returns data based on data storage and search
Forwarder (FWD) that captures new data and transfers it to the indexer

It can be divided into. When this distributed configuration is adopted, The search command received by the search head, To multiple indexers (also known as peers), Request distributed search processing.

At this time, the App on the search head also It is passed to the indexer as part of the instruction and executed on the indexer.

With a custom search command, etc. If you have written a process using Python, Searchhead and indexer settings (specifying Python environment) It would be nice if they were the same Search head is v3.7 compatible, If you specify that the indexer works with v2.7, v3 compliant, v2 incompatible notation Distributed processing on the indexer fails The scenario is waiting.

For such a situation, the notation for both Python v2 / v3 is Recommended.

What is the difference between Python v2 / v3?

It seems that it will gradually deviate from the main subject I'll wrap up the details of the differences related to the Python version, Please refer to other articles for details.

Porting from Python 2 to Python 3
Changes from Python 2 to Python 3.0
[With an example the decisive difference between Python 2.7.x and 3.x](https://postd.cc/the-key-differences-between-python-2-7-x-and-python-3- x-with-examples /)
Because I can't migrate to Python3 smoothly, let's consider compatibility 2-3

In my experience

print from statement to function (requires parentheses after print)
The behavior of the division operators (/, //) has changed.
The character code support that I struggled with was malfunctioning.
The reduce etc. that I used to go through have been moved to functools.
ConfigParser is now configparser.

I remember being addicted to it.

When rewriting to compatible notation, I would like to introduce two, six and tox. In a nutshell, six is for absorbing differences such as function names, tox is a unit test for both v2 / v3.

Differences between Python v2 and v3 Efforts to absorb in the library six is It's also included in the library for Splunk App development.

Also, Python 2 and Python 3 respectively To run unit tests in your environment tox A framework is also available. Unit test of the same code, It can be run in multiple interpreter environments (such as Python v2 and v3).

Until the check that there is no grammatical problem To some extent, it can be handled mechanically with a framework or the like.

Rewrite your own Splunk App.

By the way, the pillow of the story has become quite long, but it's about time for the main subject.

For app compatibility check, The Splunk Platform Upgrade Readiness App (https://splunkbase.splunk.com/app/4698/) is also available, You may need a Splunk Enterprise environment, so First, let's take a quick look at AppInspect's REST API. Later, we'll talk about the Splunk Platform Upgrade Readiness App later.

Compatibility Check Part 1: AppInspect REST API

Obtaining an authentication token

$ curl -X GET \
       -u {splunk.com username} \
       --url "https://api.splunk.com/2.0/rest/login/splunk"

You will be asked for your password Enter the password for your splunk.com user account.

{
    "data": {
        "token": "{Token string. Less than,$See in TOKEN}",
        "user": {
            "email": "{Registered email address}",
            "groups": [
                "Beta Users"
            ],
            "name": "{Registered name}",
            "username": "{splunk.com username}"
        }
    },
    "msg": "Successfully authenticated user and assigned a token",
    "status": "success",
    "status_code": 200
}

Here, we will check the token string in the response using the following steps. (The token string is quite long, so it may be better to put it in a shell variable etc.)

In the following, it is assumed that the token obtained here is set in the environment variable TOKEN. (Refer to $ TOKEN)

Issuance of App inspection request

$ curl -X POST
       -H "Authorization: bearer $TOKEN"
       -H "Cache-Control: no-cache"
       -F "app_package=@\"line-alert-for-splunk_100.tgz\""
       -F "included_tags=py3_migration"
       --url "https://appinspect.splunk.com/v1/app/validate"

{
    "request_id": "a32e91f8-7767-400f-a33f-xxxxxxxxxxxx",
    "message": "Validation request submitted.",
    "links": [
        {
            "rel": "status",
            "href": "/v1/app/validate/status/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx"
        },
        {
            "rel": "report",
            "href": "/v1/app/report/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx"
        },
        {
            "rel": "package",
            "href": "/v1/app/package/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx"
        }
    ]
}

Use the request ID included in the response to query the results.

Check App inspection status and results

To check the progress of the inspection request Query the status endpoint.

$ curl -X GET
       -H "Authorization: bearer $TOKEN"
       -H "Cache-Control: no-cache"
       --url https://appinspect.splunk.com/v1/app/validate/status/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx

{
    "request_id": "a32e91f8-7767-400f-a33f-xxxxxxxxxxxx",
    "status": "SUCCESS",
    "info": {
        "error": 0,
        "failure": 0,
        "skipped": 0,
        "manual_check": 0,
        "not_applicable": 3,
        "warning": 0,
        "success": 10
    },
    "links": [
        {
            "rel": "self",
            "href": "/v1/app/validate/status/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx"
        },
        {
            "rel": "report",
            "href": "/v1/app/report/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx"
        }
    ]
}

If the status is SUCCESS, you are done.

Use the report endpoint to get the results.

$ curl -X GET
       -H "Authorization: bearer $TOKEN"
       -H "Cache-Control: no-cache"
       --url https://appinspect.splunk.com/v1/app/report/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx

If not specified, json will be returned. You can also get the result in HTML by specifying Content-Type as appropriate.

$ curl -X GET
       -H "Authorization: bearer $TOKEN"
       -H "Content-Type: text/html"
       -H "Cache-Control: no-cache"
       --url https://appinspect.splunk.com/v1/app/report/a32e91f8-7767-400f-a33f-xxxxxxxxxxxx

This is easier to see. If Failures and Error in the Totals column are zero, it's a relief.

Compatibility Check # 2: Splunk Platform Upgrade Readiness App

Next, in the Splunk Platform Upgrade Readiness App, I will introduce the check method.

Originally, when upgrading from Splunk v7.x to Splunk v8.0 It's an app to check, It is convenient because you can also check for Python 3 compatibility.

However, to do Upgrade source Splunk Enterprise v7.1, v7.2, v7.3 Either environment is required. In addition, it should be noted ** Doesn't work with upgraded Splunk 8! **

If you are lucky enough to have the target v7.x series, install the App and From the App list in the Splunk Web UI Open the Splunk Platform Upgrade Readiness App.

Select the Run New Scan button in the upper right From the Scan Settings pull-down Select Scan custom selection of apps.

A list of installed apps will be displayed on the right side of the screen. Select the app you want to check from the list.

Select the Scan button.

Wait for a while and you should see the result. It depends on the files included in the app, but it can take a few minutes to a few tens of minutes, so Is it good to wait patiently?

When completed, the GUI will display Scan completed You can see the scan results.

By the way, the operation log of Upgrade Readiness App is In $ SPLUNK_HOME / var / log / upgrade_readiness_app / upgrade_readiness.log It will be output.

2019-12-11 16:57:44,671 INFO 140663382782080 - Scan initiated
2019-12-11 16:57:44,671 INFO 140663382782080 - Retrieving key to write progress
2019-12-11 16:57:44,771 INFO 140663382782080 - Found key for existing entry: 5df0a1788f02502eb0569f21
2019-12-11 16:57:44,829 INFO 140663382782080 - Total 1 apps found for user: admin
2019-12-11 16:57:44,905 INFO 140663382782080 - 0 apps out of 1 scanned. Scanning App: Microsoft Teams alert for Splunk
2019-12-11 16:57:59,288 INFO 140084004323456 - Handling a request
2019-12-11 16:57:59,289 INFO 140084004323456 - Executing function, name=get_read_progress

(Omission)

2019-12-11 17:19:17,115 INFO 140663382782080 - Deployment scanned successfully for user: admin

In Public Documents (https://docs.splunk.com/Documentation/UpgradeReadiness/2.0.0/Use/Use),

Some Splunk apps are too large to scan. If you cannot scan a Splunk app, follow the app's documentation for updates on Python 3 readiness. I also find a proviso that says, so I pray not to fail.

It's not productive to wait while squeezing, Let's take a look at the refurbishment points for the Splunk App. If you look at the public document Act on scan results (https://docs.splunk.com/Documentation/UpgradeReadiness/2.0.0/Use/Use#Act_on_scan_results), The following items are described.

Remove Advanced XML (deprecated in Splunk v6.3.0).
Disable Splunk Web Legacy mode (deprecated in Splunk v6.4.0).
Do not use the file name test.py.
Do not use the M2Crypto library.
Make the Mako template for the application server feature Python 3 compatible.
Update the CherryPy endpoint for the application server feature.
Update other Python files.

It is also a good idea to carry out the parts that can be dealt with in advance.

When the scan is complete, a list of results will be displayed.

Check 7: You can find that the Status: of Python scripts is ** Warning **.

This app is modeled with the slightly older Add-on Builder, so It seems that the files are completely stuck.

It is displayed on the right side of each item in the Issues list. Click the See Issues link The correction points are displayed (this is useful).

Scan for Upgrade Readiness App It looks like you're looking at a Python file independently (my imagination) Even when v2 / v3 is conditional branched and processed in another file, etc. It seems that there is a possibility that it will be a candidate for repair. I'm scared of the number of scan results for a moment, Check it out and fix it if necessary.

(Extra edition) I found a problem with the compatibility of third-party apps other than my own.

If it was published on Splunkbase, Let's look for an updated version.

If support continues by the author and maintainer It may be a good idea to ask for repairs.

If you have the courage, it may be a good idea to repair it yourself, After confirming that there is no problem with the license, while considering the operation impact on others, I think it would be good to be supported. When it's sunny and the corresponding version is completed You can also contact the author and have them merge It may be the real thrill of open source.

Upload updated version to Splunkbase.

Finally, upload the updated version and reapply.

Select the target app from Your Apps and select From the Administrator Tools menu at the top Select Manage App.

Select New Version at the top right.

The Splunkbase Developer Distribution Agreement will be displayed. Agree

A drop area called Version: New Release will be displayed. Upload a new version of the app.

You will be taken to a page where you can write Release Notes, etc., so please write them as appropriate. There is also a Splunk Version Compatibility column, so I also checked 8.0.

If you save, the application is complete. After that, let's pray that the examination will pass safely.

Finally.

In this article, how to use Splunk such as search, Without touching on a convenient Add-on introduction or how to use a nifty configuration, I've put the spotlight on how to develop and publish Splunk App.

I would appreciate it if you could find something useful.

Of course, the updated version of the app It's also important to run it in a Splunk v8 environment and check its operation.

Not limited to Splunk App, when modifying applications It may also be required to check the part that there is no difference in the business logic. It may be an ideal theory, but I prepared a unit test properly and prepared it properly. Checking that the same result can be obtained with Python 2 and Python 3 with tox etc. I think it's a way to prevent regression.

Regarding the new release of your own App, I have written an article called My First Splunk App on Splunkbase, so if you are interested, please have a look.

Well then, Happy Merry Christmas !!