[PYTHON] Future of Blackbird

By the last round of the Advent Calendar (actually it's 2015, but www), I would like to talk about what kind of implementation will be done in the future under the title of Future of Blackbird, not only function implementation but also deb package and RPM.

About pip

Actually, there are some plugins that have been upgraded to pip, and some that have not been upgraded. I think we have to do a little more integration in that area.

In the end, I feel that it will be easier if all plugins can be installed with pip install blackbird-*.

About blackbird Documentation

Right now, the official documentation is here. To be honest, I just displayed README.md on GitHub Pages, which I wrote with a little effort.

So I feel that it's bad if I don't do Documentation properly. This time, I'd like to write Documentation such as Qiita's Advent Calendar, which is just like the content. Documentation I write it properly.

About blackbird

Then there are various things about blackbird itself.

Using embedded Python

Like the RPM package and the deb package, there are sys.path issues, module imports, and so on. There was talk of making Python embedded to avoid conflicts ... (It seems to be called fluentd)

Actually, there was a little background like the one below, isn't it?

1. AWS's ECS (EC2 Container Sercice) agent (ecs-agent) is supposed to be started by docker, but ecs-agent is written in golang, ...
2. Golang is suitable for monitoring agents if you only need to distribute pre-built binaries ??
3. No, golang is overkill, what to do with existing plugins ?? If you have to fork and get it, it's tough.
4. Wouldn't it be happy if you made embedded Python and included it in the package ??
5. Well, that's the point

That's why I decided to do embedded Python. By the way, I'm thinking that the embedded Python version is ** 2.7.X **.

Implementing blackbird client mode

I'm talking about wanting to implement client mode. If you make it embedded, honestly plugin management will be under embedded site-packages just by specifying the full path of pip. I think there are talks about wanting the latest version on github, and when I have an oleore plugin in the first place, I don't think pip is all that is needed.

So for example

blackbird-plugin install dynamodb

If you do something like pip install,

blackbird-plugin install Vagrants/blackbird-dynamodb

If you do something like git clone https://github.com/Vagrants/blackbird-dynamodb.git I'm thinking of aiming for such an implementation. Besides that

• Get the version and list of currently installed plugins (blackbird-plugin list)
• Get what value you can get with the currently installed plugin in one shot one shot mode

• This can be used to check if the value can actually be obtained, and if it can also be used for debugging.

About logging

Right now it's in LTSV format (even though it's just log_level and time, Process name, Thread name, message), The LTSV format is not too domestic in the first place,

"%(time) [%(log_level)] %(threadName) %(contents)"

I'm talking about how to make it simpler.

After that, I'm spitting directly to the file now, but I think it would be nice if I could spit it to syslog as well and style it as I like.

Implementing Multiple Queue and Importing Multiple Sender Plugin

I'm sorry that the current state is very disappointing, but since there is only one Queue for one blackbird process, If you send it once, it ends there, but this has some problems as follows (I think you all know)

1. Cannot send items to multiple Zabbix Servers
2. I can't seem to want to send it to Zabbix Server and Grafana because the message disappears when one sender plugin retrieves the item.

So I'm talking about what I want to do, implement multiple Queues, and then route which Queue is for which sender plugin. I was talking about having to implement it, and it was added to the future task by +1.

Tasks

From here on, it's more like our own Task than a new feature, but if you add the finer level ones,

• Elimination of Config OBJ
• Signal handling
• --configtest implementation
  • /usr/bin/blackbird --test or --configtest
• Test! (Unit) test! (Integration) Test!

It looks like this.

Kicking out ConfigObj

I'm using ConfigObj, an extended version of ConfigParser that also does validation. Actually, this is a new version, but it's a module that hasn't been maintained for about 2 years ...

Let's get rid of this and write a parser and a validator on our own.

Implementing --configtest option

Often, you do something like /etc/init.d/nginx configtest, which makes it easier to operate even in a configuration management application. It's a story, and I'm going to implement it.

Handling any signals (HUP, TERM, USR1)

In the first place, I was talking about /etc/init.d/blackbird reload, but for that purpose, when I received SIGHUP You have to stop Thread, stop config, and do something like that. So, I also thought that it was necessary to spit out all Queues when receiving SIGTERM and stop after sending. Therefore, we will do signal handling.

Test! Test! Test!

Test I will do my best because there are quite a few sweet spots.

It looks like this! The Advent Calendar is actually 2015 with plenty of room, but I think everyone will start working from tomorrow. Let's work hard in 2014 as well.