This article describes how to use ** Alibaba Cloud **'s ** LOG Java Producer **, an easy-to-use and highly configurable ** Java ** library that helps you send data to ** Log Service **. Introducing.

This blog is a translation from the English version. You can check the original from here. We use some machine translation. We would appreciate it if you could point out any translation errors. *

background

Alibaba Cloud LOG Java Producer is for Java applications running on big data and high concurrency scenarios. A high-performance write LogHub library designed for. Compared to using APIs and SDKs, using Alibaba Cloud LOG Java Producer has many advantages such as high performance, separation of computing and I / O logic, and controllable resource usage. there is. To understand how Producer works and how it works, see the article Alibaba Cloud LOG Java Producer-a powerful tool for migrating logs to the cloud. This article focuses on how to use Producer.

How to use

There are three steps you can take to use Producer, as shown in the figure below.

Creating a Producer

The following objects are involved in creating a Producer:

Project config

The ProjectConfig object contains service endpoint information for the target project and access credentials that indicate the identity of the caller.

Service endpoint

The final access address consists of the project name and the service endpoint. For more information on how to determine project endpoints, see Service Endpoints.

Access credentials

You can set the Producer's Access Key or Security Token Service (STS) token. If you want to use the STS token, you must periodically create a new ProjectConfig object and put it in ProjectConfig.

ProjectConfigs If you need to write data to different projects, you can create multiple ProjectConfig objects and put them in ProjectConfigs. ProjectConfigs maintain the composition of different projects through maps. The key of the map is the project name and the value is the client of the project.

ProducerConfig ProducerConfig is used to set outbound policies. You can specify different values for different business scenarios. The following table describes a description of these parameters.

For details, see [TimeoutException](https://github.com/aliyun/aliyun-log-java-producer/blob/master/src/main/java/com/aliyun/openservices/aliyun/log/producer/errors/TimeoutException. java? spm = a2c65.11461447.0.0.3ab4677338bUtr & file = TimeoutException.java) and [Attempts](https://github.com/aliyun/aliyun-log-java-producer/blob/master/src/main/java/com/ See aliyun / openservices / aliyun / log / producer / Attempt.java? Spm = a2c65.11461447.0.0.3ab4677338bUtr & file = Attempt.java).

LogProducer LogProducer is an implementation class of Producer that only accepts the producerConfig parameter. After preparing producerConfig, create an instance of Producer as follows.

Producer producer = new LogProducer(producerConfig);

Creating an instance of Producer creates a series of threads. This consumes a considerable amount of resources. We recommend that you use only one Producer instance for an application. The threads for the Producer instance are listed below.

In addition, all the methods provided by LogProducer are thread-safe. These methods can be safely executed in a multithreaded environment.

Send data

After you create an instance of Producer, you can send data using the methods it provides.

Parameter description

Producer offers multiple ways to send data. The parameters of these methods are as follows.

To merge different data into a big batch, the data must have the same project, logstore, topic, source, and shardHash properties. It is recommended that you control the range of values for these five properties in order for the data merge feature to work properly and to save memory resources. If the values of a field, such as a topic, have too many different values, we recommend that you add these values to logItem instead of using the topic directly.

Acquisition of data transmission result

Producer sends data asynchronously. The data transmission result must be obtained from the future object or callback returned by Producer.

Future The send method returns ListenableFuture. In ListenableFuture, you can not only block the I / O thread and call the get () method to get the data transmission result, but also register the callback. The callback will be called after the Future settings are complete. The following snippet shows how to use ListenableFuture. You need to register a FutureCallback for this future and send the callback to the EXECUTOR_SERVICE thread pool provided by your application for execution. For the complete sample code, see [SampleProducerWithFuture.java](https://github.com/aliyun/aliyun-log-producer-sample/blob/master/src/main/java/com/aliyun/openservices/aliyun/log /producer/sample/SampleProducerWithFuture.java?spm=a2c65.11461447.0.0.3ab4677338bUtr&file=SampleProducerWithFuture.java).

ListenableFuture<Result> f = producer.send("project", "logStore", logItem);
Futures.addCallback(f,
                    new FutureCallback<Result>() {
                        @Override
                        public void onSuccess(@Nullable Result result) {
                        }

                        @Override
                        public void onFailure(Throwable t) { 
                        }
                    },
                    EXECUTOR_SERVICE);

Callback

In the future, you can also register a callback when you call the send method to get the data transmission result. The code snippet is as follows. The complete sample code is [SampleProducerWithCallback.java](https://github.com/aliyun/aliyun-log-producer-sample/blob/master/src/main/java/com/aliyun/openservices/aliyun/log/ Please refer to producer / sample / SampleProducerWithCallback.java? Spm = a2c65.11461447.0.0.3ab4677338bUtr & file = SampleProducerWithCallback.java).

producer.send(
        "project",
        "logStore",
        logItem,
        new Callback() {
            @Override
            public void onCompletion(Result result) {
            }
        });

Callbacks are implemented by Producer's internal thread. The space occupied by the batch is freed only after the corresponding callback has been executed. To avoid blocking the Producer and reducing overall throughput, avoid performing time-consuming operations on callbacks. It is also not recommended to call the send method to retry sending the producer batch. You can increase the value of the retries parameter or retry sending the batch in the callback of the ListenableFuture object.

Comparison of Future and callbacks

Should I choose Future or Callback to get the data transmission result? If the processing logic after getting the result is relatively simple and doesn't block the Producer's I / O thread, use a direct callback. Otherwise, we recommend that you use ListenableFuture to execute subsequent processing logic in a separate thread (pool).

Shut down Producer

You need to shut down Producer if you are sending less data or if you want to end the current process. Doing so will allow you to fully process the Producer's cached data.

Safe shutdown

In most cases it is advisable to shut down safely. You can safely shut down the Producer by calling the close () method. This method is only returned when all of Producer's cached data has been processed, all threads have terminated, registered callbacks have been executed, and all futures have been configured.

You have to wait for all the data to be processed, but after shutting down Producer, the cached batch will be processed immediately and will not be retried if it fails. So, unless the callback is blocked, you can usually return immediately with the close method.

Limited shutdown

If you are likely to be blocked when the callback is executed and you want a short return of the close method, use restricted shutdown mode. You can implement a restricted shutdown using the close (long timeoutMs) method. If it does not shut down completely after the specified timeoutMs, an IllegalStateException exception is thrown. In this case, Producer will be shut down regardless of whether the cached data has been processed or the registered callback has been executed.

Sample application

Alibaba Cloud LOG Java Producer Sample Application is available to make Producer learning easier. Did. The sample covers everything from Producer creation to shutdown.

Alibaba Cloud is the No. 1 (2019 Gartner) cloud infrastructure operator in the Asia Pacific region with two data centers in Japan and more than 60 availability zones in the world. Click here for more information on Alibaba Cloud. Alibaba Cloud Japan Official Page *

How to use Alibaba Cloud LOG Java Producer