Get video information from Nikorepo and throw it to Slack

This is the article on the 24th day of Nandemo Ali Advent Calendar 2018. It's already 25 days, but I think one day is an error. The code itself can be found on github.

Introduction

It is a story of a bot that automatically posts the video URL to Slack from Nico Nico Douga's Nico Repo. Actually, it is not completed yet, but after that it is good that it is not the main point such as executing it regularly and dealing with unexpected input. I will do the rest little by little after the master's thesis.

Thing you want to do

Nikorepo, unless you follow the poster on Twitter, it is essential to search for new videos of your favorite series. To be honest, there is a lot of noise and it is annoying. I just want information on new videos, but depending on the video, I filled in the Nikorepo with "Posted", "Registered in my list", "Nikoni advertised", "n played", "ranked in nth place" I will do my best. There is a mute function, but since it is a function to mute a specific notification of a specific user, it cannot be deleted as you want to delete the content you want to delete.

As for the mood, once the video information appears, I want to exclude the same video from Nikorepo for a while. So I want to make a bot that gets only video information from Nikorepo, deletes duplicates, and then throws it to Slack.

It's hard to understand, but like this, it becomes a bot that puts the URL of the video without duplication.

Implementation

To be honest, I don't write anything because I have hair on Hello World.

HttpClient In fact, what I wanted to do was half a bonus, and the main purpose was to try using the classes around HttpClient added in Java 11. When sending HttpRequest in Java Until a while ago, it was like kneading HttpURLConnection or using a library such as Apache Commons, but HttpClient has been added from Java 11.

I've omitted it subtly, but it's OK if you throw the header value into the builder, build it, and send it. It's very easy to understand compared to something like HttpURLConnection that opens and casts. If you are using Java 11, I feel that you can write without problems without using the library.

HttpClient.java

	private Map<String, String> headers;
	private HttpClient httpClient;

	public HttpResponse<String> sendPost(String messages) throws IOException, InterruptedException {
		Builder builder = HttpRequest.newBuilder(URI.create(uri));
		headers.entrySet().stream()
				.forEach(entry -> builder.header(entry.getKey(), entry.getValue()));
		builder.POST(BodyPublishers.ofString(messages));
		return httpClient.send(builder.build(), BodyHandlers.ofString());
	}

Slack RTM API

Just connect WebSocket to the wss returned by throwing a post to the connect url prepared by Slack. If you don't ping regularly, the connection will be cut off, so for the time being, ping is thrown every second.

SlackClient.java

public class SlackClient{
	
	private ConnectionInfo connectionInfo;
	private SlackListener listener = new SlackListener();
	private ResponseProcessor processor = new ResponseProcessor();
	private SlackSpeaker speaker;
	private String token;

	public SlackClient(String token) {
		this.token = token;
	}

	public static void main(String[] args) {
		if(args.length != 1)
			return;
		SlackClient client = new SlackClient(args[0]);
		try {
			client.start();
		} catch (IOException | InterruptedException | ExecutionException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}
	
	private boolean connect() throws IOException, InterruptedException {
		Map<String, String> headers =  Map.of("Content-type", "application/x-www-form-urlencoded");
		Map<String, String> postMessages = Map.of("token", token);
		SimpleHttpClient httpClient = new SimpleHttpClient(SlackAPI.CONNECT.getURIText(), headers, postMessages);
		
		HttpResponse<String> response = httpClient.sendPost();
		
		Gson gson = new Gson();
		ConnectionInfo connectionInfo = 
				gson.fromJson(response.body(), ConnectionInfo.class);
		this.connectionInfo = connectionInfo;
		System.out.println(gson.toJson(connectionInfo));
		
		return connectionInfo.isSucceed();
	}
	
	private WebSocket createWebSocket() throws InterruptedException, ExecutionException {
		HttpClient client = HttpClient.newHttpClient();
		CompletableFuture<WebSocket> future = client
				.newWebSocketBuilder()
				.buildAsync(URI.create(connectionInfo.getURI()), listener);
		return future.get();
	}
	
	public void start() throws IOException, InterruptedException, ExecutionException {
		connect();
		
		speaker = new SlackSpeaker(createWebSocket());
		listener.subscribe(processor);
		processor.subscribe(speaker);
		
		while(true) {
			speaker.sendPing();
			Thread.sleep(1000);;
		}
	}
}

Receiving messages from slack, processing message contents, and sending messages are implemented separately because I wanted to separate the classes.

For the implementation of this area, java.util.concurrent.Flow introduced from Java 9 is used. Publisher, Processor, and Subscriber have sections that I honestly don't understand and use in an atmosphere, but even if they are implemented roughly, they can be processed asynchronously in a Publisher-Subscriber pattern, which is convenient.

The Listener interface of WebSocket had the same atmosphere, so it was easy to attach.

SlackListener.java

public class SlackListener extends SubmissionPublisher<String> implements Listener{
	private List<CharSequence> messageParts = new ArrayList<>();
	private CompletableFuture<?> accumulatedMessage = new CompletableFuture<>();
	
	@Override
	public CompletionStage<?> onText(WebSocket webSocket, CharSequence data, boolean last){
		messageParts.add(data);
		webSocket.request(1);
		if(last) {
			submit(String.join("", messageParts));
			messageParts = new ArrayList<>();
			accumulatedMessage.complete(null);
			CompletionStage<?> cf = accumulatedMessage;
			accumulatedMessage = new CompletableFuture<>();
			return cf;
		}
		return accumulatedMessage;
	}
	
	@Override
	public void onError(WebSocket webSocket, Throwable error) {
		error.printStackTrace();
	}
}

Message processing class with the desire to add various functions other than the Nikorepo throwing function in the future. The channel posted by the bot is also acquired here.

You can also get the channel by throwing it in conversations.list. This time, the Json of the message sent from slack normally included the channelId, so it was "Is it easier to pull from there?", So the bot posts to the Channel where the user entered a specific Command. Implemented as.

I decided to pass the email address and password required to log in to NicoNico via a Slack message. This one seemed to be easier when connecting a compound or another person who needs to log in.

ResponseProcessor.java

public class ResponseProcessor extends SubmissionPublisher<TransmissionMessage> 
		implements Processor<String, TransmissionMessage> {
	private Subscription subscription;
	private List<String> activeChannels = Collections.synchronizedList(new ArrayList<>());
	private NiconicoClient niconicoClient = new NiconicoClient(this::sendMessage);
	private Gson gson = new Gson();
	
	@Override
	public void onSubscribe(Subscription subscription) {
		this.subscription = subscription;
		subscription.request(1);
	}

	@Override
	public void onNext(String message) {
		System.out.println("onMessage : " + message);
		MessageType type = convertType(message);
		
		switch(type) {
		case MESSAGE:
			processMessage(message);
			break;
		case LOG:
			processLog(message);
			break;
		}
		subscription.request(1);
	}
	
	private void sendMessage(String message) {
		activeChannels.parallelStream()
				.map(channel -> new TalkMessage(message, channel))
				.forEach(this::submit);
	}
	
	private void processLog(String message) {
		LogMessage log = gson.fromJson(message, LogMessage.class);
		if(log.isOk())
			return;
		System.err.println(log.getError());
	}
	
	private void processMessage(String message) {
		ResponseMessage response = gson.fromJson(message, ResponseMessage.class);
		String text = response.getText();
		if(text != null && text.startsWith("command:")) {
			processCommand(text.split("(^command): *")[1], response.getChannel());
			return;
		}
	}
	
	private void processCommand(String command, String channel) {
		String[] array = command.split(" ");
		switch(array[0]) {
		case "activate":
			switch(array[1]) {
			case "bot":
				activeChannels.add(channel);
				sendMessage("I will post to this channel.");
				break;
			case "nicorepo":
				if(activeChannels.isEmpty())
					break;
				try {
					String email = array[2].split("\\|")[0].split(":")[1];
					niconicoClient.login(email, array[3]);
					niconicoClient.getMyRepoData();
				} catch (IOException | InterruptedException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}
				break;
			}
			break;
		}
	}
	
	private MessageType convertType(String message) {
		try {
			InputStream inputStream = new ByteArrayInputStream(message.getBytes("utf-8"));
			JsonReader reader = new JsonReader(new InputStreamReader(inputStream,"utf-8"));
			reader.beginObject();
			while(reader.hasNext()) {
				switch(reader.nextName()) {
				case "type":
					return MessageType.toMessageType(reader.nextString());
				case "ok":
					return MessageType.LOG;
				}
				break;
			}
			inputStream.close();
			reader.close();
		} catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		return MessageType.OTHER;
	}

	@Override
	public void onError(Throwable throwable) {
		// TODO Auto-generated method stub
		throwable.printStackTrace();
	}

	@Override
	public void onComplete() {
		// TODO Auto-generated method stub
		
	}
}

There is nothing special to say about the message sending class, so cut it.

Nico Nico Douga API

Basically the same feeling as Slack connect. Since the login process of Nico Nico Video is automatically redirected after the login process and the session information when logging in is erased, it is necessary to set Redirect.NEVER in the login process and acquire the cookie without redirecting. The endpoint of My Repo did not come out even if I googled it, so I opened My Page while opening the developer tools of the browser [Endpoint like that](http://www.nicovideo.jp/api/nicorepo/timeline/ I searched for my / all? client_app = pc_myrepo).

NiconicoClient.java

public class NiconicoClient {
	
	private LoginInfo loginInfo;
	private HttpCookie cookie;
	private Deque<NiconicoReport> reports = new ArrayDeque<>();
	private long latestReportId = 0;
	private Consumer<String> sendMessage;
	
	public NiconicoClient(Consumer<String> sendMessage) { 
		this.sendMessage = sendMessage;
	}
	
	public boolean login(String mail, String password) throws IOException, InterruptedException {
		loginInfo = new LoginInfo(mail, password);
		Map<String, String> headers = Map.of("Content-type", "application/x-www-form-urlencoded");
		Map<String, String> postMessages = Map.of("next_url", "",
				"mail", loginInfo.getMail(), 
				"password", loginInfo.getPassword());
		SimpleHttpClient httpClient = 
				new SimpleHttpClient(NiconicoAPI.LOGIN.getURIText(), Redirect.NEVER, headers, postMessages);
		HttpResponse<String> response = httpClient.sendPost();
		
		if(!httpClient.isPresentCookieHandler())
			return false;
		CookieStore store = ((CookieManager)httpClient.getCookieHandler()).getCookieStore();
		cookie = store.getCookies().stream()
				.filter(cookie -> cookie.getName().equals("user_session") &&
						!cookie.getValue().equals("deleted"))
				.findAny().orElse(null);
		loginInfo.setSession(cookie.toString());
		return loginInfo.isLogin();
	}
	
	public void getMyRepoData() throws IOException, InterruptedException {
		Map<String, String> headers = Map.of("Cookie", loginInfo.getCookie());
		SimpleHttpClient httpClient = 
				new SimpleHttpClient(NiconicoAPI.MYREPO.getURIText(), Redirect.ALWAYS, headers, Map.of());
		HttpResponse<String> response = httpClient.sendGet();

		MyRepoData data = new Gson().fromJson(response.body(), MyRepoData.class);
		List<NiconicoReport> list = data.getReports().stream()
				.filter(report -> report.getLongId() > latestReportId)
				.sorted(Comparator.comparingLong(NiconicoReport::getLongId))
				.collect(Collectors.toList());
		latestReportId = list.get(list.size()-1).getLongId();
		String watchUri = NiconicoAPI.WATCH_PAGE.getURIText();
		list.stream()
				.filter(report -> report.getVideo() != null)
				.map(report -> report.getVideo().getWatchId())
				.distinct()
				.forEach(id -> sendMessage.accept(watchUri + id));
		
	}
}

If you throw the URL after checking if it has already been posted with the one obtained from My Repo and throwing the URL, you can post the video URL of My Repo to Slack without duplication like the first image.

Where the implementation is not finished

I wrote most of the processing overnight from yesterday, but there are various omissions and implementation is not finished.

I've only got it once now, but of course I want to run it regularly and throw it automatically once it's enabled. The rest is basically premised on your own use, so exception handling is appropriate (if you throw a slightly strange Command, it will probably stop). In the future, I would like to be able to exchange commands in natural language.

For the time being, except for the last, when I feel like it, it ends with a choi choi, so I want to start working after the master's thesis.

Personal addiction point

Slack's API, basically, if you send an invalid message, an error will be returned properly (channel ID is different), but if the message to be sent does not conform to the format (even if it is not in Json format), even an error message It doesn't come back. I implemented up to connect, sent even an empty string and tried to see if the error was returned properly, and I got stuck for a while. To check if you can communicate properly with WebSocket, try with wscat or the minimum configuration.

I got the information with Niconico's API and output it with System.out.println for the time being to check if it was taken properly, but for some reason the text was not displayed. As a matter of fact, the content itself could be obtained normally, so if you throw all the shit long sentences in Eclipse, the display will be buggy. The console is bad.