Migrate from on-premise Pukiwiki to esa.io \ (⁰⊖⁰) /

Overview

We used to operate on-premise PukiWiki before I joined the company, but this time I moved to esa.io \ (⁰⊖⁰) / The main reason is that I got into esa, but I also had a lot of dissatisfaction with on-premise management, such as wanting to use Markdown rather than Wiki notation. This article is like a work log that quickly migrated the articles stored in PukiWiki to esa.io. I am ashamed to announce the migration program, so if you want to do the same thing, please refer to it.

What is esa.io

https://esa.io/ Simply put, it's a Wiki-like information sharing tool. I like the fact that the hurdles for sharing information are very low, and the various "accessible to itchy places" features that make editing and managing articles super easy. When I actually used it in the free trial, it was a series of "Oh!".

--You can paste screenshots immediately with Ctrl + v! --When you enter edit mode with Ctrl + e, the place you saw opens immediately! --Isn't it possible to write a flowchart with Mermaid.js! ――Can you squash the Revision that you inadvertently increased?

I will move \ (⁰⊖⁰) /

Such a policy

--Since each article is saved as a txt file on the PukiWiki server, we will post it one by one using the esa API. --Since the original article is in wiki notation, convert it to Markdown and post it. ――When you finish posting all, update once to put an internal link (link from article to article) (links in esa can not be posted unless the article number is decided, so post once and post the article number Need to be confirmed) --The image will not be migrated, but it will be linked to the image of the source Wiki, so it can be displayed in esa.

What I gave up

--Uncommon (infrequently used) Wiki notation does not convert to Markdown --Attachments do not migrate (it was helpful because they are not attached much in the first place) ――I don't care about other details

Migration procedure

The article txt file is stored in the "wiki" folder on the PukiWiki server. Copy this to the designated location. Then run the migration program below and you're good to go. The migration program was made in java.

Since it uses esa's web API, get an access token in advance. You can get it from "Applications" in the team settings, in the "Personal access tokens" part.

Migration program

It's a disposable program, so I made it with great momentum. I haven't made any efforts to make it easier to read, so I think it's difficult to do when you want to play with it a little. .. .. If you have something like "I don't know what it means here", please ask in the comments. Please change the part marked with ★ in the program according to your environment before executing. The esa API is a specification that can only be called up to 75 times in 15 minutes, but if that limit is exceeded, I try to wait until I can call it again.

Main program

public class WikiMigrationToEsa
{
	private static Client apiClient = ClientBuilder.newClient();
	
	private static MutableList<Pair<Integer, Twin<String>>> documentList = Lists.mutable.empty();
	
	//★ Specify the folder where the wiki article files are located
	private static final String wikiFilePath = "C:\\Users\\hoge\\wiki";
	//★esa.io access token
	private static final String authorizationKey = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX";
	//★ Team name
	private static final String teamName = "hogehoge";
	
	public static void main(String[] args)
	{
		//Esa wiki article files one by one.Post to io
		try (final Stream<Path> pathStream = Files.walk(Paths.get(wikiFilePath))) 
		{
			pathStream
				.map(path -> path.toFile())
				.filter(file -> !file.isDirectory())
				.filter(file -> file.getName().toLowerCase().endsWith(".txt"))
				.forEach(file -> postWikiFile(file.toPath()));
		} catch (final IOException e) {
			e.printStackTrace();
			return;
		}
		//For posted articles, esa links in the wiki.Update to io format
		patchInnerLink();
	}
	
	public static void postWikiFile(Path wikiItemFile)
	{
		EsaPostItemDto content = new EsaPostItemDto();
		content.post = new EsaPostBodyDto();
		StringBuilder bodyText = new StringBuilder();
		//title
		String wikiFileName = StringUtils.substringBefore(wikiItemFile.toFile().getName(), ".");
		byte[] bytes;
		try
		{
			bytes = Hex.decodeHex(wikiFileName);
		} catch (DecoderException e)
		{
			e.printStackTrace();
			return;
		}
		String itemName = "";
		try
		{
			itemName = new String(bytes, "EUC-JP");
		} catch (UnsupportedEncodingException e)
		{
			e.printStackTrace();
			return;
		}
		//Addressed an issue where half-width slashes in titles would be recognized as categories
		itemName = RegExUtils.replaceAll(itemName, "/", "&#47;");
		
		try (FileInputStream fileInputStream = new FileInputStream(wikiItemFile.toFile());
				InputStreamReader inputStreamReader = new InputStreamReader(fileInputStream, "EUC-JP");
				BufferedReader bufferedReader = new BufferedReader(inputStreamReader);
			)
		{
			String lineTmp;
			int lineNo = 0;
			boolean isCodeBlock = false;
			boolean isTable = false;
			while ((lineTmp = bufferedReader.readLine()) != null) {
				lineNo++;
				byte[] lineBytes = lineTmp.getBytes();
				String line = new String(lineBytes, "UTF-8");
				if(lineNo == 1)
				{
					if(StringUtils.startsWithIgnoreCase(line, "#freeze"))
					{
						return;
					}
				}
				if(StringUtils.startsWithIgnoreCase(line, "#ref") || StringUtils.startsWithIgnoreCase(line, "&ref"))
				{
					String fileName = StringUtils.substringBetween(line, "(", ")");
					fileName = RegExUtils.replaceAll(fileName, "(^[^\\.]+\\.[^,]+),.+", "$1");
					line = "![" + fileName + "](http://hoge/fuga/index.php?plugin=attach&refer="+URLEncoder.encode(itemName,"EUC-JP")+"&openfile="+URLEncoder.encode(fileName,"EUC-JP")+")";
				}
				if(!isCodeBlock && StringUtils.startsWith(line, " "))
				{
					isCodeBlock = true;
					bodyText.append("```");
					bodyText.append(System.getProperty("line.separator"));
				}
				if(isCodeBlock)
				{
					if(StringUtils.startsWith(line, " "))
					{
						bodyText.append(line);
						bodyText.append(System.getProperty("line.separator"));
						continue;
					}
					else
					{
						bodyText.append("```");
						bodyText.append(System.getProperty("line.separator"));
						isCodeBlock = false;
					}
				}
				if(!isTable && StringUtils.startsWith(line, "|"))
				{
					isTable = true;
					int columnCount = StringUtils.countMatches(line, "|");
					bodyText.append(System.getProperty("line.separator"));
					if(StringUtils.endsWith(line, "h"))
					{
						bodyText.append(StringUtils.substringBeforeLast(line, "h"));
						bodyText.append(System.getProperty("line.separator"));
						bodyText.append(StringUtils.repeat("|", "-----", columnCount));
						bodyText.append(System.getProperty("line.separator"));
						continue;
					}
					else
					{
						bodyText.append(StringUtils.repeat("|", "header", columnCount));
						bodyText.append(System.getProperty("line.separator"));
						bodyText.append(StringUtils.repeat("|", "-----", columnCount));
						bodyText.append(System.getProperty("line.separator"));
						bodyText.append(line);
						bodyText.append(System.getProperty("line.separator"));
						continue;
					}
				}
				if(isTable)
				{
					if(StringUtils.startsWith(line, "|"))
					{
						bodyText.append(line);
						bodyText.append(System.getProperty("line.separator"));
						continue;
					}
					else
					{
						bodyText.append(System.getProperty("line.separator"));
						isTable = false;
					}
				}
				line = RegExUtils.replaceAll(line, "\\[#[0-9a-z]+\\]$", "");
				line = RegExUtils.replaceAll(line, "^\\*\\*\\*", "### ");
				line = RegExUtils.replaceAll(line, "^\\*\\*", "## ");
				line = RegExUtils.replaceAll(line, "^\\*", "# ");
				line = RegExUtils.replaceAll(line, "^---", "        - ");
				line = RegExUtils.replaceAll(line, "^--", "    - ");
				line = RegExUtils.replaceAll(line, "^-", "- ");
				line = RegExUtils.replaceAll(line, "^\\+\\+\\+", "        1\\. ");
				line = RegExUtils.replaceAll(line, "^\\+\\+", "    1\\. ");
				line = RegExUtils.replaceAll(line, "^\\+", "1\\. ");
				line = RegExUtils.replaceAll(line, "^\\+", "1\\. ");
				line = RegExUtils.replaceAll(line, "''", "\\*\\*");
				line = RegExUtils.replaceAll(line, "%%", "~~");
				line = RegExUtils.replaceAll(line, "\\[\\[([^\\]^:^>]+)[:|>]([^:^/^\\]]+://[^\\]]+)\\]\\]","\\[$1\\]\\($2\\)");
				bodyText.append(line);
				bodyText.append(System.getProperty("line.separator"));
			}
		}
		catch (IOException e)
		{
			e.printStackTrace();
			return;
		}
		
		content.post.body_md = bodyText.toString();
		content.post.name = itemName;
		content.post.wip = false;
		content.post.message = "Migrating from the wiki";
		content.post.user = "esa_bot";
		
		Response apiResponse = apiClient.target("https://api.esa.io/v1/teams/" + teamName + "/posts")
			.request()
			.header("Authorization", "Bearer " + authorizationKey)
			.header("Content-Type", "application/json")
			.post(Entity.entity(content, MediaType.APPLICATION_JSON));
		System.out.println(apiResponse.getStatus());
		JSONObject postResponseBody = new JSONObject(apiResponse.readEntity(String.class));
		System.out.println("number : " + postResponseBody.getLong("number"));
		documentList.add(
				Tuples.pair(
						postResponseBody.getInt("number")
						,Tuples.twin(
								itemName
								, bodyText.toString()
								)
						)
					);
		waitAPILimit(apiResponse);
	}
	
	
	private static void patchInnerLink()
	{
		Pattern linkPattern = Pattern.compile("(\\[\\[[^\\]^>]+>[^\\]^:^/]+\\]\\]|\\[\\[[^\\]^>]+\\]\\])");
		documentList.each(post -> {
			Twin<String> postItem = post.getTwo();
			Matcher postMatcher = linkPattern.matcher(postItem.getTwo());
			StringBuffer sb = new StringBuffer();
			while(postMatcher.find())
			{
				String linkString = postMatcher.group();
				//linkString may or may not have an alias, so in the case of an alias>Take out after
				if(StringUtils.contains(linkString, ">"))
				{
					linkString = StringUtils.substringAfter(linkString, ">");
				}
				final String linkItemName = RegExUtils.removeAll(linkString, "\\[|\\]");
				Pair<Integer, Twin<String>> linkedItem = documentList.select(post2 -> StringUtils.equals(linkItemName, post2.getTwo().getOne())).getFirst();
				if(linkedItem != null)
				{
					postMatcher.appendReplacement(sb, "[#" + linkedItem.getOne() + ": " + linkItemName + "](/posts/"+linkedItem.getOne()+")");
				}
			}
			postMatcher.appendTail(sb);
			
			EsaPostItemDto content = new EsaPostItemDto();
			content.post = new EsaPostBodyDto();
			content.post.body_md = sb.toString();
			content.post.name = postItem.getOne();
			content.post.wip = false;
			content.post.message = "Internal link update";
			content.post.updated_by = "esa_bot";
			Response patchResponse = apiClient
					.property(HttpUrlConnectorProvider.SET_METHOD_WORKAROUND, true)
					.target("https://api.esa.io/v1/teams/" + teamName + "/posts/" + post.getOne())
					.request()
					.header("Authorization", "Bearer " + authorizationKey)
					.header("Content-Type", "application/json")
					.method("PATCH", Entity.entity(content, MediaType.APPLICATION_JSON))
					;
			System.out.println(patchResponse.getStatus());
			waitAPILimit(patchResponse);
		});
	}
	
	private static void waitAPILimit(Response apiResponse)
	{
		String remaining = apiResponse.getHeaderString("X-RateLimit-Remaining");
		System.out.println("X-RateLimit-Remaining : " + remaining);
		if(StringUtils.equals(remaining, "0"))
		{
			System.out.println("X-Rate-Limit-Reset : " + apiResponse.getHeaderString("X-RateLimit-Reset"));
			OffsetDateTime reset = OffsetDateTime.ofInstant(Instant.ofEpochSecond(Long.valueOf(apiResponse.getHeaderString("X-RateLimit-Reset"))), ZoneId.systemDefault());
			System.out.println("X-Rate-Limit-Reset : " + reset.format(DateTimeFormatter.ISO_DATE_TIME));
			//Add 1 minute considering the possibility that the clock is not correct
			reset = reset.plusMinutes(1);
			while(reset.isAfter(OffsetDateTime.now(ZoneId.systemDefault())))
			{
				try
				{
					TimeUnit.MINUTES.sleep(1);
				} catch (InterruptedException e)
				{
					e.printStackTrace();
					System.exit(0);
				}
			}
		}
	}
}

DTO for expressing JSON of esa API

EsaPostBodyDto.java


public class EsaPostBodyDto
{
	public String name;
	public String body_md;
	public List<String> tags;
	public String category;
	public boolean wip;
	public String message;
	public String user;
	public String updated_by;
}

EsaPostItemDto.java


public class EsaPostItemDto
{
	public EsaPostBodyDto post;
}

Library etc.

pom.xml


	<dependencies>
		<dependency>
			<groupId>commons-net</groupId>
			<artifactId>commons-net</artifactId>
			<version>3.3</version>
		</dependency>
		<dependency>
			<groupId>commons-logging</groupId>
			<artifactId>commons-logging</artifactId>
			<version>1.2</version>
		</dependency>
		<!-- servlet-api -->
		<!-- https://mvnrepository.com/artifact/javax.servlet/servlet-api -->
		<dependency>
			<groupId>javax.servlet</groupId>
			<artifactId>servlet-api</artifactId>
			<version>2.5</version>
			<scope>provided</scope>
		</dependency>
		<!-- Apache Commons Lang -->
		<dependency>
			<groupId>org.apache.commons</groupId>
			<artifactId>commons-lang3</artifactId>
			<version>3.8</version>
		</dependency>
		<!-- https://mvnrepository.com/artifact/commons-codec/commons-codec -->
		<dependency>
			<groupId>commons-codec</groupId>
			<artifactId>commons-codec</artifactId>
			<version>1.11</version>
		</dependency>
		<!-- jersey -->
		<!-- https://mvnrepository.com/artifact/org.glassfish.jersey.media/jersey-media-json-jackson -->
		<dependency>
			<groupId>org.glassfish.jersey.media</groupId>
			<artifactId>jersey-media-json-jackson</artifactId>
			<version>2.25</version>
		</dependency>
		<dependency>
			<groupId>org.glassfish.jersey.containers</groupId>
			<artifactId>jersey-container-servlet</artifactId>
			<version>2.25</version>
		</dependency>
		<!-- eclipse collections -->
		<dependency>
			<groupId>org.eclipse.collections</groupId>
			<artifactId>eclipse-collections-api</artifactId>
			<version>8.0.0</version>
		</dependency>
		<dependency>
			<groupId>org.eclipse.collections</groupId>
			<artifactId>eclipse-collections</artifactId>
			<version>8.0.0</version>
		</dependency>
		<!-- https://mvnrepository.com/artifact/org.json/json -->
		<dependency>
			<groupId>org.json</groupId>
			<artifactId>json</artifactId>
			<version>20180813</version>
		</dependency>
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>4.11</version>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>org.hamcrest</groupId>
			<artifactId>hamcrest-library</artifactId>
			<version>1.3</version>
			<scope>test</scope>
		</dependency>
	</dependencies>

reference

esa API Reference

Recommended Posts

Migrate from on-premise Pukiwiki to esa.io \ (⁰⊖⁰) /
Migrate from JUnit 4 to JUnit 5
How to migrate from JUnit4 to JUnit5
Migrate Ubuntu 18.04 LTS from HDD to smaller SSD
Migrate from Java to Server Side Kotlin + Spring-boot
Migrate from Transport Client to Rest High Level Client
[Swift, ARKit] Migrate from deprecated hitTest to raycastQuery
Changes from Java 8 to Java 11
Sum from Java_1 to 100
From Java to Ruby !!
Moved from iBATIS to MyBatis3
Try Spring Boot from 0 to 100.
Migration from Cobol to JAVA
Switch from slim3-gen to slim3-gen-jsr269
Moving from AWS to PaizaCloud
New features from Java7 to Java8
Migrate GitBucket database to postgreSQL
Connect from Java to PostgreSQL
Convert from ○ months to ○ years ○ months
Rewriting from applet to application
Change from SQLite3 to PostgreSQL
From Ineffective Java to Effective Java