We used to operate on-premise PukiWiki before I joined the company, but this time I moved to esa.io \ (⁰⊖⁰) / The main reason is that I got into esa, but I also had a lot of dissatisfaction with on-premise management, such as wanting to use Markdown rather than Wiki notation. This article is like a work log that quickly migrated the articles stored in PukiWiki to esa.io. I am ashamed to announce the migration program, so if you want to do the same thing, please refer to it.
https://esa.io/ Simply put, it's a Wiki-like information sharing tool. I like the fact that the hurdles for sharing information are very low, and the various "accessible to itchy places" features that make editing and managing articles super easy. When I actually used it in the free trial, it was a series of "Oh!".
--You can paste screenshots immediately with Ctrl + v! --When you enter edit mode with Ctrl + e, the place you saw opens immediately! --Isn't it possible to write a flowchart with Mermaid.js! ――Can you squash the Revision that you inadvertently increased?
--Since each article is saved as a txt file on the PukiWiki server, we will post it one by one using the esa API. --Since the original article is in wiki notation, convert it to Markdown and post it. ――When you finish posting all, update once to put an internal link (link from article to article) (links in esa can not be posted unless the article number is decided, so post once and post the article number Need to be confirmed) --The image will not be migrated, but it will be linked to the image of the source Wiki, so it can be displayed in esa.
--Uncommon (infrequently used) Wiki notation does not convert to Markdown --Attachments do not migrate (it was helpful because they are not attached much in the first place) ――I don't care about other details
The article txt file is stored in the "wiki" folder on the PukiWiki server. Copy this to the designated location. Then run the migration program below and you're good to go. The migration program was made in java.
Since it uses esa's web API, get an access token in advance. You can get it from "Applications" in the team settings, in the "Personal access tokens" part.
It's a disposable program, so I made it with great momentum. I haven't made any efforts to make it easier to read, so I think it's difficult to do when you want to play with it a little. .. .. If you have something like "I don't know what it means here", please ask in the comments. Please change the part marked with ★ in the program according to your environment before executing. The esa API is a specification that can only be called up to 75 times in 15 minutes, but if that limit is exceeded, I try to wait until I can call it again.
public class WikiMigrationToEsa
{
private static Client apiClient = ClientBuilder.newClient();
private static MutableList<Pair<Integer, Twin<String>>> documentList = Lists.mutable.empty();
//★ Specify the folder where the wiki article files are located
private static final String wikiFilePath = "C:\\Users\\hoge\\wiki";
//★esa.io access token
private static final String authorizationKey = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX";
//★ Team name
private static final String teamName = "hogehoge";
public static void main(String[] args)
{
//Esa wiki article files one by one.Post to io
try (final Stream<Path> pathStream = Files.walk(Paths.get(wikiFilePath)))
{
pathStream
.map(path -> path.toFile())
.filter(file -> !file.isDirectory())
.filter(file -> file.getName().toLowerCase().endsWith(".txt"))
.forEach(file -> postWikiFile(file.toPath()));
} catch (final IOException e) {
e.printStackTrace();
return;
}
//For posted articles, esa links in the wiki.Update to io format
patchInnerLink();
}
public static void postWikiFile(Path wikiItemFile)
{
EsaPostItemDto content = new EsaPostItemDto();
content.post = new EsaPostBodyDto();
StringBuilder bodyText = new StringBuilder();
//title
String wikiFileName = StringUtils.substringBefore(wikiItemFile.toFile().getName(), ".");
byte[] bytes;
try
{
bytes = Hex.decodeHex(wikiFileName);
} catch (DecoderException e)
{
e.printStackTrace();
return;
}
String itemName = "";
try
{
itemName = new String(bytes, "EUC-JP");
} catch (UnsupportedEncodingException e)
{
e.printStackTrace();
return;
}
//Addressed an issue where half-width slashes in titles would be recognized as categories
itemName = RegExUtils.replaceAll(itemName, "/", "/");
try (FileInputStream fileInputStream = new FileInputStream(wikiItemFile.toFile());
InputStreamReader inputStreamReader = new InputStreamReader(fileInputStream, "EUC-JP");
BufferedReader bufferedReader = new BufferedReader(inputStreamReader);
)
{
String lineTmp;
int lineNo = 0;
boolean isCodeBlock = false;
boolean isTable = false;
while ((lineTmp = bufferedReader.readLine()) != null) {
lineNo++;
byte[] lineBytes = lineTmp.getBytes();
String line = new String(lineBytes, "UTF-8");
if(lineNo == 1)
{
if(StringUtils.startsWithIgnoreCase(line, "#freeze"))
{
return;
}
}
if(StringUtils.startsWithIgnoreCase(line, "#ref") || StringUtils.startsWithIgnoreCase(line, "&ref"))
{
String fileName = StringUtils.substringBetween(line, "(", ")");
fileName = RegExUtils.replaceAll(fileName, "(^[^\\.]+\\.[^,]+),.+", "$1");
line = "![" + fileName + "](http://hoge/fuga/index.php?plugin=attach&refer="+URLEncoder.encode(itemName,"EUC-JP")+"&openfile="+URLEncoder.encode(fileName,"EUC-JP")+")";
}
if(!isCodeBlock && StringUtils.startsWith(line, " "))
{
isCodeBlock = true;
bodyText.append("```");
bodyText.append(System.getProperty("line.separator"));
}
if(isCodeBlock)
{
if(StringUtils.startsWith(line, " "))
{
bodyText.append(line);
bodyText.append(System.getProperty("line.separator"));
continue;
}
else
{
bodyText.append("```");
bodyText.append(System.getProperty("line.separator"));
isCodeBlock = false;
}
}
if(!isTable && StringUtils.startsWith(line, "|"))
{
isTable = true;
int columnCount = StringUtils.countMatches(line, "|");
bodyText.append(System.getProperty("line.separator"));
if(StringUtils.endsWith(line, "h"))
{
bodyText.append(StringUtils.substringBeforeLast(line, "h"));
bodyText.append(System.getProperty("line.separator"));
bodyText.append(StringUtils.repeat("|", "-----", columnCount));
bodyText.append(System.getProperty("line.separator"));
continue;
}
else
{
bodyText.append(StringUtils.repeat("|", "header", columnCount));
bodyText.append(System.getProperty("line.separator"));
bodyText.append(StringUtils.repeat("|", "-----", columnCount));
bodyText.append(System.getProperty("line.separator"));
bodyText.append(line);
bodyText.append(System.getProperty("line.separator"));
continue;
}
}
if(isTable)
{
if(StringUtils.startsWith(line, "|"))
{
bodyText.append(line);
bodyText.append(System.getProperty("line.separator"));
continue;
}
else
{
bodyText.append(System.getProperty("line.separator"));
isTable = false;
}
}
line = RegExUtils.replaceAll(line, "\\[#[0-9a-z]+\\]$", "");
line = RegExUtils.replaceAll(line, "^\\*\\*\\*", "### ");
line = RegExUtils.replaceAll(line, "^\\*\\*", "## ");
line = RegExUtils.replaceAll(line, "^\\*", "# ");
line = RegExUtils.replaceAll(line, "^---", " - ");
line = RegExUtils.replaceAll(line, "^--", " - ");
line = RegExUtils.replaceAll(line, "^-", "- ");
line = RegExUtils.replaceAll(line, "^\\+\\+\\+", " 1\\. ");
line = RegExUtils.replaceAll(line, "^\\+\\+", " 1\\. ");
line = RegExUtils.replaceAll(line, "^\\+", "1\\. ");
line = RegExUtils.replaceAll(line, "^\\+", "1\\. ");
line = RegExUtils.replaceAll(line, "''", "\\*\\*");
line = RegExUtils.replaceAll(line, "%%", "~~");
line = RegExUtils.replaceAll(line, "\\[\\[([^\\]^:^>]+)[:|>]([^:^/^\\]]+://[^\\]]+)\\]\\]","\\[$1\\]\\($2\\)");
bodyText.append(line);
bodyText.append(System.getProperty("line.separator"));
}
}
catch (IOException e)
{
e.printStackTrace();
return;
}
content.post.body_md = bodyText.toString();
content.post.name = itemName;
content.post.wip = false;
content.post.message = "Migrating from the wiki";
content.post.user = "esa_bot";
Response apiResponse = apiClient.target("https://api.esa.io/v1/teams/" + teamName + "/posts")
.request()
.header("Authorization", "Bearer " + authorizationKey)
.header("Content-Type", "application/json")
.post(Entity.entity(content, MediaType.APPLICATION_JSON));
System.out.println(apiResponse.getStatus());
JSONObject postResponseBody = new JSONObject(apiResponse.readEntity(String.class));
System.out.println("number : " + postResponseBody.getLong("number"));
documentList.add(
Tuples.pair(
postResponseBody.getInt("number")
,Tuples.twin(
itemName
, bodyText.toString()
)
)
);
waitAPILimit(apiResponse);
}
private static void patchInnerLink()
{
Pattern linkPattern = Pattern.compile("(\\[\\[[^\\]^>]+>[^\\]^:^/]+\\]\\]|\\[\\[[^\\]^>]+\\]\\])");
documentList.each(post -> {
Twin<String> postItem = post.getTwo();
Matcher postMatcher = linkPattern.matcher(postItem.getTwo());
StringBuffer sb = new StringBuffer();
while(postMatcher.find())
{
String linkString = postMatcher.group();
//linkString may or may not have an alias, so in the case of an alias>Take out after
if(StringUtils.contains(linkString, ">"))
{
linkString = StringUtils.substringAfter(linkString, ">");
}
final String linkItemName = RegExUtils.removeAll(linkString, "\\[|\\]");
Pair<Integer, Twin<String>> linkedItem = documentList.select(post2 -> StringUtils.equals(linkItemName, post2.getTwo().getOne())).getFirst();
if(linkedItem != null)
{
postMatcher.appendReplacement(sb, "[#" + linkedItem.getOne() + ": " + linkItemName + "](/posts/"+linkedItem.getOne()+")");
}
}
postMatcher.appendTail(sb);
EsaPostItemDto content = new EsaPostItemDto();
content.post = new EsaPostBodyDto();
content.post.body_md = sb.toString();
content.post.name = postItem.getOne();
content.post.wip = false;
content.post.message = "Internal link update";
content.post.updated_by = "esa_bot";
Response patchResponse = apiClient
.property(HttpUrlConnectorProvider.SET_METHOD_WORKAROUND, true)
.target("https://api.esa.io/v1/teams/" + teamName + "/posts/" + post.getOne())
.request()
.header("Authorization", "Bearer " + authorizationKey)
.header("Content-Type", "application/json")
.method("PATCH", Entity.entity(content, MediaType.APPLICATION_JSON))
;
System.out.println(patchResponse.getStatus());
waitAPILimit(patchResponse);
});
}
private static void waitAPILimit(Response apiResponse)
{
String remaining = apiResponse.getHeaderString("X-RateLimit-Remaining");
System.out.println("X-RateLimit-Remaining : " + remaining);
if(StringUtils.equals(remaining, "0"))
{
System.out.println("X-Rate-Limit-Reset : " + apiResponse.getHeaderString("X-RateLimit-Reset"));
OffsetDateTime reset = OffsetDateTime.ofInstant(Instant.ofEpochSecond(Long.valueOf(apiResponse.getHeaderString("X-RateLimit-Reset"))), ZoneId.systemDefault());
System.out.println("X-Rate-Limit-Reset : " + reset.format(DateTimeFormatter.ISO_DATE_TIME));
//Add 1 minute considering the possibility that the clock is not correct
reset = reset.plusMinutes(1);
while(reset.isAfter(OffsetDateTime.now(ZoneId.systemDefault())))
{
try
{
TimeUnit.MINUTES.sleep(1);
} catch (InterruptedException e)
{
e.printStackTrace();
System.exit(0);
}
}
}
}
}
EsaPostBodyDto.java
public class EsaPostBodyDto
{
public String name;
public String body_md;
public List<String> tags;
public String category;
public boolean wip;
public String message;
public String user;
public String updated_by;
}
EsaPostItemDto.java
public class EsaPostItemDto
{
public EsaPostBodyDto post;
}
pom.xml
<dependencies>
<dependency>
<groupId>commons-net</groupId>
<artifactId>commons-net</artifactId>
<version>3.3</version>
</dependency>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.2</version>
</dependency>
<!-- servlet-api -->
<!-- https://mvnrepository.com/artifact/javax.servlet/servlet-api -->
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
<version>2.5</version>
<scope>provided</scope>
</dependency>
<!-- Apache Commons Lang -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.8</version>
</dependency>
<!-- https://mvnrepository.com/artifact/commons-codec/commons-codec -->
<dependency>
<groupId>commons-codec</groupId>
<artifactId>commons-codec</artifactId>
<version>1.11</version>
</dependency>
<!-- jersey -->
<!-- https://mvnrepository.com/artifact/org.glassfish.jersey.media/jersey-media-json-jackson -->
<dependency>
<groupId>org.glassfish.jersey.media</groupId>
<artifactId>jersey-media-json-jackson</artifactId>
<version>2.25</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.containers</groupId>
<artifactId>jersey-container-servlet</artifactId>
<version>2.25</version>
</dependency>
<!-- eclipse collections -->
<dependency>
<groupId>org.eclipse.collections</groupId>
<artifactId>eclipse-collections-api</artifactId>
<version>8.0.0</version>
</dependency>
<dependency>
<groupId>org.eclipse.collections</groupId>
<artifactId>eclipse-collections</artifactId>
<version>8.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.json/json -->
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20180813</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-library</artifactId>
<version>1.3</version>
<scope>test</scope>
</dependency>
</dependencies>
Recommended Posts