[PYTHON] I made a tool to compile Hy natively

Introduction

In Previous post, I introduced a tool for executing and archiving source code. Introducing this time is a tool that natively compiles Hy to create executable files and shared libraries. You can also convert it to Python or C language. It is registered in PyPI under the name HyCC. However, there is still the possibility of ** unexpected bugs **, so please use it at your own risk **. By the way, ** all are implemented in Hy **.

--Hy official documentation -Hy's Github repository -Qiita: Squid Lisp written in Python: Hy -Qiita: I made a package to create an executable file from Hy source code

How to use

Please install from pip. ** However, the operation in the Windows environment has not been confirmed. ** **

$ pip install hycc

~~ In addition, if you use the latest version of PyPI, Hy (v0.12.1), some code may cause an error due to a bug in Hy, so please install the latest version from Github. ~~ ** 2017/07/01 postscript: This work is no longer necessary due to the update of Hy and HyCC. **

Now that the preparation is complete, let's take the following code as an example.

hello.hy

(defn hello []
  (print "hello!"))

(defmain [&rest args]
  (hello))

Create an executable file

$ hycc hello.hy

This will create an executable binary called hello in the current directory.

$ ./hello
hello!

Create a shared library

Build with the --shared option.

$ hycc hello.hy --shared

This will create a shared object called hello.so in the current directory. Python has the ability to import shared objects as modules, but Hy is of course the same. Therefore, this hello.so can be used as follows.

Hy to hello.import so

(import hello)
(hello.hello)
; >hello!

Hello from python.import so

import hello
hello.hello()
# >hello!

Natively compiled modules and executables are ** about 2-8 times faster **. So I think it's more useful than byte-compiling to a .pyc file using hyc. And of course ** faster than Python **. In addition, Cython is used internally, but the image of speed is as follows.

** C <Cython (with type specification) << Cython (without type specification) <HyCC <Python <Hy **

It is generally ** upward compatible with hyc **, but there is one caveat **. Since it is related to the internal mechanism, I will explain it while explaining the mechanism from the following.

How it works

Roughly speaking, Hy's code is converted to Python, converted to C via Cython, and compiled. However, as mentioned in Last post, Python code generated using hy2py that comes standard with Hy. ** Doesn't work as it is **. Therefore, you can't just hy2py and cythonize.

Convert Hy to Python

In the first place ** Why doesn't the code that hy2py spits work **? Because Python uses an invalid identifier. The invalid identifiers in Python are:

1. • Identifier starting with a number 0-9 *
2. Identifiers containing characters other than * _, a-z, A-Z, 0-9 *

Take the following code as an example.

sample.hy

(reduce + [1 2 3])
; > 6

Converting this with hy2py gives the following.

Code that hy2py spits

from hy.core.language import reduce
from hy.core.shadow import +
# +Is an invalid identifier
reduce(+, [1, 2, 3])

Here, + is the above 2. Because it corresponds to, it is useless. Simply replacing this with another valid name, such as ʻadd`, won't work.

Of the code that hy2py spits+Replace with add

from hy.core.language import reduce
from hy.core.shadow import add
# ImportError
reduce(add, [1, 2, 3])

Obviously, the hy.core.shadow module does not have the name ʻadd defined, so it will be ʻImportError. HyCC first generates the following Python code from Hy source to solve this problem.

HyCC way

import hy.core.language as _
reduce = _.getattr("reduce")
import hy.core.shadow as _
+ = _.getattr("+")
reduce(+, [1, 2, 3])

This code is equivalent to the code that hy2py spits, but you can replace+with any name. HyCC avoids errors by properly combining access at the AST level and access at the source code level. Similarly, the member access of the object is rewritten with getattr and setattr.

Precautions when using HyCC (** important **)

About the precautions when using HyCC mentioned earlier. As explained so far, HyCC replaces invalid identifiers with valid ones with some ingenuity. At this time, only one side effect or problem will occur.

HyCC bad guy

(def hoge/fuga 0)
(print (get (globals) "hoge/fuga"))
; > 0

The above code is converted by HyCC to the following Python code.

Code generated by HyCC

from __future__ import print_function
import hy
hogex2Ffuga = 0L
print(globals()[u'hoge/fuga'])

You can ignore the addition of some imports. In the original Hy code, hoge / fuga was an invalid name in Python. Therefore, it is replaced with hogex2Ffuga in the code generated by HyCC.

Executing this code will result in the following error.

  File "test.py", line 4, in <module>
    print(globals()[u'hoge/fuga'])
KeyError: u'hoge/fuga'

Did you understand? The negative effect of replacing invalid identifiers with valid ones is that the globals, locals, and ʻinspect` modules may not work properly in some cases.

We are considering some countermeasures for this problem, but ** Hy itself already has a similar problem ** in the first place. In Hy, hoge! Is replaced with hoge_bang andhoge?Is replaced with ʻis_hoge` at the stage of parsing. Therefore, the following code does not work properly.

Hy and bad guy

(def hoge! 0)
(print (get (globals) "hoge!"))
; > KeyError!

Therefore, the use of globals etc. can be said to be ** cautions when using HyCC rather than cautions when using HyCC **.

*** 2017/06/04 postscript *** ** Supported by update! ** Specifically, Key error is avoided by wrapping globals and locals in a mystery class like dict. As usual, the ʻinspect module does not work properly, but since ** Cython itself does not support ʻinspect **, there is nothing I can do about it. For details, please refer to this commit.

Bonus (convert to Python that works properly)

HyCC also has a function to convert from Hy to Python like hy2py.

$ hycc hello.hy --python

This will output hello.py to the current directory. If you pay attention to the above notes, it works well unlike the code output by ** hy2py **. Surprisingly, this feature may be in higher demand. Similarly, you can convert it to C language with the --clang option, but it is delicate whether it can be used.

Known bug (2017/06/13 postscript)

Shared library from import fails

As I wrote in github issues, is it a Python specification? When getting a submodule with gettatr for a module object It seems to be ʻAttribute Error` if the submodule contains a shared library. I'm thinking about how to deal with it.

in conclusion

Tool to compile Hy natively was introduced. All development is done on github, so feel free to [pull request](https://github.com/koji-kojiro/hylang-hycc/ You can throw pulls) or issues. Of course, comments here are also welcome. Hy is still a developing fucking minor language, but I hope that it will become more sophisticated as the number of users increases and discussions become more active. Thank you for reading this far.