Comint Intercept: Combine shell/eshell/term mode

This is another Emacs package I developed recently to make my life better.

https://github.com/hying-caritas/comint-intercept

Comint Intercept

Why

Intercept the input in comint-mode. This can be used to run eshell command, or run some command in a terminal buffer from the command line in shell buffer. That is, this is to combine the best part of shell, eshell and term mode. For example, you can run eshell `grep’ in the shell buffer, or when you run `top’ in the shell buffer, a terminal buffer is poped to run it. SSH is supported to run command remotely.

Install

The preferred install method is through package manager: package.el. comint-intercept is available in MELPA package repository.

Or you can download the comint-intercept.el, add its path into Emacs load path, and load the file. For example, add

(add-to-list 'load-path "<directory of comint-intercept.el>")
(require 'comint-intercept)

in your Emacs init file.

Usage

A minor mode named comint-intercept-mode is defined. To toggle it in a shell buffer or some other comint mode buffer, you can run command comint-intercept-mode. Or you can enable comint-intercept-mode automatically when you create a new shell mode buffer via,

(add-hook 'shell-mode-hook 'comint-intercept-mode)

Then you can run eshell command via

$ e <eshell command>

Or run some eshell command directly via customizing comint-intercept-eshell-commands.

To run command in term mode, you can use

$ t <command> <args>...

Or run commands in term mode directly via customizing comint-intercept-term-commands.

If you want something more general, you can customize comint-intercept-pattern-actions, which is an alist maps input pattern (regexp) to action to take (function). The input string will be fed to the action function. Then you can process the input string via eshell-command or comint-intercept-term-command provided by the package.

project-shells: Manage shell buffers for each project

This is my first Emacs package distributed via Emacs package manager. It is available in MELPA package repository. The github project page is here.

Why bother?

This is to manage multiple shell (or terminal, eshell) buffers for each project. For example, to develop for Linux kernel, I usually use one shell buffer to configure and build kernel, one shell buffer to run some git command not supported by magit, one shell buffer to run qemu for built kernel, one shell buffer to ssh into guest system to test. Different set of commands is used by the shell in each buffer, so each shell should have different command history configuration, and for some shell, I may need different setup. And I have several projects to work on. In addition to project specific shell buffers, I want some global shell buffers, so that I can use them whichever project I am working on. Project shells is an Emacs package to let my life easier via helping me to manage all these shell/terminal/eshell buffers.

Install

The preferred install method is through package manager: package.el. project-shells is available in MELPA package repository.

Or you can download the project-shells.el, add its path into emacs load path, and load the file. For example, add

(add-to-list 'load-path "<directory of project-shells.el>")
(require 'project-shells)

in your Emacs init file.

Usage

To enable the project-shells minor mode globally, global-project-shells-mode can be used, or it can enabled in a buffer with project-shells-mode. The default prefix key is “C-c s”, which can be changed via customizing project-shells-keymap-prefix.

The package could also be setup with enabling minor mode via call project-shells-setup, the parameter is the keymap to add project shells key binding. For example, to use project shells with projectile you can put the following forms in your init file,

(project-shells-setup projectile-mode-map)

You can add project-shells key binding in other keymap too, or you can create your own keymap.

After enabling the project-shells minor mode or binding the key, you can create or switch to the shell buffers via,

<prefix key> <shell key>

The default <shell key>s are “1”, “2”, “3”, “4”, “5”, “6”, “7”, “8”, “9”, “0”, “-“, “=”, so you can have 12 shell/terminal/eshell buffers for each project. You can change the number of buffers and keys to activate them via customizing project-shells-keys. By default “1” – “0” will create shell buffers, “-” will create terminal buffers, “=” will create eshell buffers.

To create or switch to the global shells, you need to use position parameter for the <shell key>. For example,

<prefix key> C-u <shell key>

Use customize-group project-shell to change the configuration.

Work with project management package

You need a project management package enabled, so that project shells knows what the current project is and what the project root directory is. The default configuration works with projectile. To work with other project management package, you need to customize project-shells-project-name-func and project-shells-project-root-func.

Shell configuration

In the default configuration, each shell program and eshell runs by project-shells will has own history file. The default history file name is “~/.sessions/<project name>/<key>/.shell_history”. The default configuration works for bash, to support other shell, the project-shells-histfile-env may need to be customized.

If some special shell configuration is needed, the shell initialize file could be added into the session directory, the default initialize file name is “~/.sessions/<project name>/<key>/.shellrc”. The default configuration works for bash, to support other shell, the project-shells-default-init-file may need to be customized.

Scheme Implementation Learning Materials

This is an email from Racket-devel mailing by Jens Axel Søgaard to answer Austin T’s question with email subject “Looking for explanation on Racket’s C code & core systems implementation (hello from Shaka Scheme)”.  I found this is very useful, so copy it here.  Thanks Jens!

> I would like to know:
> – What should I read?

Since Scheme is an old language the literature on Scheme compilers is surprisingly rich.

Some of the classics (can’t recommend these enough):
“Lisp in Small Pieces” by Christian Queinnec
This book is a delight to read – and even includes a chapter on compiling to C.

“Three Implementation Models for Scheme” by R. Kent Dybvig
Dybvig’s thesis. Very clear explanations. Explains how to implement  using a stack of frames (like C).

> – Expressing Scheme types in C data structures

Take a look at how Larceny represents Scheme values. The approach is fairly typical.
http://www.larcenists.org/Documentation/Documentation0.97/LarcenyNotes/note2-repr.html

> Garbage collection vs. reference counting

Reference counting has problems with cyclic data that becomes dead.  A simple stop-copy garbage collector would be a good place to start.

“Programming Languages: Application and Interpretation” (PLAI) by Shriram Krishnamurthi has a chapter on memory management:
http://cs.brown.edu/courses/cs173/2012/book/Memory_Management.html
(There is a nice language for implementation garbage collectors available.  See http://blog.brownplt.org/2013/02/19/teaching-gc.html .  I can’t find the linked exercises – have they moved?)

> Parsing input from characters & type of algorithm used

The Scheme Compiler Workshop has a section on writing a scanner and parser:
http://www.cs.indiana.edu/eip/compile/scanparse.html


Jens Axel Søgaard

Learn Lisp? Use Lisp?

I am a lisp/scheme hobbyist. So I learn something about lisp, I have read some manuals of Common lisp, Emacs lisp, r5rs, r6rs, racket and part of guile. But as any other IT technology, if I don’t touch them for some time, I will forget almost everything about them soon. And an important way to learn something is to use it. So I try to use Lisp as much as possible.

  • Emacs is my editor, email client (gnus), TODO/task manager (org mode, partly) and many more. And I try to customize Emacs to make it fit my requirement.
    • Emacs runs in daemon mode + a shell/terminal manager can replace screen or tmux. I write a shell/terminal manager by myself: project-shells, it can create/switch multiple shell/terminal buffers for each project.
      • Emacs runs in daemon mode could be attached to a ssh session, so I can resume my work remotely when needed. This can replace the session attachment/detachment functionality of screen/tmux.
      • The shell/terminal manager could replace the terminal management functionality of screen/tmux. But the terminal mode of Emacs is not good enough to run some program, sometimes I still have to use screen for that. But fortunately, at most times, shell mode is sufficient for me.
  • I use the racket pict module to draw figure. That is a good tool, and I like to draw figure in program way than WYSIWYG mode, because I feel I can control the figure at will and it is easier to change the figure in some situation. I write some code to enhance the functionality of racket pict module, the code is put here.
  • I have used racket slideshow module to write some slides. Because I need to share the slides with others, I may not write all my slides with it, but I think it is a good tool and I will try to use it as much as possible.
  • I have written a simple IMAP mail filter with racket net/imap module to filter my email for one of my email service used. I plan to enhance net/imap and use my imap filter for my other email services and publish the code at my github.
  • I use Guix distribution on one of my main notebook, that is a distribution written in guile. And uses many guile software, such as shepherd, mcron, etc. I read some code of Guix and contribute to Guix project to make it works better for me. I plan to use Guix as my main distribution in the future. But there is still some feature missing in Guix, so I only use it on one of my main notebook now.
  • I use Stumpwm on Guix as my window manager. It is implemented with Common Lisp. So it is highly customizable. I have fixed several bugs in stump and its modules. I may customize more in the future. But I will reduce my usage of Common Lisp in the future and use more Scheme in the future. So I may try guilewm or something else in the future.
  • I have tries to write shell scripts with Common Lisp and implement a simple shell in Common Lisp: hysh. But I found Common Lisp is not good for this task. The typical usage model of Common Lisp is to run functions in REPL, not run function in shell. I plan to implement a simple shell in racket firstly with process*/ports and place*, this may be not perfect, but should be easy to implement.

Meta programming and lisp

Lisp is recognized to be one of the best programming language on meta programming. I think that mainly comes from its feature of program as data, that is, lisp program itself is most commonly used lisp data structure, such as list, symbol, etc. Based on that, lisp has a very powerful macro system, which is kind of static, and finally, the program can be generated and compiled or interpreted easily at run time, that is even more dynamic.

Is this kind of meta programming possible in other language? C programming language has macro too. The standard C macro is too limited in feature, but we can implement a much more powerful macro system for C too. And we can generate C program text and invoke the C compiler at run time too. That way, C can achieve similar meta programming capability.

Even it is possible to do meta programming in other languages, it may be more convenient to do that in lisp. Because lisp program is structural data, such as list, symbol, etc, while C program text is just un-structural string. It is much easier to program the data structure instead of the string except some relative simple operations. But if a standardized abstract syntax tree is defined for C, and a compiler backend is made available at run time to compile the abstract syntax tree, we can work with abstract syntax tree when do meta programming in C too.

But I still guess lisp could be better than other languages on meta programming, although I haven’t very rich experience on meta programming. The possible reasons could be:

  • The syntax of lisp is easier to program than abstract syntax tree of other programming languages (such as C). Because it is common to parse and generate program in lisp.
  • There is no standardized syntax tree defined for many other languages, even if it is possible. Maybe the abstract syntax tree is too complex for people to program in.
  • Even if an abstract syntax tree is defined for some language, people may feel like they are programming two different languages, one is the language itself, the other is its abstract syntax tree.

Inline Caching

Inline caching is an optimizing technology for “dynamic language” compiling, more details about it can be found in https://en.wikipedia.org/wiki/Inline_caching. I am working on cl-gobject-introspection (https://github.com/andy128k/cl-gobject-introspection), which is a common-lisp binding of gobject introspection. Inline caching should be helpful for its performance. So I do some experiment in implementing inline caching in common lisp. Given a sample class/object implementation as follow,

(defclass some-obj ()
  ((some-class :initarg :some-class :reader some-class-of)
   (fields :initarg :fields :type simple-vector)))

(defclass some-class ()
  ((fields-desc :initarg :fields-desc :type sequence)
   (methods-hash :initarg :methods-hash :type hash-table)))

(declaim (ftype (function (some-class symbol) function) some-class-get-method))
(defun some-class-get-method (some-class method)
  (declare (type symbol method)
	   (type some-class some-class))
  (with-slots (methods-hash)
      some-class
    (gethash method methods-hash)))

(declaim (ftype (function (some-class) some-obj) some-class-make-instance))
(defun some-class-make-instance (some-class)
  (with-slots (fields-desc)
      some-class
    (let ((inst (make-instance 'some-obj :some-class some-class
			       :fields (make-array (list (length fields-desc))))))
      (funcall (some-class-get-method some-class 'init) inst)
      inst)))

(defun send (obj method &rest args)
  (declare (type some-obj obj)
	   (type symbol method))
  (let* ((cls (some-class-of obj))
	 (func (some-class-get-method cls method)))
    (apply func obj args)))

And a simple test program as follow,

(defun make-trivial-some-class ()
  (let ((methods-hash (make-hash-table :test #'eq)))
    (setf (gethash 'init methods-hash)
	  (lambda (some-obj)
	    (declare (type some-obj some-obj))
	    (with-slots (fields)
		some-obj
	      (setf (aref fields 0) 0))))
    (setf (gethash 'inc methods-hash)
	  (lambda (some-obj val)
	    (declare (type some-obj some-obj))
	    (with-slots (fields)
		some-obj
	      (incf (aref fields 0) val))))
    (setf (gethash 'dec methods-hash)
	  (lambda (some-obj val)
	    (declare (type some-obj some-obj))
	    (with-slots (fields)
		some-obj
	      (decf (aref fields 0) val))))
    (make-instance 'some-class :fields-desc '(value)
		   :methods-hash methods-hash)))

(declaim (type fixnum +stress-loops+))
(defconstant +stress-loops+ 1000000)

(defun stress1 ()
  (let* ((c (make-trivial-some-class))
	 (o (some-class-make-instance c)))
    (iter (for i :from 0 :to +stress-loops+)
	  (declare (type fixnum i))
	  (send o 'inc 1)
	  (send o 'dec 1))))

For each method invoking (message sending), a hash table is searched to find the real function to call. Because the class of the object is only known at execution time (this is why it is called dynamic), the real function to call can be determined only at execution time too. Inline caching can be used to optimize for this situation. Because at most call sites, most objects has the same class. An inline optimized version is as follow,

(defmacro send-with-cache (class-var func-var obj method &rest args)
  (with-gensyms (robj rsome-class)
    `(let* ((,robj ,obj)
	    (,rsome-class (some-class-of ,robj)))
       (declare (type function ,func-var))
       (unless (eq ,class-var ,rsome-class)
	 (setf ,func-var (some-class-get-method ,rsome-class ,method)
	       ,class-var ,rsome-class))
       (funcall ,func-var ,robj ,@args))))

(defmacro cache-defun (name params &body body)
  (let ((cache-vars nil))
    (flet ((transform (form)
	     (let ((sb-walker:*walk-form-expand-macros-p* t))
	       (sb-walker:walk-form
		form nil
		(lambda (subform context wenv)
		  (declare (ignore wenv))
		  (cond ((and (eq context :eval)
			      (listp subform)
			      (eq (car subform) 'cache-send))
			 (let ((class-var (gensym))
			       (func-var (gensym)))
			   (push class-var cache-vars)
			   (push func-var cache-vars)
			   `(send-with-cache ,class-var ,func-var ,@(cdr subform))))
			(t
			 subform)))))))
      (let ((ndefun (transform `(defun ,name ,params
				  ,@body))))
	`(let ,(iter (for var :in cache-vars)
		     (collect `(,var nil)))
	   ,ndefun)))))

(cache-defun stress2 ()
  (let* ((c (make-trivial-some-class))
	 (o (some-class-make-instance c)))
    (iter (for i :from 0 :to +stress-loops+)
	  (declare (type fixnum i))
	  (cache-send o 'inc 1)
	  (cache-send o 'dec 1))))

The last real function to call and its class is cached in 2 lexical variables. For subsequent invoking, the cached function is used directly if the class is same as the cached one. A code walker is used to generate lexical variables definition and use them in each call site. Unfortunately, it is quite hard/complex to implement a portable code walker, so the code walker of SBCL is used. It appears that many common lisp implementations have code walker implementation by themselves, so similar stuff can be implemented in some other common lisp implementations easily as well.

How about the performance gain of inline caching? Result is as follow on a low voltage mobile CPU.

CL-USER> (time (cl-inline-cache::stress1))
Evaluation took:
  0.183 seconds of real time
  0.184000 seconds of total run time (0.184000 user, 0.000000 system)
  100.55% CPU
  436,898,331 processor cycles
  0 bytes consed
  
NIL
CL-USER> (time (cl-inline-cache::stress2))
Evaluation took:
  0.107 seconds of real time
  0.108000 seconds of total run time (0.108000 user, 0.000000 system)
  100.93% CPU
  256,277,313 processor cycles
  0 bytes consed
  
NIL

The performance gain is about 70%.

I should have said that the meta programming capability of common lisp makes this much easier to be implemented or even possible. But I found that this can be implemented in C with static local variable even easier. Never the less, I still can say that common lisp provides greater flexibility.

The inline caching here convert hash table look up into indirect call. While in typical implementation, this can be further optimized into direct call. That could be implemented in common lisp via converting the code and call compile function at run time.