M-x shell open shell in other windows

Question

In previous emacs, "M-x shell" new a shell buffer in current windows.

but recently, I update my emacs to GNU Emacs 26.0.50.2. "M-x shell" new a shell buffer in the other windows. I search google and could not find answer. does anybody know how to prevent this behavior.

Answer 1

The reason is that shell uses (pop-to-buffer buffer) instead of (switch-to-buffer buffer). I don't know how to advise the function, so I can't give you an appropriate answer. However if you only want shell to work the way you want you can simply add the whole function in your config.

Maybe somebody else can replace the function with an advice. I would be interested in that solution.

(defun shell (&optional buffer)
  "Run an inferior shell, with I/O through BUFFER (which defaults to `*shell*').
Interactively, a prefix arg means to prompt for BUFFER.
If `default-directory' is a remote file name, it is also prompted
to change if called with a prefix arg.

If BUFFER exists but shell process is not running, make new shell.
If BUFFER exists and shell process is running, just switch to BUFFER.
Program used comes from variable `explicit-shell-file-name',
 or (if that is nil) from the ESHELL environment variable,
 or (if that is nil) from `shell-file-name'.
If a file `~/.emacs_SHELLNAME' exists, or `~/.emacs.d/init_SHELLNAME.sh',
it is given as initial input (but this may be lost, due to a timing
error, if the shell discards input when it starts up).
The buffer is put in Shell mode, giving commands for sending input
and controlling the subjobs of the shell.  See `shell-mode'.
See also the variable `shell-prompt-pattern'.

To specify a coding system for converting non-ASCII characters
in the input and output to the shell, use \\[universal-coding-system-argument]
before \\[shell].  You can also specify this with \\[set-buffer-process-coding-system]
in the shell buffer, after you start the shell.
The default comes from `process-coding-system-alist' and
`default-process-coding-system'.

The shell file name (sans directories) is used to make a symbol name
such as `explicit-csh-args'.  If that symbol is a variable,
its value is used as a list of arguments when invoking the shell.
Otherwise, one argument `-i' is passed to the shell.

\(Type \\[describe-mode] in the shell buffer for a list of commands.)"
  (interactive
   (list
    (and current-prefix-arg
         (prog1
             (read-buffer "Shell buffer: "
                          ;; If the current buffer is an inactive
                          ;; shell buffer, use it as the default.
                          (if (and (eq major-mode 'shell-mode)
                                   (null (get-buffer-process (current-buffer))))
                              (buffer-name)
                            (generate-new-buffer-name "*shell*")))
           (if (file-remote-p default-directory)
               ;; It must be possible to declare a local default-directory.
               ;; FIXME: This can't be right: it changes the default-directory
               ;; of the current-buffer rather than of the *shell* buffer.
               (setq default-directory
                     (expand-file-name
                      (read-directory-name
                       "Default directory: " default-directory default-directory
                       t nil))))))))
  (setq buffer (if (or buffer (not (derived-mode-p 'shell-mode))
                       (comint-check-proc (current-buffer)))
                   (get-buffer-create (or buffer "*shell*"))
                 ;; If the current buffer is a dead shell buffer, use it.
                 (current-buffer)))

  ;; On remote hosts, the local `shell-file-name' might be useless.
  (if (and (called-interactively-p 'any)
           (file-remote-p default-directory)
           (null explicit-shell-file-name)
           (null (getenv "ESHELL")))
      (with-current-buffer buffer
        (set (make-local-variable 'explicit-shell-file-name)
             (file-remote-p
              (expand-file-name
               (read-file-name
                "Remote shell path: " default-directory shell-file-name
                t shell-file-name))
              'localname))))

  ;; The buffer's window must be correctly set when we call comint (so
  ;; that comint sets the COLUMNS env var properly).
  (switch-to-buffer buffer)
  (unless (comint-check-proc buffer)
    (let* ((prog (or explicit-shell-file-name
                     (getenv "ESHELL") shell-file-name))
           (name (file-name-nondirectory prog))
           (startfile (concat "~/.emacs_" name))
           (xargs-name (intern-soft (concat "explicit-" name "-args"))))
      (unless (file-exists-p startfile)
        (setq startfile (concat user-emacs-directory "init_" name ".sh")))
      (apply 'make-comint-in-buffer "shell" buffer prog
             (if (file-exists-p startfile) startfile)
             (if (and xargs-name (boundp xargs-name))
                 (symbol-value xargs-name)
               '("-i")))
      (shell-mode)))
  buffer)

Answer 2

Unless you're customizing the names of your shell buffers, this should be all you need:

(add-to-list 'display-buffer-alist
             `(,(regexp-quote "*shell") display-buffer-same-window))

To handle all shell buffers, whatever their names, you can use this advice:

(defun shell-same-window-advice (orig-fn &optional buffer)
  "Advice to make `shell' reuse the current window.

Intended as :around advice."
  (let* ((buffer-regexp
          (regexp-quote
           (cond ((bufferp buffer)  (buffer-name buffer))
                 ((stringp buffer)  buffer)
                 (:else             "*shell*"))))
         (display-buffer-alist
          (cons `(,buffer-regexp display-buffer-same-window)
                display-buffer-alist)))
    (funcall orig-fn buffer)))

(advice-add 'shell :around #'shell-same-window-advice)

Answer 3

Add this line in your .emacs file:

(push (cons "\\*shell\\*" display-buffer--same-window-action) display-buffer-alist)

This fixed it for me. I'm using eMacs 25.2 (9.0) on a Mac, and was getting really annoyed by the shell opening up in another frame, or even a new frame when there was only one. This is the source where I got this answer from.