I think you want to move the pane_in_mode check. Try:
bind -T root WheelUpPane if -Ft= '#{mouse_any_flag}' 'send-keys -M' 'if -Ft= "#{pane_in_mode}" "send-keys -M" "copy-mode -u"'
You want your mode key binding to happen if you are in a mode,
regardless of mouse_any_flag, so remove the pane_in_mode check from the
true branch of mouse_any_flag.
You only want to send copy-mode -u if you aren't in a mode, otherwise
you want to send the key through to the mode so it'll get your halfpage
binding.
This flag checking is a bit horrible, maybe we can make it better...
On Wed, Jun 17, 2015 at 06:47:44PM +0000, Kaushal wrote:
>    Thanks.
> 
>    I almost got it working.
>    The only part that's not working is where I have the WheelUpPane bound to
>    different command in **-T root and -t emacs-copy. It looks like the -T
>    root command gets the higher priority (the binding to halfpage-up is
>    ineffective). Can we not have both?
>    These are my current bindings for WheelUpPane:
>    # Do mouse wheel-up to enter copy mode and do page-up
>    bind -T root WheelUpPane if -Ft= '#{mouse_any_flag}' 'if -Ft=
>    "#{pane_in_mode}" "copy-mode -u" "send-keys -M"' 'copy-mode -u'
>    # Once in copy-mode, mouse wheel scrolls scrolls by half pages
>    bind -temacs-copy WheelUpPane ** halfpage-up
>    How can I fix the bindings so that when not in copy-mode, WheelUpPane
>    enters copy-mode but when once in copy-mode, WheelUpPane does only
>    half-page scrolls.
>    On Wed, Jun 17, 2015 at 12:54 PM Nicholas Marriott
>    <[1]
nicholas...@gmail.com> wrote:
> 
>      On Tue, Jun 16, 2015 at 06:01:43PM +0000, Kaushal wrote:
>      >** ** The `npage` command binding works (but the result is very jumpy,
>      unlike
>      >** ** the default scroll behavior of 1.9a). But it is possible to
>      control how
>      >** ** many lines to scroll? In the source code, I see references to
>      terms like
>      >** ** halfpage-down and halfpage-up but I don't know how to use them in
>      >** ** bindings.
> 
>      You need to bind the key in the mode keys table:
> 
>      ** ** ** ** tmux bind -temacs-copy WheelUpPane halfpage-up
> 
>      (or vi-copy if you use mode-keys=vi)
> 
>      >** ** Another related question I have is: How to quit the copy-mode if
>      I right
>      >** ** click on a pane? I would like to have this behavior:**
>      >** ** if copy-mode**
>      >** ** ** **quit copy-mode
>      >** ** else
>      >** ** ** **mark the pane (currently what right-click does if mouse is
>      on)
> 
>      ** ** ** ** tmux bind -temacs-copy MouseUp3Pane cancel
> 
>      >** ** I am simply trying to replicate the mouse-mode behavior of 1.9a.
>      >** ** On Mon, Jun 15, 2015 at 12:43 PM Kaushal
>      <[1][2]
kausha...@gmail.com>
>      >** ** wrote:
>      >
>      >** ** ** Resolved the PageDown action on WheelDownPane using the below:
>      >** ** ** bind -T root WheelDownPane send-keys npage
>      >** ** ** So far, it seems to work fine; please comment if that behavior
>      should be
>      >** ** ** configured in a different way.
>      >** ** ** --
>      >** ** ** Kaushal Modi
>      >** ** ** On Mon, Jun 15, 2015 at 11:40 AM, Kaushal
>      <[2][3]
kausha...@gmail.com>
>      >** ** ** wrote:
>      >
>      >** ** ** ** Thanks! I tried that and it worked after I added "-T root"
>      after bind.
>      >** ** ** ** I have a followup question:
>      >** ** ** ** - Doing a wheelup enters the copy mode fine and it scrolls
>      the pane by
>      >** ** ** ** pages. A single "click" of wheelup (I use discrete mouse
>      wheeling)
>      >** ** ** ** scrolls the pane by one "page". But the wheeldown action
>      scrolls down
>      >** ** ** ** by just one line.
>      >** ** ** ** How do I control by how many lines the pane should scroll
>      when doing
>      >** ** ** ** wheelup and wheeldown?
>      >** ** ** ** --
>      >** ** ** ** Kaushal Modi
>      >** ** ** ** On Sat, Jun 13, 2015 at 4:28 AM, Nicholas Marriott
>      >** ** ** ** <[3][4]
nicholas...@gmail.com> wrote:
>      >
>      >** ** ** ** ** On Fri, Jun 12, 2015 at 07:24:32PM +0000, Kaushal wrote:
>      >** ** ** ** ** >** ** Hi,
>      >** ** ** ** ** >** ** I liked the mouse scroll behavior in tmux 1.9a
>      where it
>      >** ** ** ** ** automatically went
>      >** ** ** ** ** >** ** into copy-mode and then scrolled the pane.
>      >** ** ** ** ** >** ** How should I bind WheelUpPane and WheelDownPane
>      to get that
>      >** ** ** ** ** same
>      >** ** ** ** ** >** ** behavior?**
>      >
>      >** ** ** ** ** Try something like (not tested):
>      >
>      >** ** ** ** ** bind WheelUpPane if -Ft= '#{mouse_any_flag}' 'if -Ft=
>      >** ** ** ** ** "#{pane_in_mode}" "copy-mode -u" "send-keys -M"'
>      'copy-mode -u'
>      >
>      > References
>      >
>      >** ** Visible links
>      >** ** 1. mailto:[5]
kausha...@gmail.com
>      >** ** 2. mailto:[6]
kausha...@gmail.com
>      >** ** 3. mailto:[7]
nicholas...@gmail.com
> 
> References
> 
>    Visible links
>    1. mailto:
nicholas...@gmail.com
>    2. mailto:
kausha...@gmail.com
>    3. mailto:
kausha...@gmail.com
>    4. mailto:
nicholas...@gmail.com
>    5. mailto:
kausha...@gmail.com
>    6. mailto:
kausha...@gmail.com
>    7. mailto:
nicholas...@gmail.com