Switching control modes?

Yes. You would send serial commands.

Refer to user manual for more information on command references.

^mmod cc nn

Where cc = motor channel and nn = mode

In your case closed loop position relative = 2 and count = 3

Closed loop position relative for motor 1
^mmod 1 2

^mmod 1 3
Closed loop count position.

If for whatever reason you are in the count mode, and you reset the counter back to 0, it would cause erratic behavior due to the fact a command is telling it to be at a specific count. Be sure to set to open loop mode first to reset encoder count value.

Open loop motor 1
^mmod 1 0

Reset encoder counter 1 to 0
!C 1 0

Our controllers use A, B, +5V, GND pins. The index pin can be left floating.

What would be the reason to use the SPI in terms of communication since the commands to the encoder would be done via serial for gather data? It says incremental output, but incremental encoders are used for a speed based operation and not a position based one like optical or absolute encoders.

The controller will gather the data such as speed and count via the A and B channels. As for resetting the counter and gathering of that data is done via serial through either a serial query command such as ?C or through a microbasic script you would need to write that would print back that data to either our console tab, or through another program such as Real Term, Hyperterminal, etc...