Slow manual bitrate switching

When specifying a manual bitrate, in our app that uses OSMF1.5, we notice the switch takes some time to occur in the cleint.

For e.g

We see the stream switch on FMS (using the admin console) in about ~5 secs. In the client app it takes ~20 secs.

Is there a way to force the client switch to happen faster? Even if it means losing the seamless switch transitions (in other words we are ok with pausing the stream during the switch for)?

Thanks,

- Abey