I dont think that it is going to work with the SL40 tied to the same port, have you tried it?

Does the SPA-4S have unswitched audio inputs?