Single Sign on from outside the network?

Hi, All

OK, so I have set up ADFS and SSO for our CRM/Office 365 users.  This works perfectly when you're within the network.

Yet, despite having an ADFS Proxy, and the query going there, users from the outside world don't seem to get the SSO benefits.

So, I wondered what I may do to help our external users, so that they all get the benefit of SSO?  Do we need to set up some sort of user certificate, or can we modify things so that auto-authentication happens?