Your website has to be visible on all platforms. Fewer people have desktops and more than 50% of purchases are made from mobile devices.
Non-responsive sites get lower ranking in Search Engines. Responsive sites get lower bounce rates i.e. people stay longer on your site.
You only get a few seconds to keep a client on your site. If they see a mess they leave. Your site must still look ok if the device is rotated either way.
Most users have multiple devices and when they view your site on one device they expect it to be useable on another device without scrolling sideways.