Demands are growing in Scotland for a "smaller state" with fewer public bodies and lower taxes, according to new research.