Scorecard visibility is governed by an account-wide setting: "Keep scorecard private."
When ON: only admins and users explicitly listed as recruiters on that specific job can view scorecards for it.
When OFF: scorecards are visible to anyone with view access to the job.
Where to change it: this is an account-level setting, not a per-view or per-card setting. Check with an admin if you're unsure of your account's current configuration.