Correct, since most people use RunSignUp I'll use it as the example, but it should work with RaceRoster and RaceEntry if they have the option to mark participants as "checked in" in their system too.
Basically if you check "Require bib #" then of course when you hit "Sync" it will only pull in participants that have been assigned a bib # on RunSignUp. This is all I ever check.
The "Checked In" does the same thing, only pull in people that are checked in either through their website or on the app. This option comes in handy when a race wants to pre-assign bib #'s but you only want to pull in participants that truly showed up at the race and not every person that registered for the race.
Side note: As a safety measure the software automatically has the "Require Check In" box checked when you open the race on race day. If you ever mistakenly click 'Sync' without that checked and pull everyone in by accident, simply delete everyone out of the timing software and check the box and click Sync again.