Project captain, Admin of the project or Project manager can restrict an access to To-Do lists by setting permissions for other users. You can set up the access when creating To-Do list or anytime later.
Go to the Users section in the project detail and set up who can see particular To-Do lists. Each user must have an access to one To-Do list in project at least. Click Can see all To-Do lists and select only those you want the user to have an access to.
Invite user to one To-Do list directly
You can invite a new user directly to particular To-Do list in project. Go to section Users on detail of the project, click on green plus icon and select user. Unselect option Invited users will see all To-Do lists and choose To-Do lists accessed by the user. Confirm by green button Save.
Remember to permit the access at least to one To-Do list, otherwise the user has an access to all of them.