Captain, Admin of the project or Project manager can restrict access to To-Do lists by setting permissions for other users. You can set up the access when creating To-Do list or anytime later.
Go to the Users section in the project detail and set up who and can see To-Do lists.
Invite user to particular To-Do list only
You can invite new user directly to particular To-Do list in project. Go to section Users on detail of the project > click on green plus button and select user > unselect option Invited users will see all To-Do lists and choose To-Do lists accessed by the user.