Usually we select objects by drawing the diagonal of a rectangle, and anything that's not already in a rectangle requires us to do multiple operations or make multiple selections while holding down Ctrl. Instead of selecting with rectangles, why not allow freeform selection with a lasso? Here's a mock-up, with more description here. Drag your mouse in the second window to make a freeform selection.
Notice how you don't have to finish the entire outline in order to select something, and how you can backtrack to erase the line if you make a mistake.
Made with Processing.js.