Should I be upset about this?

How about preparing a list, hanging it by the door with a DOLLAR SIGN and and envelope.??? Then, when one of them asks you about it, tell them: "Here's the money (have it there), with an extra $5.00....would you please let me come get groceries with you?" That's payment for gas and their trouble. Might work?