I found what my problem was, and this may help other people so I will post as an answer.

Make sure you don't have anything else marking your emails as read.  I accidentally had an old development instance of Odoo get restarted when a server restarted, and this old instance of Odoo also had an active connection to my main email account I setup for the emails being sent into Odoo (catchall).  

So when that other "rouge" cron job would run, it would grab the new emails into that old Odoo installation, mark the emails as read, and therefore my correct Odoo instance would not pull those emails in because they were already marked read.

This may not solve the problem for everyone, but it fixed it for me.  Now all my emails are correctly fetched on schedule, and are routed to the correct places as long as whomever responded didn't ruin the email headers in some way.

Hope this can help some people.

If you want to test this, just turn off fetching on your Odoo instance, and watch the email inbox to see if mails are being marked as read.  If Odoo isn't fetching emails, but emails are being marked read, then you need to find out what is marking those emails outside of Odoo and stop that action.