What a fabulous vacation that will be! Prince Edward Island! Ahhh....I've only been there once briefly, but the landscape is such that it will keep me wanting to go back.
I will have to think in regards to the Maine public gardens. Sadly nothing is coming to mind. However....! Just before you cross into Maine, you will go through Portsmouth, NH. I think that if you have never been there. You would love Strawberry Banke and Prescott Park. Strawberry Bank is the old port part of Portsmouth. They have many, many old homes that you can tour. But most importantly, they have gardens! And then, Prescott Park has a few nice size gardens. Mostly annuals with some perenials. If you are in the area, it is worth a stop and Portsmouth is definately a place that most could easily spend the day.
When I was a small child, Prescott Park would have a "free" day at the end of the year. Free meaning that you could go over and pick the flowers from their garden or dig up, etc. My Nana used to take advantage of that day. They seemed to have stopped for some reason when I was a young teen. It was a pretty cool thing to do though.
strawberrybank.org is their web site. Sadly I just looked and it doesn't do justice to the place in my opinion.
I feel like a tour guide for NH instead of Maine...

but The Fuller Gardens in North Hampton, NH is another worth while stop. fullergardens.org is their site. It is about a 25-30 minute drive from North Hampton to Portsmouth.
OK Something for Maine. If you are able to stop at Acadia National Park, the veiws are amazing!! Not specifically gardens, but a sight to behold.
Just call me "Julie your tour director"!