oxygen is the enemy of beer. beer in fermentors, brite tanks, kegs, bottles and cans are in an O2-free enviro. when filling bottles, the bottle is purged with CO2 before being filled. (aside: cans are very rarely flushed first - takes a super-high-end machine to do this - so the beer is exposed to O2 before being sealed. but once sealed, no O2 can get in - vs. bottles that start with almost no O2 but some will eventually creep in because crown caps aren't perfect seals).
as soon as beer is exposed to oxygen, it starts to degrade. during a growler fill the beer is exposed to a lot of O2 so the clock starts ticking as soon as it's filled. 2 or 3 days is the standard answer you'll get for "how soon should i drink it", but the real answer is "as soon as possible, ideally right away."