Will a tag manager affect my website's performance?

No, tag managers behave asynchronously to avoid degrading website performance and offering a poor user experience. This means that when a user loads a page on your website, the tags you've added won't block the page from rendering for the user. Contrast this with synchronous behavior, which would stop the page from fully loading until all of the tags had finished loading.

Google Tag Manager's Developer Guide explains how async behavior works in GTM.