Both FTM and Ancestry don't seem to provide info on size because both describe the process as using a standard photo (zoomed to an area set by the user) rather than a dedicated portrait photo.
See Software MacKiev for general media info.
https://support.mackiev.com/017658-Comparing-FTM-and-Ancestr...Here are some thoughts:
Despite FTM using a square and Ancestry using a circle the "sides" of both are a 1:1 ratio regardless of the actual measurement used. i.e. 2" by 2" (2" diameter) or 3" by 3" (3" diameter) both scale the same. If you use a square as long as you center your face it won't matter when the circle clips the square's corners and then you only need one profile photo shape. [If you are using a photo editing program like Photoshop simply draw a selection circle from the center point of the square to the edge of the square to temporarily see what will be clipped in a circle profile shape.]
If you are just worried about online/screen viewing:
Standard web screen resolution is 72 pixels per inch. For a while it was recommended using 96 ppi but high def really figures now to about 140 ppi per one website. However, most computer screens are likely still standard.
So a 3" photo at 72 dpi/ppi is about 3" on screen (disregarding if you have your browser view zoomed in or out) and trying to enlarge the view of it will result is a diggy appearance because all there is is 72 ppi to work with. To enlarge the view you'll need to provide some additional resolution (dpi/ppi) or a larger size of the image. If you are just worried about online viewing 200 ppi/dpi should work but size will come from the photo you are working with.
PPI is pixels per inch which refers to the pixels monitor screens are made of while DPI is dots per inch and is a printing term referring to dots of ink on paper (or another surface)
Some photo editing programs use these terms interchangeably.
If you have Photoshop (Elements or CC) if you scan a small photo at higher resolution (like 600 or higher), you can usually increase the size of the photo by reducing resolution (uncheck resample image) in Resize Image. [ Always work with a copy file not your original scan so you can always go back and not have to rescan the image.] But remember, if your original photo is fuzzy your scan will be fuzzy. Sometimes you can crisp it up by working on (editing) it but not always.
Example: a 2" x 2" image scanned at 600 ppi can be resized with resample image unchecked to 200 ppi so it becomes a 6" x 6" image. If 6"x6" is bigger than you need/want, you can drop the size but keep the resolution by checking the resample image before changing the size so only the size is effected.
Profile photos in FTM's or Ancestry's Family Group Sheets and trees do not print very large so the 200 should more than suffice for those situations. You just have to decide what size image to use.
Printing
Standard minimum for commercial printing (and many photo printing places) is 300 dpi at the size it is to be printed at. Most home printers can work with less resolution but remember you won't be able to enlarge those photos very much so more resolution helps. Since you have your images on your computer you can tweak swap as needed for your projects. Those accessing your online images will be restrained by the size/resolution you select.
So if you plan to use your dedicated profile photos for more than just online, use a 300 ppi resolution and make them a decent size like 3" x 3" if possible.
Lastly, don't forget image compression can effect quality so do not over compress your files like jpg that have compression options.
Hopefully this makes sense for you.
For scanning one's photo collection for preservation which is "another story," there are other recommendations on size and resolution and file format.