It's a good idea to keep all icons and images in one or more folders.
Right click on the project, and use F# Power Tools / New Folder to create a folder named Images.
On disk, place your icon in the new Images folder.
Back in Visual Studio, right click on Images, and use Add / Existing Item, then show All Files (.)** to see the icon file so that you can select it, and then Add it.
Select the icon file, and set its Build Action to Resource.
In MainWindow.xaml, use the Icon attribute like this. Surrounding lines are shown for context.
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
Title="First Demo" Height="200" Width="300"
Icon="Images/MainWindow.ico">
<Canvas>
Before you run, do a Rebuild, and not just a Build. This is because Visual Studio doesn't always put the icon file into the executable unless you rebuild.
It is the window, and not the application, that now has an icon. You will see the icon in the top left of the window at runtime, and you will see it in the task bar. The Task Manager and Windows File Explorer will not show this icon, because they display the application icon rather than the window icon.