A few random thoughts:
Another finger is touching the string behind where you are fretting it. You don’t notice until you pull off the fretting finger.
You may be moving your finger in a way that creates a “pull off”. A pull off is a deliberate attempt to play a new note on the string by basically picking it with your fretting finger. you may be doing this accidentally.
The above may cause the string to vibrate enough it buzzes against an adjacent finger.
Your fingernail could be striking the string as you pull it off.
Almost all of these things will naturally get better with time. Of course if you can identify the problem you can work on it specifically and speed up the learning process. At first I never felt a string buzzing against my finger. Now I can feel it and adjust.