I ran a vacuum over the ports just for an added element of safety, though, like you, I didn't go gouging down in to the ports.About the same thing i did. I used a screwdriver to loosen it a bit, nothing fell in, and then wiped outward with my thumb. Nothing moved but what I had worked loose a little at the edge. I just don't see this stuff coming loose down the road. I would be more worried if I scraped this stuff off and vacuumed it out, and the vacuum didn't pick something up.
The main driver for even bothering was that I didn't want to dislodge that lip when I bolted the intake back up.
WARNING to anyone stumbling across this thread in the future: if you are NOT sure that nothing is going to dislodge and fall in toward the valve then it's advised that you do NOT try to do what is being discussed here. Each is responsible for their own decisions. Every situation, no matter how similar it may seem to another, is different.