A* Search but we don't know the heuristics for all states

Question

I'm trying to do a project about A* search (Finding a path between source and target where each block can be a path with cost 4, a brick with cost 10 or a hole with cost 50 ) but it's different in a way that you don't have the heuristics for all states initially and you can calculate the heuristic only for the state you are in. My first thought was to visit all the adjacent blocks and store the heuristics and come back to the previous block in order to find all heuristics and calculate f and decide which node to expand but that would be a big waste of cost as you would lose at least 4 scores for visiting each node. My TA said there is a better way of doing this but I can't figure out how it might be possible not to calculate all the heuristic values as the one we are not calculating may be the one that is actually more efficient.