Why was American government's response so soft towards the Chinese government's very aggressive messages about Pelosi's Taiwan visit?

Recently the U.S. announced that Pelosi would be visiting Taiwan. The Chinese government threatened that if that happened, they would respond militarily. Xi even told Biden the following words: "Those who play with fire will perish by it. It is hoped that the U.S. will be clear-eyed about this.". These are very strong words to be told to the president of America. But the Americans are even discussing the cancellation of the visit. Is it not damaging to the international reputation of the U.S.? Why are they letting another country to interfere with their visit to their strategic partner? Please, correct me if any of my assumptions above are wrong.