The US defense secretary, Ash Carter, has accused Russia of endangering world order and said the Pentagon was searching for creative ways to deter Kremlin aggression and protect its allies.

Speaking in California after eight days of travel in Asia, Carter also expressed concern about China’s expanding influence and growing military might. But he reserved his strongest words for Russia.

Carter told an audience at the Ronald Reagan Presidential Library in Simi Valley that Russia was undertaking “challenging activities” at sea, in the air, in space and in cyberspace.

“Most disturbing, Moscow’s nuclear sabre-rattling raises questions about Russian leaders’ commitment to strategic stability, their respect for norms against the use of nuclear weapons, and whether they respect the profound caution nuclear-age leaders showed with regard to the brandishing of nuclear weapons,” he said.

His remarks were perhaps the strongest he has expressed about America’s former Cold War foe.

“We do not seek a cold, let alone a hot, war with Russia,” he said. “We do not seek to make Russia an enemy. But make no mistake; the United States will defend our interests, our allies, the principled international order, and the positive future it affords us all.”

The backdrop to Carter’s remarks is the reality that after more than two decades of dominating great-power relations, the United States is seeing Russia reassert itself and China expand its military influence beyond its own shores.

Carter cited several pillars of the international order that he argued should be defended and strengthened: peaceful resolution of disputes, freedom from coercion, respect for state sovereignty and freedom of navigation.

“Of course, neither Russia nor China can overturn that order,” he said. “But both present different challenges for it.”

Under the presidency of Vladimir Putin, Russia is challenging the US in many arenas, including the Arctic, where last year Moscow said it was reopening 10 former Soviet-era military bases that were closed after the Cold War ended in 1991. Russia also is flying more long-range air patrols off US shores.

Carter left open the possibility that Russia’s role in Syria could evolve into one the US can embrace.

“It is possible – we’ll see – Russia may play a constructive role in resolving the civil war,” he said.

In a question-and-answer session with his audience, Carter said he believed Putin “hasn’t thought through very thoroughly” his objectives in Syria. He called the Russian approach there “way off track”.

As Russia makes what Carter characterised as threatening statements about its potential use of nuclear weapons, the US was modernising its entire nuclear arsenal — not only the submarines, bomber aircraft and land-based missiles that are armed with long-range nuclear weapons, but also the weapons themselves.

“We’re investing in the technologies that are most relevant to Russia’s provocations, such as new unmanned systems, a new long-range bomber, and innovation in technologies like the electromagnetic railgun, lasers and new systems for electronic warfare, space and cyberspace, including a few surprising ones I really can’t describe here,” he said.

Carter said China was the single most influential player in Asia’s future, and he noted that earlier this week he went aboard an American aircraft carrier in the South China Sea to demonstrate US commitment to freedom of navigation. Washington objects to China’s claims of territorial limits around disputed artificial islands there.

“As a rising power, it’s to be expected that China will have growing ambitions and a modernizing military,” he said. “But how China behaves will be the true test of its commitment to peace and security.”