Thank you. Strangely, on 12.2.0 it made things actually worse, see the log below. As you see from the log I have verified which library gets mapped into WolframKernel process' address space. I will test with the other versions as well and will let you know the result:

$ pwd
/opt/Wolfram/WolframEngine/12.2/SystemFiles/Libraries/Linux-ARM
$ sudo mv libgmp.so.10 libgmp.so.10.orig
$ apt list --installed | grep libgmp10

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libgmp10/oldstable,now 2:6.1.2+dfsg-4 armhf [installed]
$ wolfram
Mathematica 12.2.0 Kernel for Linux ARM (32-bit)
Copyright 1988-2021 Wolfram Research, Inc.

In[1]:= !grep libgmp /proc/$(pgrep WolframKernel)/maps
71522000-7158a000 r-xp 00000000 b3:02 271806     /usr/lib/arm-linux-gnueabihf/libgmp.so.10.3.2
7158a000-71599000 ---p 00068000 b3:02 271806     /usr/lib/arm-linux-gnueabihf/libgmp.so.10.3.2
71599000-7159a000 r--p 00067000 b3:02 271806     /usr/lib/arm-linux-gnueabihf/libgmp.so.10.3.2
7159a000-7159b000 rw-p 00068000 b3:02 271806     /usr/lib/arm-linux-gnueabihf/libgmp.so.10.3.2

In[2]:= AbsoluteTiming[N[Pi,10^6];]     

Out[2]= {4.58025, Null}

$ sudo mv libgmp.so.10.orig libgmp.so.10
$ wolfram
Mathematica 12.2.0 Kernel for Linux ARM (32-bit)
Copyright 1988-2021 Wolfram Research, Inc.

In[1]:= !grep libgmp /proc/$(pgrep WolframKernel)/maps      
71538000-71595000 r-xp 00000000 b3:02 394435     /opt/Wolfram/WolframEngine/12.2/SystemFiles/Libraries/Linux-ARM/libgmp.so.10
71595000-715a4000 ---p 0005d000 b3:02 394435     /opt/Wolfram/WolframEngine/12.2/SystemFiles/Libraries/Linux-ARM/libgmp.so.10
715a4000-715a5000 r--p 0005c000 b3:02 394435     /opt/Wolfram/WolframEngine/12.2/SystemFiles/Libraries/Linux-ARM/libgmp.so.10
715a5000-715a6000 rw-p 0005d000 b3:02 394435     /opt/Wolfram/WolframEngine/12.2/SystemFiles/Libraries/Linux-ARM/libgmp.so.10

In[2]:= AbsoluteTiming[N[Pi,10^6];]                

Out[2]= {3.28563, Null}

In the above I considered degradation of 12.2 (going from 2 to 3.25 seconds) negligible, but degradation of 12.3.1 (going from 2 to 7.67 seconds) substantial and deserving attention.

Thank you for reporting this issue, it should be corrected in our next release.

In the meantime, the following may be used as a workaround

$ sudo apt install libgmp10

$ sudo mv /opt/Wolfram/WolframEngine/12.3/SystemFiles/Libraries/Linux-ARM/libgmp.so.10 \
  /opt/Wolfram/WolframEngine/12.3/SystemFiles/Libraries/Linux-ARM/libgmp.so.10.bak