Reducing the Number of Physical Volumes (PVs) in LVM

The legacy “MBR” partitioning scheme limits the number of drives to four and, on some older virtual machines, I fell into a trap of expanding LVMs by adding more physical volumes. This was not an issue until I went to reboot the machine and it failed.

Note – all of the below was performed on a production system without service interruption. There are easier ways of doing this if the server could be taken offline.

Here was the problem:

root@vm01:/root# pvs -o+pv_used
  PV         VG   Fmt  Attr PSize   PFree Used
  /dev/xvda1 sys  lvm2 a--  <10.00g    0  <10.00g
  /dev/xvdb1 data lvm2 a--  <30.00g    0  <30.00g
  /dev/xvdc1 data lvm2 a--  <50.00g    0  <50.00g
  /dev/xvde1 data lvm2 a--  <50.00g    0  <50.00g
  /dev/xvdf1 data lvm2 a--  <50.00g    0  <50.00g
  /dev/xvdg1 data lvm2 a--  <60.00g    0  <60.00g

I ended up with six physical volumes. My “data” volume group spanned xvdb through xvdg (five PVs). To resolve this, my plan was to add a seventh (!), move xvdb’s data there, then replace xvdb with a much larger volume and then move everything from xvdc, xvde, xvdf, xvdg to the new xvdb.

First, I extended the data VG by adding a new PV xvdh1 at 31G (just an extra 1G to ensure I would have no issues moving xvdb’s 30G over). I then moved the data out of xvdb via:

# # move data off xvdb
# pvmove /dev/xvdb1
/dev/xvdb1: Moved: 0.05%
/dev/xvdb1: Moved: 5.57%
...
/dev/xvdb1: Moved: 98.32%
/dev/xvdb1: Moved: 100.00%

# # reduce the data VG by removing xvdb1
# vgreduce data /dev/xvdb1

# # finally, remove the PV itself
pvremove /dev/xvdb1

On the hypervisor (Xenserver / XCP-ng) in my case, I resized the underlying virtual drive for xvdb by deactivating it, resizing it, and re-activating it. Then on the virtual machine, I used cfdisk to resize it to use the additional space. Then:

# # create a new PV
# pvcreate /dev/xvdb1

# # extend the data VG to use the new PV
# vgextend data /dev/xvdb1

Then, for each of xvdh1, xvdg1, xvdf1, xvde1, and xvdc1 (denoted as xvdX below) do the following:

# # move data off xvdX
# pvmove /dev/xvdX

# # reduce the data VG by removing xvdX
# vgreduce data /dev/xvdX

# # reduce the data VG by removing xvdX
pvremove /dev/xvdX

Then, on the hypervisor, remove these hard drives from the virtual machine leaving only xvda and the much enlarged xvdb.

root@vm01:/root# pvs -o+pv_used
  PV         VG   Fmt  Attr PSize    PFree   Used
  /dev/xvda1 sys  lvm2 a--   <10.00g      0  <10.00g
  /dev/xvdb1 data lvm2 a--  <250.00g <10.02g 239.98g