I used to find the concept of the chemical potential rather confusing.
Hence, it is not surprising that students struggle too.
I could say the mantra that "the chemical potential is the energy required to add an extra particle to the system" but how it then appeared in different thermodynamic identities and the Fermi-Dirac distribution always seemed a bit mysterious.
However, when I first taught statistical mechanics 15 years ago I used the great text by Daniel Schroeder. He has a very nice discussion that introduces the chemical potential. He considers the composite system shown below, where a moveable membrane connects two systems A and B. Energy and particles can be exchanged between A and B. The whole system is isolated by the environment and so the equilibrium state is the one which maximises the total entropy of whole system.
Thermal equilibrium (i.e. there is no net exchange of energy between A and B) occurs if the temperature of A equals that of B. Thus, temperature is the thermodynamic state variable that tells us where two systems are in thermal equilibrium.
Diffusive equilibrium (i.e. there no net exchange of particles between A and B) occurs if the chemical potential of particles in A equals that in B, where the chemical potential is defined as
Thus, the chemical potential is the thermodynamic state variable/function that tells us whether or not two systems are in diffusive equilibrium.
Doug Natelson also has a post about this topic. He mentions the American Journal of Physics article on the subject by Ralph Baierlein, drawing heavily from his textbook. However, I did not find that article very helpful, particularly as he mostly uses a microscopic approach, i.e. statistical mechanics. (Aside: the article does have some interesting history in it though).
I prefer to first use a macroscopic thermodynamic approach before a microscopic one as, I discussed in my post, What is temperature?