The available module that Refused to Load: Debugging HPC Module Extensions with Lmod

If you’ve ever spent 20 minutes trying to import pandas on a high-performance computing (HPC) cluster only to be greeted by ModuleNotFoundError: No module named 'pandas', you’re not alone.

This is the story of how I (and a user on our cluster) finally got pandas working — and what you should do the next time module load pandas fails.

---

The Problem

The user was trying to run a simple Python script (rename.py) that starts with:

import pandas as pd

They first checked what was available:

module spider pandas

This showed several versions:

Versions:
  pandas/2.1.3 (E)
  pandas/2.2.2 (E)
  pandas/2.3.0 (E)
  pandas/2.3.1 (E)

All of them marked with (E), meaning they are extensions provided by another module.

They tried the obvious command:

module load pandas/2.2.2

And got this unhelpful error:

Lmod has detected the following error: These module(s) or extension(s) exist but cannot be loaded as requested: “pandas/2.2.2”

Lmod even gave a hint:

Try: module spider pandas/2.2.2 to see how to load the module(s).

---

The Investigation

Following the advice, we ran:

module spider pandas/2.3.1

This time we got the crucial information:

pandas: pandas/2.3.1 (E)

This extension is provided by the following modules. To access the extension you must load one of the following modules...

SciPy-bundle/2025.07-gfbf-2025b

There it was. Pandas wasn’t a standalone module. It was bundled inside SciPy-bundle/2025.07-gfbf-2025b.

---

The Solution

We simply loaded the providing module:

module load SciPy-bundle/2025.07-gfbf-2025b

After that, pandas became available.

To verify everything was working correctly, we ran these checks:

module list                          # See what's loaded
which python                         # Confirm we're using the right Python
python -c "import pandas as pd; print(pd.__version__)"

Once the version printed successfully, the original script worked:

python rename.py

---

Why Did This Happen?

On modern HPC systems using Lmod + EasyBuild, many Python packages (especially scientific ones) are not installed as independent modules. Instead, they are packaged into larger bundles such as:

• SciPy-bundle
• Python-bundle-PyPI
• Python

These bundles install a complete, consistent scientific Python environment (NumPy, SciPy, pandas, matplotlib, etc.) that has been tested together.

When you see (E) next to a module name, it almost always means:
“This package lives inside another module. Load the parent first.”

Directly loading the extension usually fails until its parent is loaded.

---

Lessons Learned & Best Practices

Here’s what I recommend the next time you hit this situation:

1. Always use module spider when something won’t load

   module spider package_name/version

2. Look for the line that says
   "This extension is provided by the following modules..."
3. Load the parent module, not the extension directly.
4. Verify your environment after loading:

   module list
   which python
   python -c "import pandas as pd; print('pandas:', pd.__version__)"

5. Check for deprecation warnings (like the one about foss/2023b and GCC 13.2.0 in this session). Newer bundles are often better maintained.
6. If you’re unsure which bundle to use, try the newest SciPy-bundle available — they usually contain the latest compatible versions of pandas, numpy, etc.

---

Final Working Commands (Summary)

# 1. Find out what provides pandas
module spider pandas/2.3.1

# 2. Load the providing bundle
module load SciPy-bundle/2025.07-gfbf-2025b

# 3. Verify
python -c "import pandas as pd; print(pd.__version__)"

# 4. Run your script
python rename.py

---

Conclusion

What looked like a broken pandas installation was actually just a misunderstanding of how Lmod extensions work on this cluster.

The real bug wasn’t in pandas — it was in the assumption that every package can be loaded independently.

Next time you see that (E) next to a module name, remember: don’t load the extension, load the bundle that contains it.

Have you run into similar issues with other packages (matplotlib, scikit-learn, torch, etc.) on your HPC? Drop a comment — these bundle-based setups are very common and the same debugging steps usually apply.

---