QBoard » Big Data » Big Data on Cloud » AssertionError: Multiple .dist-info directories on Data Science Experience
User Dashboard

AssertionError: Multiple .dist-info directories on Data Science Experience

Back To Topics
Tags : None
  • 126

    In a Python 3.5 notebook, backed by an Apache Spark service, I had installed BigDL 0.2 using pip. When removing that installation and trying to install version 0.3 of BigDL, I get this error: (linebreaks added for readability)

    AssertionError: Multiple .dist-info directories:
/gpfs/fs01/user/scbc-4dbab79416a6ec-4cf890276e2b/.local/lib/python3.5/site-packages/BigDL-0.3.0.dist-info,
/gpfs/fs01/user/scbc-4dbab79416a6ec-4cf890276e2b/.local/lib/python3.5/site-packages/BigDL-0.2.0.dist-info

    However, neither of these directories exists:

    !ls -al /gpfs/fs01/user/scbc-4dbab79416a6ec-4cf890276e2b/.local/lib/python3.5/site-packages/
total 0
drwx------ 2 scbc-4dbab79416a6ec-4cf890276e2b users 4096 Nov  8 06:12 .
drwx------ 3 scbc-4dbab79416a6ec-4cf890276e2b users 4096 Nov  8 06:12 ..
      October 23, 2021 4:27 PM IST
    0
  • 317

    I had this issue, too.

    I looked for global build dirs and deleted them. I also had to delete an existing build directory under

    ~/.pip/build

     

    After that the error disappeared
     
     
      October 30, 2021 2:05 PM IST
    0
  • 346 3

    First, you should never run 'sudo pip'.

    If possible you should use your system package manager because it uses GPG signatures to ensure you're not running malicious code.

    Otherwise, try upgrading setuptools:

    easy_install -U setuptools

     

    Alternatively, try:

    pip install --user <somepackage>

     

    This is of course for "global" packages. You should ideally be using virtualenvs.

     
      November 13, 2021 2:19 PM IST
    0
  • 461
    The directory paths in the error message are wrong. The Python 3.5 kernel on DSX specifies a build directory for pip by setting the environment variable PIP_BUILD. The multiple dist-info directories are there:

    !printenv PIP_BUILD ; ls -l $PIP_BUILD/*
/tmp/scbc-4dbab79416a6ec-4cf890276e2b/pip-build
total 0
drwx------ 8 scbc-4dbab79416a6ec-4cf890276e2b users 117 Nov  7 02:02 bigdl
drwx------ 2 scbc-4dbab79416a6ec-4cf890276e2b users 135 Nov  7 02:02 BigDL-0.2.0.dist-info
drwx------ 2 scbc-4dbab79416a6ec-4cf890276e2b users 135 Nov  8 06:12 BigDL-0.3.0.dist-info​


    To fix the problem, remove the build directory:

    !rm -rf $PIP_BUILD


    After that, pip can install the package without problems:

    !pip install --no-dependencies bigdl==0.3
Collecting bigdl==0.3
  Using cached BigDL-0.3.0-py2.py3-none-manylinux1_x86_64.whl
Installing collected packages: bigdl
Successfully installed bigdl​
      October 28, 2021 4:29 PM IST
    0