Environment Manager and Python Package Management¶
Conda vs. Mamba vs. Venv¶
In [ ]:
Copied!
%%bash
# user guide https://mamba.readthedocs.io/en/latest/user_guide/mamba.html
# Conda provides a complete package management solution that handles not just
# python packages but also system-level dependencies.
# mamba is just a faster conda, which uses the conda repos but simply faster.
# Here is tutorial for mamba: https://statistics.berkeley.edu/computing/conda
pip install transformers numpy torch
mamba info
# Prioritize using mamba > conda> virtualenv > venv
%%bash
# user guide https://mamba.readthedocs.io/en/latest/user_guide/mamba.html
# Conda provides a complete package management solution that handles not just
# python packages but also system-level dependencies.
# mamba is just a faster conda, which uses the conda repos but simply faster.
# Here is tutorial for mamba: https://statistics.berkeley.edu/computing/conda
pip install transformers numpy torch
mamba info
# Prioritize using mamba > conda> virtualenv > venv
Requirement already satisfied: transformers in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (4.48.0)
Requirement already satisfied: numpy in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (2.2.1)
Requirement already satisfied: torch in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (2.5.1)
Requirement already satisfied: filelock in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from transformers) (3.16.1)
Requirement already satisfied: huggingface-hub<1.0,>=0.24.0 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from transformers) (0.27.1)
Requirement already satisfied: packaging>=20.0 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from transformers) (24.2)
Requirement already satisfied: pyyaml>=5.1 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from transformers) (6.0.2)
Requirement already satisfied: regex!=2019.12.17 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from transformers) (2024.11.6)
Requirement already satisfied: requests in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from transformers) (2.32.3)
Requirement already satisfied: tokenizers<0.22,>=0.21 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from transformers) (0.21.0)
Requirement already satisfied: safetensors>=0.4.1 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from transformers) (0.5.2)
Requirement already satisfied: tqdm>=4.27 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from transformers) (4.67.1)
Requirement already satisfied: typing-extensions>=4.8.0 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from torch) (4.12.2)
Requirement already satisfied: networkx in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from torch) (3.4.2)
Requirement already satisfied: jinja2 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from torch) (3.1.5)
Requirement already satisfied: fsspec in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from torch) (2024.9.0)
Requirement already satisfied: sympy==1.13.1 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from torch) (1.13.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from sympy==1.13.1->torch) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from jinja2->torch) (3.0.2)
Requirement already satisfied: charset_normalizer<4,>=2 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from requests->transformers) (3.4.1)
Requirement already satisfied: idna<4,>=2.5 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from requests->transformers) (3.10)
Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from requests->transformers) (2.3.0)
Requirement already satisfied: certifi>=2017.4.17 in /Users/jcao/mamba/envs/cs5293-1/lib/python3.10/site-packages (from requests->transformers) (2024.12.14)
libmamba version : 2.0.5
mamba version : 2.0.5
curl version : libcurl/8.7.1 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.63.0
libarchive version : libarchive 3.7.7 zlib/1.2.12 liblzma/5.6.3 bz2lib/1.0.8 liblz4/1.10.0 libzstd/1.5.6
envs directories : /Users/jcao/mamba/envs
package cache : /Users/jcao/mamba/pkgs
/Users/jcao/.mamba/pkgs
environment : cs5293-1 (active)
env location : /Users/jcao/mamba/envs/cs5293-1
user config files : /Users/jcao/.mambarc
populated config files :
virtual packages : __unix=0=0
__osx=15.2=0
__archspec=1=arm64
channels : https://conda.anaconda.org/conda-forge/noarch
https://conda.anaconda.org/conda-forge/osx-arm64
base environment : /Users/jcao/mamba
platform : osx-arm64
The Difference Among UV, Poetry, PIP, Conda, and virtualenv¶
Please read https://www.datacamp.com/tutorial/python-uv
Use Conda in Google Colab (ignoring this if you are running this on local environment)¶
If you don't have a laptop, and a tablet or phone is you have, or you only want to use a browser for coding this course. Then you should consider using Google Colab. Here is a tutorial on using Conda on Google Colab. https://colab.research.google.com/drive/1c_RGCgQeLHVXlF44LyOFjfUW32CmG6BP
Key points:
- On Google Colab, you only can use
baseenvironment in google colab even if you create a new environment. NO WORKAROUND SO FAR!!! - However, you could update the base environment with new packages, including python version, and other pacakges.
In [9]:
Copied!
# If you plan to use google colab do the following three lines for conda setup.
!pip install -q condacolab
# If you plan to use google colab do the following three lines for conda setup.
!pip install -q condacolab
In [ ]:
Copied!
import condacolab
condacolab.install()
# you may need to restart your runtime kernel to make it take effect!
import condacolab
condacolab.install()
# you may need to restart your runtime kernel to make it take effect!
Check the Conda for Google Colab Setup¶
In [ ]:
Copied!
import condacolab
condacolab.check()
import condacolab
condacolab.check()
Pip (Using pip to install all python related libraries)¶
Introducing Jupyter NoteBook¶
In a Jupyter Notebook, the "!" symbol is used to run a shell command directly from within a code cell, while the "%" symbol is used to execute "magic commands" which are special functions provided by the IPython kernel, allowing for more advanced operations like controlling cell behavior or manipulating data within the notebook itself. https://opensarlab-docs.asf.alaska.edu/user-guides/jupyter_magic/#:~:text=In%20IPython%20syntax%2C%20the%20exclamation,the%20command%20in%20the%20shell.
%%Cell magics start with %% and affect the contents of an entire cell.
In [ ]:
Copied!
%%bash
ls -l
whoami
%%bash
ls -l
whoami
total 128 -rw-r--r--@ 1 jcao staff 24783 Jan 19 18:02 Assignment_1.ipynb -rw-r--r--@ 1 jcao staff 10968 Jan 18 10:28 UsingWordNet.ipynb -rw-r--r--@ 1 jcao staff 8719 Jan 19 18:42 get_bert_vocab.ipynb -rw-r--r--@ 1 jcao staff 9045 Jan 19 14:41 setup_tutorial.ipynb jcao
However, when we just execute a single line of command. We could just ! to start that command line. In this tutorial, we use both %%bash or ! for running linux commands.
In [ ]:
Copied!
!ls -hl
!ls -hl
total 128 -rw-r--r--@ 1 jcao staff 24K Jan 19 18:02 Assignment_1.ipynb -rw-r--r--@ 1 jcao staff 11K Jan 18 10:28 UsingWordNet.ipynb -rw-r--r--@ 1 jcao staff 8.5K Jan 19 18:42 get_bert_vocab.ipynb -rw-r--r--@ 1 jcao staff 8.8K Jan 19 14:41 setup_tutorial.ipynb
In [ ]:
Copied!
%%bash
# In a terminal, you could run a Python file(.py) directly
python src/python_tutorial/hello_world.py
%%bash
# In a terminal, you could run a Python file(.py) directly
python src/python_tutorial/hello_world.py
@global is initialized the file is run or load: Hello, World! Main functon Base Init Base Init ================================================ BeforeSay:1737316319.656283 using are using passArg_say: Hello CS-5293 guys! AfterSay: 1737316319.656296 ================================================ ================================================ BeforeSay:1737316319.6563 using are using passArg_say for 3 times (case_senstive=False): hello cs-5293 students! hello cs-5293 students! hello cs-5293 students! AfterSay: 1737316319.656306 ================================================ ================================================ BeforeSay:1737316319.656309 using are using callFunc_say for 3 times (case_senstive=False): @say_hello: Hello, world! @say_hello: Hello, world! @say_hello: Hello, world! AfterSay: 1737316319.656314 ================================================
In [ ]:
Copied!
%%bash
# Alternatively, in Python, every single Python file(.py) is a Python "Module". You could also use -m swith to call a module directly.
python -m src.python_tutorial.hello_world
%%bash
# Alternatively, in Python, every single Python file(.py) is a Python "Module". You could also use -m swith to call a module directly.
python -m src.python_tutorial.hello_world
current python path is: /Users/jcao/git-workspace/assignment-1 /Users/jcao/mamba/envs/cs5293-0/lib/python310.zip /Users/jcao/mamba/envs/cs5293-0/lib/python3.10 /Users/jcao/mamba/envs/cs5293-0/lib/python3.10/lib-dynload /Users/jcao/mamba/envs/cs5293-0/lib/python3.10/site-packages __init__.py is used to better organize code and to run any initialization that should be run automatically when the package is imported @global is initialized the file is run or load: Hello, World! Main functon Base Init Base Init ================================================ BeforeSay:1737317857.907497 using are using passArg_say: Hello CS-5293 guys! AfterSay: 1737317857.907507 ================================================ ================================================ BeforeSay:1737317857.90751 using are using passArg_say for 3 times (case_senstive=False): hello cs-5293 students! hello cs-5293 students! hello cs-5293 students! AfterSay: 1737317857.907514 ================================================ ================================================ BeforeSay:1737317857.907516 using are using callFunc_say for 3 times (case_senstive=False): @say_hello: hello, world! @say_hello: hello, world! @say_hello: hello, world! AfterSay: 1737317857.907519 ================================================
However, you could not directly call a package, where a python package a collection of modules.
Pythonic Python¶
Please try to learn by youself about these free resourse on Pythonic Python: https://builtin.com/data-science/pythonic#:~:text=Pythonic%20is%20a%20term%20used,and%20not%20deciphering%20existing%20code.
It is your design choices on how to design python modules, packages, subpackages. But it is often better to put the reuseable code into ".py" file and put them out of the jupyter notebook. Furthermore, to distinguish your own reuseable code with external libraries. It is recommended to alwasy keep a "src" folder for your own modules. Google Python Guideline: https://google.github.io/styleguide/pyguide.html Then you could use the designed modules/packages/subpackages in the jupyter notebooks.
In [ ]:
Copied!
import sys
# explictly to add your local src into the PYTHONPATH
sys.path.append("src")
from python_tutorial import hello_world
print(f"The global variable 'case_sensitive' will be imported to the notebook, {hello_world.case_sensitive}")
print(f"The global func 'say_helloworld' will be imported to the notebook, {hello_world.hello_str(case_sensitive=False)}")
import sys
# explictly to add your local src into the PYTHONPATH
sys.path.append("src")
from python_tutorial import hello_world
print(f"The global variable 'case_sensitive' will be imported to the notebook, {hello_world.case_sensitive}")
print(f"The global func 'say_helloworld' will be imported to the notebook, {hello_world.hello_str(case_sensitive=False)}")
The global variable 'case_sensitive' will be imported to the notebook, True The global func 'say_helloworld' will be imported to the notebook, hello
In [ ]:
Copied!
%%bash
# Alternatively, in Python, every single Python file(.py) is a Python "Module". You could also use -m swith to call a module directly.
python -m src
python -m src.python_tutorial
python -m src.python_tutorial.hello_world
%%bash
# Alternatively, in Python, every single Python file(.py) is a Python "Module". You could also use -m swith to call a module directly.
python -m src
python -m src.python_tutorial
python -m src.python_tutorial.hello_world
/Users/jcao/mamba/envs/cs5293-0/bin/python: No module named src.__main__; 'src' is a package and cannot be directly executed /Users/jcao/mamba/envs/cs5293-0/bin/python: No module named src.python_tutorial.__main__; 'src.python_tutorial' is a package and cannot be directly executed
__init__.py is used to better organize code and to run any initialization that should be run automatically when the package is imported __init__.py is used to better organize code and to run any initialization that should be run automatically when the package is imported @global is initialized the file is run or load: Hello, World! Main functon Base Init Base Init ================================================ BeforeSay:1737316953.509111 using are using passArg_say: Hello CS-5293 guys! AfterSay: 1737316953.509123 ================================================ ================================================ BeforeSay:1737316953.509126 using are using passArg_say for 3 times (case_senstive=False): hello cs-5293 students! hello cs-5293 students! hello cs-5293 students! AfterSay: 1737316953.50913 ================================================ ================================================ BeforeSay:1737316953.509133 using are using callFunc_say for 3 times (case_senstive=False): @say_hello: hello, world! @say_hello: hello, world! @say_hello: hello, world! AfterSay: 1737316953.509136 ================================================