EXXACT server usage (Neural Networks) - Revision history

Admin boerj2 at 18:33, 19 March 2024

2024-03-19T18:33:01Z

← Older revision		Revision as of 14:33, 19 March 2024
Line 109:		Line 109:
	</syntaxhighlight>		</syntaxhighlight>

	When the shell script is used, adjust your string as follows:<syntaxhighlight>		When the shell script is used, adjust your string as follows:<syntaxhighlight lang="bash">
	#Docker Tensorflow		#Docker Tensorflow
	$ docker run -d --rm --name [MacID] -u $(id -u):$(id -g) --gpus all -e USERID:$UID -v /~~data~~/nnuser/[MacID]/:/data tensorflow/tensorflow:latest-gpu /bin/bash /data/entrypoint.sh		$ docker run -d --rm --name [MacID] -u $(id -u):$(id -g) --gpus all -e USERID:$UID -v /home/nnuser/[MacID]/:/data tensorflow/tensorflow:latest-gpu /bin/bash /data/entrypoint.sh
	</syntaxhighlight><syntaxhighlight>		</syntaxhighlight><syntaxhighlight lang="bash">
	#Docker Pytorch		#Docker Pytorch
	$ docker run -d --rm --name [MacID] -u $(id -u):$(id -g) --gpus all -e USERID:$UID -v /~~data~~/nnuser/[MacID]/:/data pytorch/pytorch:latest /bin/bash /data/entrypoint.sh		$ docker run -d --rm --name [MacID] -u $(id -u):$(id -g) --gpus all -e USERID:$UID -v /home/nnuser/[MacID]/:/data pytorch/pytorch:latest /bin/bash /data/entrypoint.sh

	</syntaxhighlight>		</syntaxhighlight>

Admin boerj2 at 18:31, 19 March 2024

2024-03-19T18:31:53Z

← Older revision		Revision as of 14:31, 19 March 2024
Line 30:		Line 30:
	## Tensorflow (https://hub.docker.com/r/tensorflow/tensorflow)		## Tensorflow (https://hub.docker.com/r/tensorflow/tensorflow)
	## PyTorch (https://hub.docker.com/r/pytorch/pytorch)		## PyTorch (https://hub.docker.com/r/pytorch/pytorch)
	# For TensorFlow: Create a docker image with the following command and let it run: <syntaxhighlight line="1" start="1">		# For TensorFlow: Create a docker image with the following command and let it run: <syntaxhighlight line="1" start="1" lang="bash">
	$ docker run -d --rm --name [MacID] --gpus all -u $(id -u):$(id -g) -v /~~data~~/nnuser/[MacID]/:/data tensorflow/tensorflow:latest-gpu python /data/main.py		$ docker run -d --rm --name [MacID] --gpus all -u $(id -u):$(id -g) -v /home/nnuser/[MacID]/:/data tensorflow/tensorflow:latest-gpu python /data/main.py

	</syntaxhighlight>This will:		</syntaxhighlight>This will:
Line 39:		Line 39:
	## Run your script with your custom code until it exits (python 3 /data/main.py) *''Note: You should replace main.py with your main executable function if it was '''not''' named main.py''		## Run your script with your custom code until it exits (python 3 /data/main.py) *''Note: You should replace main.py with your main executable function if it was '''not''' named main.py''
	## Use GPU acceleration and the latest TensorFlow version		## Use GPU acceleration and the latest TensorFlow version
	# For PyTorch: Create a docker image with the following command and let it run: <syntaxhighlight line="1" start="1">		# For PyTorch: Create a docker image with the following command and let it run: <syntaxhighlight line="1" start="1" lang="bash">
	$ docker run -d --rm --name [MacID] --gpus all -u $(id -u):$(id -g) -v /~~data~~/nnuser/[MacID]/:/data pytorch/pytorch:latest python /data/main.py		$ docker run -d --rm --name [MacID] --gpus all -u $(id -u):$(id -g) -v /home/nnuser/[MacID]/:/data pytorch/pytorch:latest python /data/main.py
	</syntaxhighlight>This will:		</syntaxhighlight>This will:
	## Create an image with your MacID as the name (--name [MacID])		## Create an image with your MacID as the name (--name [MacID])

Xu518: VPN link updated

2024-03-06T19:21:00Z

VPN link updated

← Older revision		Revision as of 15:21, 6 March 2024
Line 1:		Line 1:
	<blockquote>''In all instances, please note that docker will be used for training''		<blockquote>''In all instances, please note that docker will be used for training''

	''A [https://uts.mcmaster.ca/services/computers-printers-and-software/virtual-private-~~networks-vpn~~/ VPN] connection is required to access this machine.''		''A [https://uts.mcmaster.ca/services/computers-printers-and-software/virtual-private-networking/ VPN] connection is required to access this machine.''

	''Booking of this resource for extended periods is required at: [https://outlook.office365.com/owa/calendar/SEPTNNTrainingEXXactServer@mcmaster.ca/bookings/ SEPT NN Training (EXXact)]''</blockquote>		''Booking of this resource for extended periods is required at: [https://outlook.office365.com/owa/calendar/SEPTNNTrainingEXXactServer@mcmaster.ca/bookings/ SEPT NN Training (EXXact)]''</blockquote>

Xu518: Created page with "
''In all instances, please note that docker will be used for training'' ''A [https://uts.mcmaster.ca/services/computers-printers-and-software/virtual-private-networks-vpn/ VPN] connection is required to access this machine.'' ''Booking of this resource for extended periods is required at: [https://outlook.office365.com/owa/calendar/SEPTNNTrainingEXXactServer@mcmaster.ca/bookings/ SEPT NN Training (EXXact)]''
=== Code Editors === It is highly s..."

2023-10-04T21:09:26Z

Created page with "<blockquote>''In all instances, please note that docker will be used for training'' ''A [https://uts.mcmaster.ca/services/computers-printers-and-software/virtual-private-networks-vpn/ VPN] connection is required to access this machine.'' ''Booking of this resource for extended periods is required at: [https://outlook.office365.com/owa/calendar/SEPTNNTrainingEXXactServer@mcmaster.ca/bookings/ SEPT NN Training (EXXact)]''</blockquote> === Code Editors === It is highly s..."

New page

<blockquote>''In all instances, please note that docker will be used for training''

''A [https://uts.mcmaster.ca/services/computers-printers-and-software/virtual-private-networks-vpn/ VPN] connection is required to access this machine.''

''Booking of this resource for extended periods is required at: [https://outlook.office365.com/owa/calendar/SEPTNNTrainingEXXactServer@mcmaster.ca/bookings/ SEPT NN Training (EXXact)]''</blockquote>

=== Code Editors ===
It is highly suggested that a code editor that integrates into remote systems is used for development alongside git.

Common IDE with remote SSH suggestions would include:

* [https://code.visualstudio.com/docs/remote/ssh Visual Studio Code (VS Code)]
* [https://docs.microsoft.com/en-us/cpp/linux/connect-to-your-remote-linux-computer?view=msvc-160 Visual Studio Community]
* [https://www.jetbrains.com/help/pycharm/configuring-remote-interpreters-via-ssh.html#ssh Jetbrains PyCharm]

Common git hosts would include:

* [https://education.github.com/ GitHub]
* [https://gitlab.com/users/sign_in GitLab]
* [https://datadev.sept.mcmaster.ca/users/sign_in SEPT Local GitLab integration]

=== Creating and running a container for your code ===

# Log into Exxact server as '''nnuser''' using a ssh client, i.e. [https://www.putty.org/ Putty], [https://github.com/PowerShell/PowerShell#get-powershell Powershell], etc. with ''nnuser@130.113.129.242'' and password ''Exxact1.''
# Create a folder with YOUR MacID using the command mkdir [MacID]
# Change directory into your directory using cd ./[MacID]. Do not use or override another users directory.
# Copy your data and python script and data into your directory using an SFTP client, i.e. FileZilla, WinSCP etc.
# Choose the trainer of choice
## Custom – A docker image you created yourself
## Tensorflow (https://hub.docker.com/r/tensorflow/tensorflow)
## PyTorch (https://hub.docker.com/r/pytorch/pytorch)
# For TensorFlow: Create a docker image with the following command and let it run: <syntaxhighlight line="1" start="1">
$ docker run -d --rm --name [MacID] --gpus all -u $(id -u):$(id -g) -v /data/nnuser/[MacID]/:/data tensorflow/tensorflow:latest-gpu python /data/main.py

</syntaxhighlight>This will:
## Create an image with your MacID as the name (--name [MacID])
## Remove the image once its done (--rm should be removed while testing and getting your files setup)
## Allow you access to the files that you saved (--user $UID)
## Run your script with your custom code until it exits (python 3 /data/main.py) *''Note: You should replace main.py with your main executable function if it was '''not''' named main.py''
## Use GPU acceleration and the latest TensorFlow version
# For PyTorch: Create a docker image with the following command and let it run: <syntaxhighlight line="1" start="1">
$ docker run -d --rm --name [MacID] --gpus all -u $(id -u):$(id -g) -v /data/nnuser/[MacID]/:/data pytorch/pytorch:latest python /data/main.py
</syntaxhighlight>This will:
## Create an image with your MacID as the name (--name [MacID])
## Remove the image once its done (--rm should be removed while testing and getting your files setup)
## Allow you access to the files that you saved (--user $UID)
## Run your script with your custom code until it exits (python 3 /data/main.py) *''Note: You should replace main.py with your main executable function if it was '''not''' named main.py''
## Use GPU acceleration and the latest PyTorch version

'''NOTE''''': It is highly recommended that you write your python code output to console as this will get both logged and allow you to see what your script is doing while running during training.''

=== Managing your container ===

==== When running: ====

# View your container at 130.113.129.242:9000 with username ''nnuser'' and password ''Exxact1''
# Click Containers: [[File:EXXACT pic1.png]]
# Find your container: [[File:EXXACT pic2.png]]
# Attach to see output: [[File:EXXACT pic3.png]]
# View Output [[File:EXXACT pic4.png]]

==== When testing (container does not include --rm statement): ====

# View your container at <nowiki>https://130.113.129.242:9443</nowiki> with username ''nnuser'' and password ''Exxact1''
# Click Containers: [[File:EXXACT pic1.png]]
# Find your container: [[File:EXXACT pic2.png]]
# Logs to see output: [[File:EXXACT pic7.png]]
# View Output: This will show errors etc. [[File:EXXACT pic8.png]]

=== IMPORTANT NOTES ===

# Save your OUTPUT of your model to your '''/data''' NOTHING will get saved that is placed ANYWHERE else
# Remove your files once done. There is limited space on the server.
# Respect other users training. This is a shared resource. Do not stop or remove their containers without permission

=== Scripts for Import - Custom requirements ===
If your script requires any imports that is non-standard to a package, you will need to adjust your execution string accordingly. <blockquote>'''NOTE''': This script requires that you finish with your training to get access to any produced files.</blockquote>You will need to create the following:

* a requirements.txt file within your python environment
** <syntaxhighlight line="1">
#Google Colab
!pip freeze > requirements.txt
</syntaxhighlight><syntaxhighlight>
#Local Python env.
pip freeze > requirements.txt
</syntaxhighlight>
* a shell script that installs the imports and runs the script. This can be copied.
** <syntaxhighlight line="1" start="1">
#entrypoint.sh (Python 3)

#!/bin/bash
python3 -m pip install requests
python3 -m pip install PIL
pip install --upgrade pip
pip install /data/requirements.txt
python3 /data/main.py
chown $USERID:$USERID -R /data
</syntaxhighlight><syntaxhighlight>
#entrypoint.sh (Python 2)

#!/bin/bash
python -m pip install requests
python -m pip install PIL
pip install --upgrade pip
pip install /data/requirements.txt
python /data/main.py
chown $USERID:$USERID -R /data
</syntaxhighlight>

When the shell script is used, adjust your string as follows:<syntaxhighlight>
#Docker Tensorflow
$ docker run -d --rm --name [MacID] -u $(id -u):$(id -g) --gpus all -e USERID:$UID -v /data/nnuser/[MacID]/:/data tensorflow/tensorflow:latest-gpu /bin/bash /data/entrypoint.sh
</syntaxhighlight><syntaxhighlight>
#Docker Pytorch
$ docker run -d --rm --name [MacID] -u $(id -u):$(id -g) --gpus all -e USERID:$UID -v /data/nnuser/[MacID]/:/data pytorch/pytorch:latest /bin/bash /data/entrypoint.sh

</syntaxhighlight>