When streaming video content over the internet, the visual quality is degrades due to the compression being applied to efficiently stream video over the internet.
With animated videos, the artifacts created by this process are especially distracting so as a fan of anime and animation in general, I looked for a solution. I’m also interested in learning ML so the solution described below was a perfect 2 birds 1 stone scenario. This was a fun project but it took quite a bit of effort to figure out. I’ve documented my failures along the way so that these mistakes can be avoided by anyone else interested in using this awesome technology!
I started with searching YouTube for ideas. Pretty quickly, I came across a video comparing the popular ML models commonly used to upscale anime. (Link) The same channel also had a video with instructions on setting up an all-in-one Docker container that contains the latest upscaling models we can use for this project. The docker image method ended up being the easiest way to get the upscaling done without having a deep understanding of what is happening under the hood. The instructions are really easy to follow if you have the appropriate hardware. (Link)
Below are the attempts I made to get this working. The first attempt failed, the second worked but had its limitations , and the last worked perfectly.
Attempt 1 – Personal Hardware – Failed
Hardware Specs:
Server: Dell R510
OS: VMware ESXi hosted Window 10 Pro VM
CPU: 12 core Intel Xeon E5-2440
RAM: 32 GB
GPU: Nvidia GeForce GT 710 2 GB
Hard Disk: 150 GB
I tried running the docker image on my Dell R510 server.
Unfortunately, getting it to run on my home lab server was challenging and I ended up conceding on the endeavor. After quite a few attempts, I realized my graphics card did not have the memory bandwidth necessary to get the inference working. The requirement was about 4 GB but my GPU maxes out at 2 GB.
Below are the steps taken to get the docker image installed on my setup:
Link to instructions: https://www.youtube.com/watch?v=B134jvhO8yk
Prerequisites:
- Install Docker
- Install Anaconda
- Install Git
- Clone https://github.com/styler00dollar/VSGAN-tensorrt-docker.git
- Build the docker image. This part can take a long time to complete!
Troubleshooting
Code that failed to complete:
vspipe -c y4m inference.py - | ffmpeg -i pipe: <Name of Video>.mkvError:
Error: [0xb01ec0000]:53477376 :DeviceActivationSize in reserveNetworkTensorMemory: at optimizer/common/tactic/optimizer.cpp: 4603 idx: 8 time: 0.001985
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Requested amount of GPU memory (4294967296 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
WARNING: [Torch-TensorRT TorchScript Conversion Context] - Skipping tactic 3 due to insuficient memory on requested size of 4294967296 detected for tactic 5.
Try decreasing the workspace size with IBuilderConfig::setMaxWorkspaceSize().
ERROR: [Torch-TensorRT TorchScript Conversion Context] - 10: [optimizer.cpp::computeCosts::2011] Error Code 10: Internal Error (Could not find any implementation for node %input.337 : Tensor = aten::_convolution(%input.335, %self.model.6.weight, %self.model.6.bias, %4, %4, %4, %6, %3, %7, %5, %6, %5, %5), scope: __module.model/__module.model.6 # /usr/local/lib/python3.8/dist-packages/torch/nn/modules/conv.py:443:0.)Conclusion
It may be possible to get this working on my home setup by decreasing the workspace size for IBuilderConfig::setMaxWorkspaceSize() as mentioned above. Considering this is already a performance intense task, this could end up not working and even if it did, its probably painfully slow. NEXT!!!!
Attempt 2 – Google Colab -Successful
Hardware Specs:
OS: Google Colab
CPU: 6 Core Intel(R) Xeon(R) CPU @ 2.20GHz
RAM: 128 GB
GPU: Nvidia Tesla T4 or Tesla P100
Disk: 150 GB
Cost: $10/Month
I conceded on my first attempt using the docker method on my own hardware. I think that I lack the appropriate hardware to finish the build step of the process. More specifically, I don’t have a compatible Nvidia GPU. In this 2nd attempt, I will be using Google Colab.
The GitHub repo has a prebuilt notebook with all the code that needs to be executed. I will be using a new notebook so that I can learn rather than just run blocks of code.
This option takes the difficulty down several notches because all the code is available and click-to-run. I did however run into some dependency issues.
What is Google Colab?
Colab allows anybody to write and execute arbitrary python code through the browser, and is especially well suited to machine learning (https://research.google.com/colaboratory/faq.html)
Link to prebuilt notebook: https://colab.research.google.com/github/styler00dollar/VSGAN-tensorrt-docker/blob/main/Colab-VSGAN.ipynb#scrollTo=8QneOIbHW9TK
Conclusion
The hardware used on Google Colab is shared so executions timing on is an issue. This makes it difficult to upscale longer videos. Just 5 minutes of a anime can take about an hour to render with ESRGAN so this is not a feasible solution. Next 🙁
Attempt 3 – Lambda Labs HPC/ML Cloud Instance
Hardware Specs:
OS: Ubuntu
CPU: 14 vCPUs AMD EPYC 7H12 64-Core Processor
RAM: 46 GB RAM
GPU: 1 Nvidia A100 (24 GB RAM)
Disk: 512 GB SSD
Cost: $0.50/Hour
Considering we are only able to render 5 minutes an hour using RealESRGAN on the Google Colab runtime, we need a more powerful system to actually complete upscaling an entire movie. Also, even with Colab Pro, we cannot leave the runtime unattended for too long because the resources will be reallocated elsewhere and stop everything in its tracks.
Lambda Labs is a deep learning hardware and cloud provider. On their site, they claim that their instances are much cheaper to rent when compared to AWS and Azure. From my experience, this is true. On both of the major cloud providers, what you get for the cost is much less than what Lambda Labs offers for about the same price. The A100 instance for $.50/hour is perfect for this project. They offer multi-GPU instances but getting the docker container to use multiple GPUs for inference could be a project all on its own. I spent a few hours trying to get this working but no luck yet.
The Lambda instance was easy to work with as well. When you launch the instance from the GUI, it opens a Jupyter Notebook connection to the instance. From here you can run the terminal as well. I recommend using the terminal and the instructions below to get it all setup.
#Create Workspace
sudo mkdir workspace
cd workspace
#Clone the git repo and move into directory
sudo git clone https://github.com/styler00dollar/VSGAN-tensorrt-docker
cd VSGAN-tensorrt-docker
#Pull the Docker Container
sudo docker pull styler00dollar/vsgan_tensorrt:latest
#Build the Docker Container. (Could take a while)
sudo docker build -t styler00dollar/vsgan_tensorrt:latest .
Conclusion
The Lambda Labs instances are very affordable when compared to AWS and Azure but keeping a close eye on usage is important to minimize unneeded costs. The on-demand instances are in high demand so they may be hard to come by. In my experience, the instances are more available late at night, Eastern Time.
Final Conclusion & Total Costs
The total cost came to about $150.
Most of the costs came from brute force attempts to manually install the inference tools on the cloud instances rather that just setting up and using the docker image. I just kept running into exceptions because the instance wasn’t perfectly setup for this project. The docker container is amazing because it just works. Check it out! (Link)
Leave a Reply