ObjectNet

ObjectNet Challenge Documentation

View the Project on GitHub abarbu/objectnet-challenge-doc-ibm-dev

ObjectNet logo

Challenge Portal

Visit the ObjectNet Challenge Portal to register for the challenge.

ObjectNet Support

Experiencing a problem or just have a general question, see ObjectNet Support

ObjectNet Challenge:

Creating your Docker image from scratch

These instructions describe how to build a generic docker image for the ObjectNet Challenge.

If your model is built using once of the following frameworks, go to the relevant documentation:

These instructions are split into two sections:

Note: Your inference code should use batching and parallel data loading to improve efficiency. If you are building your own customized docker image with your own code it is highly recommended to use optimized inferencing techniques to ensure your submission will complete within the time limit set by the challenge organisers.

Section 1: Creating a Dockerfile for your model and code

A Dockerfile is used to dockerise your model and code. The Dockerfile describes the dependencies required to build the Docker image which are encapsulated within the image when it is built.

For more information on creating a Dockerfile visit dockerfile best practices, or the Dockerfile reference.

Below is an example Dockerfile for a model built using TensorFlow with associated python code used for inference:

# Build arguments to control base tensorflow image. Tensorflow build
# tags for docker images can be found at:
# https://hub.docker.com/r/tensorflow/tensorflow/tags
#
# From the command line (`docker build...`) use:
#  --build-arg GPU: omit or "-gpu" to build a GPU enabled image,
#                   or "" to build non GPU image.
#  --build-arg TENSORFLOW_VERSION: omit to use "2.3.0", otherwise
#                   specify version.
#  Default is to use: tensorflow/tensorflow:2.3.0-gpu

ARG GPU="-gpu"
ARG TENSORFLOW_VERSION="2.3.0"

# Build based on official TensorFlow docker images available on docker hub:
#   https://www.tensorflow.org/install/docker

FROM tensorflow/tensorflow:${TENSORFLOW_VERSION}${GPU}

# Need to redefine the ARGS here so we can print them out
ARG GPU
ARG TENSORFLOW_VERSION


RUN echo "Building from tensorflow image: tensorflow/tensorflow:$TENSORFLOW_VERSION$GPU"

# Add metadata
LABEL maintainer="your-email-address"
LABEL version="0.1"
LABEL description="Docker image for ObjectNet AI Challenge."

# Set working directory
WORKDIR /workspace

# Install python packages listed in requirements.txt
COPY requirements.txt /workspace
RUN pip install -r requirements.txt

# Copy (recursively) all files from the current directory into the
# image at /workspace
COPY . /workspace

# Define the command to execute when the container is run
ENTRYPOINT ["python","./app.py"]

The remaining parts of this section will go through each line in the example and explain its purpose.

ARG (Define docker build arguments)

#  Default is to use: tensorflow/tensorflow:2.3.0-gpu
ARG GPU="-gpu"
ARG TENSORFLOW_VERSION="2.3.0"

The ARG instruction block defines 2 docker build arguments. These build arguments are used in the Dockerfile to parameterise build steps. In this case, the build arguments are used to paramaterise the FROM instruction which defines the base image to build on. The values hardcoded into the Dockerfile act as defaults. These defaults can be overridden by providing arguments to the docker build command. For example:

$ docker build --build-arg TENSORFLOW_VERSION=2.2.1 --build-arg GPU="-gpu" ....

FROM (Pull from a base image)

# Build based on official TensorFlow docker images available on docker hub:
#   https://www.tensorflow.org/install/docker
FROM tensorflow/tensorflow:${TENSORFLOW_VERSION}${GPU}

The FROM command establishes which existing Docker image your build will be based upon. In this case, the official tensorflow images are being used. The tensorflow version along with GPU support are parameterised.

Whenever possible, use the current Official Repositories as the basis for your customised image - see Other Docker resources for more information.

ARG + RUN (Print docker base image)

# Need to redefine the ARGS here so we can print them out
ARG GPU
ARG TENSORFLOW_VERSION


RUN echo "Building from tensorflow image: tensorflow/tensorflow:$TENSORFLOW_VERSION$GPU"

The build arguments must be redefined before they can be used elsewhere in the Dockerfile. In this case the build arguments are used in a RUN statement which prints out an information message indicating the exact image used as the base of the build. The block of code is not essential but is useful to understand the exact version of the image used in the FROM instruction.

LABEL (Add metadata to an image)

# Add metadata
LABEL maintainer="your-email-address"
LABEL version="0.1"
LABEL description="Docker image for ObjectNet AI Challenge."

Adds metadata to the image specifying the maintainer, version and description.

WORKDIR (Set working directory)

# Set working directory
WORKDIR /workspace

The WORKDIR instruction sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile

COPY & RUN (Install dependencies)

# Install python packages listed in requirements.txt
COPY requirements.txt /workspace
RUN pip install -r requirements.txt

Copies the requirements.txt file from the current directory into the Docker image, and then uses the RUN command to do a pip install (within the image) of the packages listed in the file. Essentially, this step ensures that any python library dependency listed in the requirements.txt file are installed into the docker image whilst it is being built.

COPY (Transfer local files into Docker image)

# Copy (recursively) all files from the current directory into the
# image at /workspace
COPY . /workspace

Copies all the files from the current directory (and any child directories) into the image being built at /workspace. This COPY step ensures all the model and executable files from the current working directory are copied into the image.

ENTRYPOINT (Make your Docker container executable)

# Define the command to execute when the container is run
ENTRYPOINT ["python","./app.py"]

ENTRYPOINT specifies the command to execute when your docker container is run. In this example, we are running the ./app.py python script.

When building your docker container you will need to modify this property to run your code which performs the inferences against the ObjectNet image set and produces the /output/predictions.csv file containing your predictions.

Section 2: Building and testing your docker image

This section guides you through the process of building a Docker image from your Dockerfile, and subsequently testing the image. You will need:

2.1 Install the Docker engine

To build and test a docker image locally you will first need to install the docker engine. Follow the instructions on installing docker, along with a quick start guide.

2.2 Installing the NVIDIA drivers

If your local machine has NVIDIA-capable GPUs and you want to test your docker image locally using these GPUs then you will need to ensure the NVIDIA drivers have been installed on your test machine.

Instructions on how to install the CUDA toolkit and NVIDIA drivers can be found here. Be sure to match the versions of CUDA/NVIDIA installed with the version of the deep learning framework and CUDA used to build your docker image.

2.3 Set up your working directory

In this step you colocate your model, associated code and newly created Dockerfile:

2.4 Build your Docker image

You can now build a Docker image (e.g. named "my-model") using the following command (note the fullstop at the end of the command):

# syntax: docker build -t <image-name>:<Tag> <Dockerfile path>
# With version tagging:
$ docker build -t  my-model:version1 .

# Or without version tagging:
$ docker build -t  my-model .

Where:

Note: When your docker image is submitted to the challenge for evaluation the image will not have internet access and as such will not be able to download model checkpoints from the internet. For this reason it is essential that your model is included in the built docker image.

Note: Docker images submitted to the ObjectNet Challenge must be based on a GPU enabled base image and use GPUs for inferencing.

2.5 Testing your Docker image

Once your docker image has been successfully built you need to run the image and validate that the output conforms to the expected format.

2.5.1 Clone the testing repository

First, clone the repo which contains the files required to test and validate your predictions:

$ git clone https://github.com/abarbu/objectnet-template-generic.git

This repo comes with a python script to validate and score the inferences (./validate_and_score.py) output from your model, as well as a set of test images (input/images) and a file containing ground truth data for those images (input/answers/answers-test.json).

2.5.2 Run the Docker image

Run your Docker image according to the Docker run guide. Ensure you mount the input directory using -v $PWD/input/images:/input/images/ and the output directory using -v $PWD/output:/output. The mounts will ensure your model uses the test images downloaded from git and stores the results in the output folder within the current directory.

For example:

$ docker run -ti --rm --gpus=all -v $PWD/input/images:/input/images -v $PWD/output:/output my-model:version1

**** params ****
images /input
output_file /output/predictions.csv
model_class_name resnext101_32x48d_wsl
model_checkpoint /workspace/model/ig_resnext101_32x48-3e41cc8a.pth
workers 16
gpus 2
batch_size 96
softmax True
convert_outputs_mode 1
****************

initializing model ...
loading pretrained weights from disk ...
Done. Number of predictions:  10

Add the --gpus=all parameter to the docker run command in order to utilise your GPUs (if available).

A successful run will result in a predicitions.csv file written to the $PWD/output path.

2.5.3 Debugging your docker image locally

If there are errors during the previous step then you will need to debug your docker container. When making changes to your code there is no need to rebuild the docker container. To quickly test your new code, simply mount the root path of your working directory as a volume when you run the container. For example:

$ docker run -ti --rm --gpus=all -v $PWD:/workspace -v $PWD/input/images:/input/ -v $PWD/output:/output --entrypoint /bin/bash my-model:version1

When the docker container is run, the local $PWD will be mounted over /workspace directory within the docker image which effectively means any code/model changes made since the last docker build command will be contained within the running container.

Important: Once you have debugged your code and the docker image runs successfully you will need to re-build the docker image.

2.5.4 Validate the predictions

In order to ensure that the predictions made by your model (and stored in $PWD/output/predictions.csv file) are structured according to the ObjectNet Challenge specifications, it is important to validate them using the validate_and_score.py script.

Move into the objectnet-template-generic directory created when you cloned the git repo in 2.5.1 Clone the testing repository

$ cd objectnet-template-generic

Then, run the following command to validate your model's output:

$ python validate_and_score.py -r -a input/answers/answers-test.json -f <path-to-predictions.csv-file>
{
  "accuracy": 20.0,
  "images_scored": 10,
  "prediction_file_errors": [],
  "prediction_file_status": "VALIDATED",
  "top5_accuracy": 30.0,
  "total_images": 10
}

Note the usage of the -a and -f flags as specified in validate_and_score.py --help below.

usage: validate_and_score.py [-h] --answers ANSWERS --filename FILENAME
                             [--range_check]

optional arguments:
  -h, --help            show this help message and exit
  --answers ANSWERS, -a ANSWERS
                        ground truth/answer file
  --filename FILENAME, -f FILENAME
                        users result file
  --range_check, -r     reject entries that have out-of-range label indices

A working Docker image which produces valid output should return "prediction_file_status": "VALIDATED".

A correctly formatted predictions.csv file will result in an output of "prediction_file_status": "VALIDATED". If you received an error when running this command ensure that you have entered the correct file locations for the answer file as well as the result file. For clarification on result file structure refer to Model prediction format

Once the output of your model has been validated, you are ready to submit the docker image to the ObjectNet Challenge .