Getting Started with Weights and Biases
It is arguable the number one job for a data scientist is to conduct experiments and to produce experimental results. Weights and Biases (wandb) is a tool data scientists can use on machine learning projects to facilitate retention, organization and reproducibility of experimental results achieved by multiple team members on a project. In this article, we walk you through all the steps necessary to incorporate wandb into a Keras based machine learning project. The main source for this tutorial is the wandb website, where you can look for clarification and/or more details.
I. Initial Steps
The first two steps to start using wandb are (1) sign-up for a new account with wandb and (2) install wandb in your development environment. This is the link to sign-up for a new account with wandb. Follow wandb’s instructions starting with this webpage.
Once you have signed-up for a new account, you can login and go to your account settings page to see (among other things) your API key(s) which we will be use in code snippets below.
As for installing wandb in a Python development environment, we used pip as instructed the pypi.org page for the wandb package.
II. Creating a Project to Share with Your Team
For the rest of this tutorial, we are going to assume a small team has a Keras based machine learning project on which each team member will be conducting experiments, the results of which they wish to share with each other. We are also assuming the team is new to wandb and wants to use free access with no concerns about the security of their models and experimental results (e.g., they are students working on a group project and are just trying to learn about wandb by incorporating into their project). For concreteness, we will be assuming the project team’s name is MultiDaskers.
Given the above scenario, one person on the team should login to wandb and create a project from the left panel of the home page.
Follow wandb prompts after selecting “Create new project”. For the team to work together (with free accounts), we chose the radio button to create an “open project” — which allows anyone both to see and update the project. We believe much greater security is available with groups and paid accounts, which we did not explore for this tutorial.
After creating a project, the wandb website provides a link to the page where each team member will go to observe and organize experimental results. Here is what that page will look like after you run several experiments.
We explain how to use this dashboard later in this tutorial.
III. Instrumenting your Python Code
It is worth elaborating on the reasons a team would use wandb. Here are the reasons that seem most motivating to us:
- They want to run ML experiments locally and share with their team members quickly and easily
- They want to run many experiments and keep track of hyperparameters, models and results automatically with minimal overhead and administration
- As an extension of the previous point, they want to identify and reproduce the best experiments as the project nears completion
So far, we have explained (1) how to sign-up for an account with wandb, (2) how to install wandb in your python development environment and (3) how to create an open project for which team members (and the public!) share the ability to see and contribute experimental results.
In this section, we show how to add a small amount of code to your Python training scripts so that your experimental results are automatically included in the wandb project dashboard. Again, we explain how to use the dashboard later in this tutorial.
For this tutorial, we show integration with a Keras based machine learning project. (Other integration options are explained in the wandb documentation.)
First change to a pre-existing Python training script: add import statements.
# imports
import wandb
from wandb.keras import WandbCallbackSecond change to a pre-existing Python training script: have your Python script login using your API key referenced above.
# login using api key for your account
wandb.login(key="9999999999999999999999999999999999999999")Third change to a pre-existing Python training script: initiate a new training run to add to your project on the wandb dashboard.
# these are just strings that your team should agree upon
# these will be used in dashboard for filtering/grouping results
project = "new_open_project"
group = "classifier training"
notes = "just me showing how notes work! :-)"# initiate a new training run
wandb.init(project=project,
group=group,
notes=notes)
The project string value should be the name of the open project created in wandb that you and your team members are sharing. As for group and notes, these are just string values that appear with each run on the wandb dashboard. On the dashboard, you can use these fields for filtering and grouping, so it is up to the team to agree on these string values to enable filtering and grouping of results (e.g., if your project involves pre-training and fine-tuning of a classifier, perhaps you want to agree that group will be either “pre-training” or “classifier training”.) You can refer to the wandb documentation for more details about the init object.
Fourth change to a pre-existing Python training script: specify hyperparameters and values that you want associated and saved with the run results.
# these get logged with the run (in 'info' section of the run)
# allows us to track hyperparameters about each run
# see below to 'de-reference' and use hyperparameters in training
# https://docs.wandb.com/ref/config
wandb.config.max_epochs = 20
wandb.config.optimizer = "Adam"You can refer to the wandb documentation for more details about the config object. After adding these hyperparameters to the wandb config object (which results in the values being associated with the run results on the wandb webpage), you must de-reference and use these values in your script. In the snippet of code below involving the .fit(), notice a value from the config object is passed to the epochs parameter.
Fifth and final change to a pre-existing Python training script: add wandb callback object to callbacks, so that wandb gets updated with results after each training step.
# add wandb callback to Keras .fit() callbacks
callbacks = [lr_scheduler, early_stopping,
WandbCallback(save_model=False)]history = model.fit(train_data,
validation_data=validation_data,
epochs=wandb.config.max_epochs,
callbacks=callbacks,
verbose=1)
You can refer to wandb documentation to learn more about the WandbCallback object. As you can see above, we use the default values for all attributes except save_model, which we set to False. If we do not set this value to False, we get a warning message during the .fit() process. In order to save the optimized model we (1) restore weights in Keras’ early stopping callback and (2) after the .fit() process is complete save the model as shown below to a directory which automatically gets synced with wandb when the script is finished running.
model.save(os.path.join(wandb.run.dir, "best_model_tf"),
save_format="tf")Before the script finishes execution, the contents of the wandb.run.dir is automatically synced with the wandb dashboard. If that directory contains a large model (e.g., 400MB), then this syncing takes many minutes to complete. You will know the syncing is complete when you see output on your console that looks something like the following.
Frankly, given the amount of time this upload takes, it might be a good idea not save your model to the wandb.run.dir directory unless you are confident your training will result in a model you wish to reuse subsequently.
Look here for information on wandb integration with other ML frameworks.
IV. Navigating the Dashboard
Every time we run a Python script (to train a model) that includes the above instrumentation, very valuable information is shared with wandb about the training run that we can access from our project’s page on the wandb website. There are probably many ways to navigate the wandb site. Here we show the starting point for our explanation of how we use the dashboard to monitor training results on a project.
The left hand panel shows all the training runs for this project. Again, a run is created each time the training script executes the wandb.init() for a project. wandb creates a string name for each run (unless you override when calling .init()). For example, the screenshot above shows many runs including a run called good-field-9. At the top of the left hand column is a search box in which you could type the name of the run.
We assume a project will include a large number of runs and show another way to navigate a larger number of runs further down in this tutorial. Before doing so, however, let’s see what information wandb collects and saves about each run.
wandb collects, organizes and saves an impressive amount of information and data (via the callback described above) about each training run. All the information and data is visible/accessible via the wandb website. We now show how to access the following data about each run:
- Training history
- Hyperparameters
- System information
- Model summary
- Training console output
- Other artifacts (including logs and model weights)
Seeing training history
From the left hand panel, click on the name of a run (e.g., good-field-9). This will take you to a page with plots of the learning rate, epoch, loss, val_loss and other metrics for the training run.
Seeing hyperparemters
From the slender left hand navigation bar, click on the “information” icon. This will take you to a page showing basic information about the run, including (1) the notes added to the init() object, (2) when the run was executed, (3) the config object value-keys, etc.
Seeing system information
From the slender left hand navigation bar, click on the “chip” icon. This will take you to a page showing information about computer on which the training was performed (e.g., system monitor of cpu utilization, memory, etc.)
Seeing model summary
From the slender left hand navigation bar, click on the “graph” icon. This will take you to a page showing information about the trained model structure.
Seeing training console output
From the slender left hand navigation bar, click on the “terminal prompt” icon. This will take you to a page showing console output generated during the training.
Seeing other artifacts (including logs and models)
From the slender left hand navigation bar, click on the “stack of pages” icon. This will take you to a page showing other artifacts collected by wandb that you can access or download. Among these are the (1) output log, (2) model structure and weights (if the model has been saved into the wandb.run.dir directory and automatically synced with wandb) and (3) requirements.txt.
You can refer to wandb documentation to learn about saving other artifacts associated with the training run that might be required to achieve reproducibility of your experiment (e.g., training data).
Other ways to navigate a larger number of runs
So, it does appear the wandb does retain and organize very important information about each training run. wandb collects this information automatically after only a small amount of Python code instrumentation is included in your training script. While this is all great, it seems likely that even a small team on a small project will create a very large number of training runs. Merely seeing all the runs (each with an arbitrary string name generated by wandb) in the left hand panel of the project home page is not a good way to find the runs of most interest to a team member. Fortunately, the wandb website has another way of navigating the potentially large number of runs: the table view.
From the slender left hand navigation bar of the project landing page, click on the “spreadsheet” icon. This will take you to a page showing the runs for the project in a spreadsheet table view. From the top of this table view, you can sort, filter and group runs by many attributes (including the group and notes attribute specified in the .init() object).
Best practices
We have not yet used wandb on a full project with multiple team members. Notwithstanding wandb’s ability to sort, filter and group runs, however, we imagine a “best practice” for team members is to prune their runs at the end of training session. It is very easy to delete runs from the dashboard. Perhaps, after 3–4 hours of training, a conscientious team member should look at the runs on the dashboard and remove runs that are clearly not worth saving for posterity.
BTW: It is worth noting that the training history plots on the wandb website are updated in real time while training is occurring, so we imagine a person conducting experiments might be looking at the dashboard continually and not just at the end of 3–4 hours of training.
V. Conclusions
wandb seems like a great tool to include in a project that involves a small team wanting to share and save results from many ML experiments. Getting started is quite easy and explained above. To recap, these are the main steps:
- Sign-up for account
- Install wandb Python package
- Create an open, shareable project on the wandb website
- Add a handful of lines of Python code to existing training scripts (to import wandb objects, to login, to initiate a training run within a project, to specify hyperparameter key-values in a wandb config object, to add a callback the Keras .fit() method on a model)
- Navigate and view results from each run on the wandb website
We are just getting started with wandb. There are clearly more features than we have demonstrated above. Please refer to wandb’s documentation for more details and additional information.
Good luck with your experiments!
