Inference and Visualization Tutorial¶
Welcome to the inference and visualization notebook! At this point, you should have a trained model and tiles to run inference. In this notebook we will run inference on a slide and visualize the results. Here are the steps we will review:
- Run inference with a trained model.
- Visualize the inference results
Run inference with a trained model¶
Often tissue-based analysis on whole slide images benefit from annotations provided by expert pathologists. However, having pathologists annotate 1000s of slides is very time consuming and expensive. To overcome this bottleneck, it is common to have pathologist annotate a subset of the slides, and use that dataset to train a model. This model is then used to label the rest of the dataset.
In the model training notebook, we trained a ResNet-18 model on a subset of our slides with the annotated regions and labels. We will now use this trained model and the prepared tiles from the test slide to run the inference step.
In [1]:
Copied!
import os
HOME = os.environ['HOME']
import os
HOME = os.environ['HOME']
In [2]:
Copied!
env DATASET_URL=file:///$HOME/vmount/PRO-12-123/
env DATASET_URL=file:///$HOME/vmount/PRO-12-123/
env: DATASET_URL=file:////home/limr/vmount/PRO-12-123/
In [3]:
Copied!
!infer_tiles --help
!infer_tiles --help
2023-04-04 18:36:07,485 - INFO - root - Initalized logger, log file at: luna.log
Usage: infer_tiles [OPTIONS] INPUT_SLIDE_TILES
Run a model with a specific pre-transform for all tiles in a slide
(tile_images), requires tiles to be saved (save_tiles) first
Inputs:
input_slide_tiles: path to tile images (.tiles.csv)
Outputs:
tile_scores
Example:
infer_tiles tiles/slide-100012/tiles
-rn msk-mind/luna-ml:main
-mn tissue_tile_net_model_5_class
-tn tissue_tile_net_transform
-wt main:tissue_net_2021-01-19_21.05.24-e17.pth
-o tiles/slide-100012/scores
Options:
-o, --output_dir TEXT path to output directory to save results
-rn, --hub_repo_or_dir TEXT repository name to pull model and weight from,
e.g. msk-mind/luna-ml
-mn, --model_name TEXT torch hub model name
-kw, --kwargs TEXT additional keywords to pass to model
initialization
-nc, --num_cores INTEGER Number of cores to use
-bx, --batch_size INTEGER batch size used for inference speedup
-m, --method_param_path TEXT path to a metadata json/yaml file with method
parameters to reproduce results
-dsid, --dataset_id TEXT Optional dataset identifier to add results to
--help Show this message and exit.
infer_tiles CLI takes in details on your trained model, and loads the tiles data for inference.
First we need to copy the model checkpoint into the classifier directory where we have a hubconf.py file that defines the model settings.
In [4]:
Copied!
! cp ../PRO-12-123/tissue_classifier_results/checkpoint_1.pt ~/vmount/classifier/model.checkpoint
! cp ../PRO-12-123/tissue_classifier_results/checkpoint_1.pt ~/vmount/classifier/model.checkpoint
Next, we need to generate tiles for the slides we want to compute predictions on. Previously, we only generated tiles for portions of the whole slide that contained annotations. Now, we want to tile the entire slide and only remove tiles that have background/glass and keep all of tiles containing tissue.
Just like before, the pipeline is to first generate_tiles for a particular slide, then to run detect_tissue to remove glass using an otsu threshold, and then to save_tiles, which we can use for inference.
In [5]:
Copied!
!generate_tiles \
file:~/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs \
--tile_size 128 --requested_magnification 20 \
-o ~/vmount/PRO-12-123/tiling/inference/tiles
!generate_tiles \
file:~/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs \
--tile_size 128 --requested_magnification 20 \
-o ~/vmount/PRO-12-123/tiling/inference/tiles
2023-04-04 18:36:11,265 - INFO - root - Initalized logger, log file at: luna.log
2023-04-04 18:36:11,268 - INFO - luna.common.utils - Started CLI Runner wtih <function generate_tiles at 0x7fa6658204c0>
2023-04-04 18:36:11,270 - INFO - luna.common.utils - Validating params...
2023-04-04 18:36:11,272 - INFO - luna.common.utils - -> Set input_slide_image (<class 'str'>) = file:~/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs
2023-04-04 18:36:11,274 - INFO - luna.common.utils - -> Set output_dir (<class 'str'>) = /home/limr/vmount/PRO-12-123/tiling/inference/tiles
2023-04-04 18:36:11,276 - INFO - luna.common.utils - -> Set tile_size (<class 'int'>) = 128
2023-04-04 18:36:11,278 - INFO - luna.common.utils - -> Set requested_magnification (<class 'float'>) = 20.0
2023-04-04 18:36:11,286 - INFO - luna.common.utils - Expanding inputs...
2023-04-04 18:36:11,287 - INFO - luna.common.utils - Attempting to read metadata at file:~/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs/metadata.yml
2023-04-04 18:36:11,291 - WARNING - luna.common.utils - Resolving a user directory, be careful!
2023-04-04 18:36:11,293 - INFO - luna.common.utils - Resolved input:
2023-04-04 18:36:11,293 - INFO - luna.common.utils - -> file:~/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs
2023-04-04 18:36:11,293 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs
2023-04-04 18:36:11,294 - INFO - luna.common.utils - Full segment key set: {}
2023-04-04 18:36:11,296 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:36:11,296 - INFO - luna.common.utils - Starting transform::generate_tiles
2023-04-04 18:36:11,296 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:36:11,508 - INFO - generate_tiles - Slide size = [53760,54840]
2023-04-04 18:36:11,510 - INFO - generate_tiles - Normalized magnification scale factor for 20.0x is 2.0
2023-04-04 18:36:11,511 - INFO - generate_tiles - Requested tile size=128, tile size at full magnficiation=256
2023-04-04 18:36:11,513 - INFO - generate_tiles - tiles x 210, tiles y 215
2023-04-04 18:36:11,552 - INFO - generate_tiles - Number of tiles in raster: 44304
2023-04-04 18:36:11,588 - INFO - generate_tiles - x_coord y_coord xy_extent tile_size tile_units
2023-04-04 18:36:11,588 - INFO - generate_tiles - address
2023-04-04 18:36:11,588 - INFO - generate_tiles - x1_y1_z20.0 256 256 256 128 px
2023-04-04 18:36:11,588 - INFO - generate_tiles - x1_y2_z20.0 256 512 256 128 px
2023-04-04 18:36:11,588 - INFO - generate_tiles - x1_y3_z20.0 256 768 256 128 px
2023-04-04 18:36:11,588 - INFO - generate_tiles - x1_y4_z20.0 256 1024 256 128 px
2023-04-04 18:36:11,588 - INFO - generate_tiles - x1_y5_z20.0 256 1280 256 128 px
2023-04-04 18:36:11,588 - INFO - generate_tiles - ... ... ... ... ... ...
2023-04-04 18:36:11,588 - INFO - generate_tiles - x208_y209_z20.0 53248 53504 256 128 px
2023-04-04 18:36:11,588 - INFO - generate_tiles - x208_y210_z20.0 53248 53760 256 128 px
2023-04-04 18:36:11,588 - INFO - generate_tiles - x208_y211_z20.0 53248 54016 256 128 px
2023-04-04 18:36:11,588 - INFO - generate_tiles - x208_y212_z20.0 53248 54272 256 128 px
2023-04-04 18:36:11,588 - INFO - generate_tiles - x208_y213_z20.0 53248 54528 256 128 px
2023-04-04 18:36:11,588 - INFO - generate_tiles -
2023-04-04 18:36:11,588 - INFO - generate_tiles - [44304 rows x 5 columns]
2023-04-04 18:36:11,671 - INFO - luna.common.utils - Code block 'transform::generate_tiles' took: 0.37402320199180394s
2023-04-04 18:36:11,673 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:36:11,673 - INFO - luna.common.utils - Done with transform, running post-transform functions...
2023-04-04 18:36:11,673 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:36:11,680 - INFO - luna.common.utils - Done.
In [6]:
Copied!
!detect_tissue \
~/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs \
~/vmount/PRO-12-123/tiling/inference/tiles \
--requested_magnification 2 \
--filter_query "(otsu_score > 0.1) & (purple_score > 0.1)" \
-o ~/vmount/PRO-12-123/tiling/inference/detect
!detect_tissue \
~/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs \
~/vmount/PRO-12-123/tiling/inference/tiles \
--requested_magnification 2 \
--filter_query "(otsu_score > 0.1) & (purple_score > 0.1)" \
-o ~/vmount/PRO-12-123/tiling/inference/detect
2023-04-04 18:36:14,285 - INFO - root - Initalized logger, log file at: luna.log
2023-04-04 18:36:14,288 - INFO - luna.common.utils - Started CLI Runner wtih <function detect_tissue at 0x7fa454813a60>
2023-04-04 18:36:14,290 - INFO - luna.common.utils - Validating params...
2023-04-04 18:36:14,293 - INFO - luna.common.utils - -> Set input_slide_image (<class 'str'>) = /home/limr/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs
2023-04-04 18:36:14,294 - INFO - luna.common.utils - -> Set input_slide_tiles (<class 'str'>) = /home/limr/vmount/PRO-12-123/tiling/inference/tiles
2023-04-04 18:36:14,296 - INFO - luna.common.utils - -> Set requested_magnification (<class 'float'>) = 2.0
2023-04-04 18:36:14,297 - INFO - luna.common.utils - -> Set filter_query (<class 'str'>) = (otsu_score > 0.1) & (purple_score > 0.1)
2023-04-04 18:36:14,299 - INFO - luna.common.utils - -> Set output_dir (<class 'str'>) = /home/limr/vmount/PRO-12-123/tiling/inference/detect
2023-04-04 18:36:14,300 - INFO - luna.common.utils - -> Set num_cores (<class 'int'>) = 4
2023-04-04 18:36:14,305 - INFO - luna.common.utils - Expanding inputs...
2023-04-04 18:36:14,306 - INFO - luna.common.utils - Attempting to read metadata at /home/limr/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs/metadata.yml
2023-04-04 18:36:14,309 - INFO - luna.common.utils - Attempting to read metadata at /home/limr/vmount/PRO-12-123/tiling/inference/tiles/metadata.yml
2023-04-04 18:36:14,315 - INFO - luna.common.utils - Expanded input:
2023-04-04 18:36:14,315 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/tiling/inference/tiles
2023-04-04 18:36:14,315 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/tiling/inference/tiles/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.tiles.parquet
2023-04-04 18:36:14,317 - INFO - luna.common.utils - Found segment keys: {'slide_id': '01OV002-bd8cdc70-3d46-40ae-99c4-90ef77'}
2023-04-04 18:36:14,319 - INFO - luna.common.utils - Resolved input:
2023-04-04 18:36:14,319 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs
2023-04-04 18:36:14,319 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs
2023-04-04 18:36:14,320 - INFO - luna.common.utils - Resolved input:
2023-04-04 18:36:14,320 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/tiling/inference/tiles/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.tiles.parquet
2023-04-04 18:36:14,320 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/tiling/inference/tiles/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.tiles.parquet
2023-04-04 18:36:14,322 - INFO - luna.common.utils - Full segment key set: {'slide_id': '01OV002-bd8cdc70-3d46-40ae-99c4-90ef77'}
2023-04-04 18:36:14,324 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:36:14,324 - INFO - luna.common.utils - Starting transform::detect_tissue
2023-04-04 18:36:14,324 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:36:14,543 - INFO - detect_tissue - Slide dimensions (53760, 54840)
2023-04-04 18:36:14,545 - INFO - detect_tissue - Thumbnail scale factor: 20.0
2023-04-04 18:36:16,462 - INFO - detect_tissue - Sample array size: (2742, 2688, 3)
2023-04-04 18:36:32,115 - INFO - detect_tissue - Starting otsu thresholding, threshold=0.7300508425245098
100%|████████████████████████████████████| 44304/44304 [01:28<00:00, 500.89it/s]
2023-04-04 18:38:04,814 - INFO - detect_tissue - Starting purple scoring
100%|████████████████████████████████████| 44304/44304 [01:29<00:00, 492.90it/s]
2023-04-04 18:39:40,158 - INFO - detect_tissue - Filtering based on query: (otsu_score > 0.1) & (purple_score > 0.1)
2023-04-04 18:39:40,174 - INFO - detect_tissue - x_coord y_coord ... otsu_score purple_score
2023-04-04 18:39:40,174 - INFO - detect_tissue - address ...
2023-04-04 18:39:40,174 - INFO - detect_tissue - x3_y58_z20.0 768 14848 ... 0.16 0.31
2023-04-04 18:39:40,174 - INFO - detect_tissue - x6_y172_z20.0 1536 44032 ... 0.47 0.43
2023-04-04 18:39:40,174 - INFO - detect_tissue - x6_y173_z20.0 1536 44288 ... 0.45 0.49
2023-04-04 18:39:40,174 - INFO - detect_tissue - x6_y174_z20.0 1536 44544 ... 0.35 0.28
2023-04-04 18:39:40,174 - INFO - detect_tissue - x6_y175_z20.0 1536 44800 ... 0.52 0.46
2023-04-04 18:39:40,174 - INFO - detect_tissue - ... ... ... ... ... ...
2023-04-04 18:39:40,174 - INFO - detect_tissue - x188_y117_z20.0 48128 29952 ... 0.19 0.19
2023-04-04 18:39:40,174 - INFO - detect_tissue - x188_y118_z20.0 48128 30208 ... 0.21 0.18
2023-04-04 18:39:40,174 - INFO - detect_tissue - x200_y167_z20.0 51200 42752 ... 0.15 0.11
2023-04-04 18:39:40,174 - INFO - detect_tissue - x201_y198_z20.0 51456 50688 ... 0.12 0.12
2023-04-04 18:39:40,174 - INFO - detect_tissue - x207_y208_z20.0 52992 53248 ... 0.11 0.24
2023-04-04 18:39:40,174 - INFO - detect_tissue -
2023-04-04 18:39:40,174 - INFO - detect_tissue - [19919 rows x 7 columns]
2023-04-04 18:39:40,306 - INFO - luna.common.utils - Code block 'transform::detect_tissue' took: 205.9781473109615s
2023-04-04 18:39:40,310 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:39:40,310 - INFO - luna.common.utils - Done with transform, running post-transform functions...
2023-04-04 18:39:40,310 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:39:40,319 - INFO - luna.common.utils - Done.
In [7]:
Copied!
!save_tiles \
~/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs \
~/vmount/PRO-12-123/tiling/inference/detect \
--num_cores 16 --batch_size 200 --dataset_id PRO_TILES_INFERENCE \
-o ~/vmount/PRO-12-123/tiling/inference/saved_tiles
!save_tiles \
~/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs \
~/vmount/PRO-12-123/tiling/inference/detect \
--num_cores 16 --batch_size 200 --dataset_id PRO_TILES_INFERENCE \
-o ~/vmount/PRO-12-123/tiling/inference/saved_tiles
2023-04-04 18:39:43,642 - INFO - root - Initalized logger, log file at: luna.log
2023-04-04 18:39:43,645 - INFO - luna.common.utils - Started CLI Runner wtih <function save_tiles at 0x7fb710429e50>
2023-04-04 18:39:43,647 - INFO - luna.common.utils - Validating params...
2023-04-04 18:39:43,650 - INFO - luna.common.utils - -> Set input_slide_image (<class 'str'>) = /home/limr/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs
2023-04-04 18:39:43,652 - INFO - luna.common.utils - -> Set input_slide_tiles (<class 'str'>) = /home/limr/vmount/PRO-12-123/tiling/inference/detect
2023-04-04 18:39:43,654 - INFO - luna.common.utils - -> Set output_dir (<class 'str'>) = /home/limr/vmount/PRO-12-123/tiling/inference/saved_tiles
2023-04-04 18:39:43,657 - INFO - luna.common.utils - -> Set num_cores (<class 'int'>) = 16
2023-04-04 18:39:43,659 - INFO - luna.common.utils - -> Set batch_size (<class 'int'>) = 200
2023-04-04 18:39:43,667 - INFO - luna.common.utils - Expanding inputs...
2023-04-04 18:39:43,669 - INFO - luna.common.utils - Attempting to read metadata at /home/limr/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs/metadata.yml
2023-04-04 18:39:43,671 - INFO - luna.common.utils - Attempting to read metadata at /home/limr/vmount/PRO-12-123/tiling/inference/detect/metadata.yml
2023-04-04 18:39:43,680 - INFO - luna.common.utils - Expanded input:
2023-04-04 18:39:43,680 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/tiling/inference/detect
2023-04-04 18:39:43,680 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/tiling/inference/detect/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77-filtered.tiles.parquet
2023-04-04 18:39:43,683 - INFO - luna.common.utils - Found segment keys: {'slide_id': '01OV002-bd8cdc70-3d46-40ae-99c4-90ef77'}
2023-04-04 18:39:43,686 - INFO - luna.common.utils - Resolved input:
2023-04-04 18:39:43,686 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs
2023-04-04 18:39:43,686 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs
2023-04-04 18:39:43,688 - INFO - luna.common.utils - Resolved input:
2023-04-04 18:39:43,688 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/tiling/inference/detect/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77-filtered.tiles.parquet
2023-04-04 18:39:43,688 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/tiling/inference/detect/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77-filtered.tiles.parquet
2023-04-04 18:39:43,692 - INFO - luna.common.utils - Full segment key set: {'slide_id': '01OV002-bd8cdc70-3d46-40ae-99c4-90ef77'}
2023-04-04 18:39:43,694 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:39:43,694 - INFO - luna.common.utils - Starting transform::save_tiles
2023-04-04 18:39:43,694 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:39:43,750 - INFO - generate_tiles - Now generating tiles with num_cores=16 and batch_size=200!
100%|█████████████████████████████████████████| 100/100 [01:32<00:00, 1.08it/s]
2023-04-04 18:41:18,938 - INFO - generate_tiles - x_coord ... tile_store
2023-04-04 18:41:18,938 - INFO - generate_tiles - address ...
2023-04-04 18:41:18,938 - INFO - generate_tiles - x3_y58_z20.0 768 ... /home/limr/vmount/PRO-12-123/tiling/inference/...
2023-04-04 18:41:18,938 - INFO - generate_tiles - x6_y172_z20.0 1536 ... /home/limr/vmount/PRO-12-123/tiling/inference/...
2023-04-04 18:41:18,938 - INFO - generate_tiles - x6_y173_z20.0 1536 ... /home/limr/vmount/PRO-12-123/tiling/inference/...
2023-04-04 18:41:18,938 - INFO - generate_tiles - x6_y174_z20.0 1536 ... /home/limr/vmount/PRO-12-123/tiling/inference/...
2023-04-04 18:41:18,938 - INFO - generate_tiles - x6_y175_z20.0 1536 ... /home/limr/vmount/PRO-12-123/tiling/inference/...
2023-04-04 18:41:18,938 - INFO - generate_tiles - ... ... ... ...
2023-04-04 18:41:18,938 - INFO - generate_tiles - x188_y117_z20.0 48128 ... /home/limr/vmount/PRO-12-123/tiling/inference/...
2023-04-04 18:41:18,938 - INFO - generate_tiles - x188_y118_z20.0 48128 ... /home/limr/vmount/PRO-12-123/tiling/inference/...
2023-04-04 18:41:18,938 - INFO - generate_tiles - x200_y167_z20.0 51200 ... /home/limr/vmount/PRO-12-123/tiling/inference/...
2023-04-04 18:41:18,938 - INFO - generate_tiles - x201_y198_z20.0 51456 ... /home/limr/vmount/PRO-12-123/tiling/inference/...
2023-04-04 18:41:18,938 - INFO - generate_tiles - x207_y208_z20.0 52992 ... /home/limr/vmount/PRO-12-123/tiling/inference/...
2023-04-04 18:41:18,938 - INFO - generate_tiles -
2023-04-04 18:41:18,938 - INFO - generate_tiles - [19919 rows x 8 columns]
2023-04-04 18:41:19,050 - INFO - luna.common.utils - Code block 'transform::save_tiles' took: 95.35192870604806s
2023-04-04 18:41:19,053 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:41:19,053 - INFO - luna.common.utils - Done with transform, running post-transform functions...
2023-04-04 18:41:19,053 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:41:19,060 - INFO - luna.common.utils - Adding feature segment /home/limr/vmount/PRO-12-123/tiling/inference/saved_tiles/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.tiles.parquet to PRO_TILES_INFERENCE
2023-04-04 18:41:19,062 - INFO - luna.common.utils - Found dataset URL = file:////home/limr/vmount/PRO-12-123/
2023-04-04 18:41:19,064 - INFO - luna.common.utils - Adding /home/limr/vmount/PRO-12-123/tiling/inference/saved_tiles/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.tiles.parquet to PRO_TILES_INFERENCE via file:////home/limr/vmount/PRO-12-123/
2023-04-04 18:41:19,066 - INFO - luna.common.utils - SEGMENT_ID=01OV002-bd8cdc70-3d46-40ae-99c4-90ef77
2023-04-04 18:41:19,068 - INFO - luna.common.utils - Writing to: //home/limr/vmount/PRO-12-123/datasets/PRO_TILES_INFERENCE/segments/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77
2023-04-04 18:41:19,218 - INFO - luna.common.utils - Done.
Now that we have a set of saved tiles, we can run inference by specifying the dataset of saved tiles and some parameters about our model.
In [8]:
Copied!
!infer_tiles ~/vmount/PRO-12-123/datasets/PRO_TILES_INFERENCE/ \
--output_dir ../PRO-12-123/sample_tiles_inference \
--hub_repo_or_dir ~/vmount/classifier \
--model_name 'tissue_classifier' \
--num_cores 4 \
--batch_size 16 \
-kw "{'network':'torchvision.models.resnet18', 'num_labels': 3}"
!infer_tiles ~/vmount/PRO-12-123/datasets/PRO_TILES_INFERENCE/ \
--output_dir ../PRO-12-123/sample_tiles_inference \
--hub_repo_or_dir ~/vmount/classifier \
--model_name 'tissue_classifier' \
--num_cores 4 \
--batch_size 16 \
-kw "{'network':'torchvision.models.resnet18', 'num_labels': 3}"
2023-04-04 18:41:23,362 - INFO - root - Initalized logger, log file at: luna.log
2023-04-04 18:41:23,365 - INFO - luna.common.utils - Started CLI Runner wtih <function infer_tile_labels at 0x7f22a2aa6d30>
2023-04-04 18:41:23,367 - INFO - luna.common.utils - Validating params...
2023-04-04 18:41:23,369 - INFO - luna.common.utils - -> Set input_slide_tiles (<class 'str'>) = /home/limr/vmount/PRO-12-123/datasets/PRO_TILES_INFERENCE/
2023-04-04 18:41:23,371 - INFO - luna.common.utils - -> Set output_dir (<class 'str'>) = ../PRO-12-123/sample_tiles_inference
2023-04-04 18:41:23,373 - INFO - luna.common.utils - -> Set hub_repo_or_dir (<class 'str'>) = /home/limr/vmount/classifier
2023-04-04 18:41:23,375 - INFO - luna.common.utils - -> Set model_name (<class 'str'>) = tissue_classifier
2023-04-04 18:41:23,377 - INFO - luna.common.utils - -> Set kwargs (<class 'dict'>) = {'network': 'torchvision.models.resnet18', 'num_labels': 3}
2023-04-04 18:41:23,379 - INFO - luna.common.utils - -> Set num_cores (<class 'int'>) = 4
2023-04-04 18:41:23,381 - INFO - luna.common.utils - -> Set batch_size (<class 'int'>) = 16
2023-04-04 18:41:23,386 - INFO - luna.common.utils - Expanding inputs...
2023-04-04 18:41:23,388 - INFO - luna.common.utils - Attempting to read metadata at /home/limr/vmount/PRO-12-123/datasets/PRO_TILES_INFERENCE/metadata.yml
2023-04-04 18:41:23,391 - INFO - luna.common.utils - Resolved input:
2023-04-04 18:41:23,391 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/datasets/PRO_TILES_INFERENCE/
2023-04-04 18:41:23,391 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/datasets/PRO_TILES_INFERENCE
2023-04-04 18:41:23,394 - INFO - luna.common.utils - Full segment key set: {}
2023-04-04 18:41:23,396 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:41:23,396 - INFO - luna.common.utils - Starting transform::infer_tile_labels
2023-04-04 18:41:23,396 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:41:23,398 - INFO - infer_tile_labels - Torch hub source = local @ /home/limr/vmount/classifier
2023-04-04 18:41:24,189 - INFO - infer_tile_labels - Using device = cpu
100%|███████████████████████████████████████| 1245/1245 [04:45<00:00, 4.35it/s]
2023-04-04 18:46:10,325 - INFO - infer_tile_labels - index ... 2
2023-04-04 18:46:10,325 - INFO - infer_tile_labels - address ...
2023-04-04 18:46:10,325 - INFO - infer_tile_labels - x3_y58_z20.0 0 ... 1.248038
2023-04-04 18:46:10,325 - INFO - infer_tile_labels - x6_y172_z20.0 1 ... -0.665792
2023-04-04 18:46:10,325 - INFO - infer_tile_labels - x6_y173_z20.0 2 ... -0.087695
2023-04-04 18:46:10,325 - INFO - infer_tile_labels - x6_y174_z20.0 3 ... -0.846900
2023-04-04 18:46:10,325 - INFO - infer_tile_labels - x6_y175_z20.0 4 ... -1.829775
2023-04-04 18:46:10,325 - INFO - infer_tile_labels - ... ... ... ...
2023-04-04 18:46:10,325 - INFO - infer_tile_labels - x188_y117_z20.0 19914 ... 0.430831
2023-04-04 18:46:10,325 - INFO - infer_tile_labels - x188_y118_z20.0 19915 ... -0.096828
2023-04-04 18:46:10,325 - INFO - infer_tile_labels - x200_y167_z20.0 19916 ... 1.615410
2023-04-04 18:46:10,325 - INFO - infer_tile_labels - x201_y198_z20.0 19917 ... 1.858101
2023-04-04 18:46:10,325 - INFO - infer_tile_labels - x207_y208_z20.0 19918 ... 1.987634
2023-04-04 18:46:10,325 - INFO - infer_tile_labels -
2023-04-04 18:46:10,325 - INFO - infer_tile_labels - [19919 rows x 14 columns]
2023-04-04 18:46:10,549 - INFO - luna.common.utils - Code block 'transform::infer_tile_labels' took: 287.1466377560282s
2023-04-04 18:46:10,553 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:46:10,553 - INFO - luna.common.utils - Done with transform, running post-transform functions...
2023-04-04 18:46:10,553 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:46:10,563 - INFO - luna.common.utils - Done.
The output of the inference is saved in a paquet file. Let's take a look at the results.
In [9]:
Copied!
!ls -lhtr ../PRO-12-123/sample_tiles_inference
!ls -lhtr ../PRO-12-123/sample_tiles_inference
total 628K -rw-r--r-- 1 limr limr 624K Apr 4 18:46 tile_scores_and_labels_pytorch_inference.parquet -rw-r--r-- 1 limr limr 694 Apr 4 18:46 metadata.yml
In [10]:
Copied!
import pandas as pd
results = pd.read_parquet("../PRO-12-123/sample_tiles_inference/tile_scores_and_labels_pytorch_inference.parquet")
display(results)
import pandas as pd
results = pd.read_parquet("../PRO-12-123/sample_tiles_inference/tile_scores_and_labels_pytorch_inference.parquet")
display(results)
| index | SEGMENT_ID | slide_id | x_coord | y_coord | xy_extent | tile_size | tile_units | otsu_score | purple_score | tile_store | 0 | 1 | 2 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| address | ||||||||||||||
| x3_y58_z20.0 | 0 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 768 | 14848 | 256 | 128 | px | 0.16 | 0.31 | /home/limr/vmount/PRO-12-123/tiling/inference/... | -3.520283 | 1.765262 | 1.248038 |
| x6_y172_z20.0 | 1 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 1536 | 44032 | 256 | 128 | px | 0.47 | 0.43 | /home/limr/vmount/PRO-12-123/tiling/inference/... | 0.077844 | -0.729116 | -0.665792 |
| x6_y173_z20.0 | 2 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 1536 | 44288 | 256 | 128 | px | 0.45 | 0.49 | /home/limr/vmount/PRO-12-123/tiling/inference/... | -0.957561 | 0.014459 | -0.087695 |
| x6_y174_z20.0 | 3 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 1536 | 44544 | 256 | 128 | px | 0.35 | 0.28 | /home/limr/vmount/PRO-12-123/tiling/inference/... | -0.496429 | 0.275059 | -0.846900 |
| x6_y175_z20.0 | 4 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 1536 | 44800 | 256 | 128 | px | 0.52 | 0.46 | /home/limr/vmount/PRO-12-123/tiling/inference/... | 0.669656 | -0.043495 | -1.829775 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| x188_y117_z20.0 | 19914 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 48128 | 29952 | 256 | 128 | px | 0.19 | 0.19 | /home/limr/vmount/PRO-12-123/tiling/inference/... | -2.039448 | 0.457396 | 0.430831 |
| x188_y118_z20.0 | 19915 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 48128 | 30208 | 256 | 128 | px | 0.21 | 0.18 | /home/limr/vmount/PRO-12-123/tiling/inference/... | -1.841984 | 0.902088 | -0.096828 |
| x200_y167_z20.0 | 19916 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 51200 | 42752 | 256 | 128 | px | 0.15 | 0.11 | /home/limr/vmount/PRO-12-123/tiling/inference/... | -2.680739 | 0.211705 | 1.615410 |
| x201_y198_z20.0 | 19917 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 51456 | 50688 | 256 | 128 | px | 0.12 | 0.12 | /home/limr/vmount/PRO-12-123/tiling/inference/... | -3.083220 | 0.549318 | 1.858101 |
| x207_y208_z20.0 | 19918 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 52992 | 53248 | 256 | 128 | px | 0.11 | 0.24 | /home/limr/vmount/PRO-12-123/tiling/inference/... | -3.263530 | 0.455547 | 1.987634 |
19919 rows × 14 columns
Visualize the inference results¶
Now we will visualize the inference results. visualize_tiles_png CLI creates heatmaps based on the scores, and saves the thumbnail images in png format.
In [11]:
Copied!
!visualize_tiles_png --help
!visualize_tiles_png --help
2023-04-04 18:46:15,577 - INFO - root - Initalized logger, log file at: luna.log
Usage: visualize_tiles_png [OPTIONS] INPUT_SLIDE_IMAGE INPUT_SLIDE_TILES
Generate nice tile markup images with continuous or discrete tile scores
Inputs:
input_slide_image: slide image (virtual slide formats compatible with openslide, .svs, .tif, .scn, ...)
input_slide_tiles: slide tiles (manifest tile files, .tiles.csv)
Outputs:
markups: markup images
Example:
visualize_tiles_png 10001.svs 10001/tiles/10001.tiles.csv
-o 10001/markups
-pl Tumor,Stroma,TILs,otsu_score
-rmg 0.5
Options:
-o, --output_dir TEXT path to output directory to save results
-pl, --plot_labels TEXT Label names (as column labels) to plot
-rmg, --requested_magnification TEXT
Magnificiation scale at which to generate
thumbnail/png images (recommended <= 1)
--mpp-units Set this flag if input coordinates are in
µm, not pixels
-m, --method_param_path TEXT path to a metadata json/yaml file with
method parameters to reproduce results
--help Show this message and exit.
If you want to evaluate your model results in detail, it is desirable to review the results and images in high-magnification. We use Digital Slide Archive (DSA) viewer to examine the high resolution image and results. DSA is a web-based platform and this enables us to easily share the images and model results with other researchers via a link.
A set of CLIs are available to help you convert your pathologist or model-generated annotations and push them to DSA. Please refer to the dsa-tools.ipynb notebook for more details.
In [12]:
Copied!
!visualize_tiles_png \
~/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs \
../PRO-12-123/sample_tiles_inference/tile_scores_and_labels_pytorch_inference.parquet \
--output_dir ../PRO-12-123/sample_tiles_viz \
--plot_labels 0 \
--requested_magnification 1
!visualize_tiles_png \
~/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs \
../PRO-12-123/sample_tiles_inference/tile_scores_and_labels_pytorch_inference.parquet \
--output_dir ../PRO-12-123/sample_tiles_viz \
--plot_labels 0 \
--requested_magnification 1
2023-04-04 18:46:18,960 - INFO - root - Initalized logger, log file at: luna.log
2023-04-04 18:46:18,963 - INFO - luna.common.utils - Started CLI Runner wtih <function visualize_tiles at 0x7f5f84edc820>
2023-04-04 18:46:18,967 - INFO - luna.common.utils - Validating params...
2023-04-04 18:46:18,970 - INFO - luna.common.utils - -> Set input_slide_image (<class 'str'>) = /home/limr/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs
2023-04-04 18:46:18,973 - INFO - luna.common.utils - -> Set input_slide_tiles (<class 'str'>) = ../PRO-12-123/sample_tiles_inference/tile_scores_and_labels_pytorch_inference.parquet
2023-04-04 18:46:18,975 - INFO - luna.common.utils - -> Set mpp_units (<class 'bool'>) = False
2023-04-04 18:46:18,977 - INFO - luna.common.utils - -> Set plot_labels (typing.List[str]) = ['0']
2023-04-04 18:46:18,980 - INFO - luna.common.utils - -> Set requested_magnification (<class 'float'>) = 1.0
2023-04-04 18:46:18,983 - INFO - luna.common.utils - -> Set output_dir (<class 'str'>) = ../PRO-12-123/sample_tiles_viz
2023-04-04 18:46:18,989 - INFO - luna.common.utils - Expanding inputs...
2023-04-04 18:46:18,991 - INFO - luna.common.utils - Attempting to read metadata at /home/limr/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs/metadata.yml
2023-04-04 18:46:18,994 - INFO - luna.common.utils - Attempting to read metadata at ../PRO-12-123/sample_tiles_inference/tile_scores_and_labels_pytorch_inference.parquet/metadata.yml
2023-04-04 18:46:18,998 - INFO - luna.common.utils - Resolved input:
2023-04-04 18:46:18,998 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs
2023-04-04 18:46:18,998 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/data/toy_data_set/01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs
2023-04-04 18:46:19,001 - INFO - luna.common.utils - Resolved input:
2023-04-04 18:46:19,001 - INFO - luna.common.utils - -> ../PRO-12-123/sample_tiles_inference/tile_scores_and_labels_pytorch_inference.parquet
2023-04-04 18:46:19,001 - INFO - luna.common.utils - -> /home/limr/vmount/PRO-12-123/sample_tiles_inference/tile_scores_and_labels_pytorch_inference.parquet
2023-04-04 18:46:19,004 - INFO - luna.common.utils - Full segment key set: {}
2023-04-04 18:46:19,006 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:46:19,006 - INFO - luna.common.utils - Starting transform::visualize_tiles
2023-04-04 18:46:19,006 - INFO - luna.common.utils - ------------------------------------------------------------
100%|███████████████████████████████████| 19919/19919 [00:17<00:00, 1128.24it/s]
2023-04-04 18:46:38,038 - INFO - visualize_tiles_png - Saved 0 visualization at ../PRO-12-123/sample_tiles_viz/tile_scores_and_labels_visualization_0.png
2023-04-04 18:46:38,045 - INFO - luna.common.utils - Code block 'transform::visualize_tiles' took: 19.036363508028444s
2023-04-04 18:46:38,047 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:46:38,047 - INFO - luna.common.utils - Done with transform, running post-transform functions...
2023-04-04 18:46:38,047 - INFO - luna.common.utils - ------------------------------------------------------------
2023-04-04 18:46:38,053 - INFO - luna.common.utils - Done.
In [13]:
Copied!
from IPython.display import Image
Image('../PRO-12-123/sample_tiles_viz/tile_scores_and_labels_visualization_0.png')
from IPython.display import Image
Image('../PRO-12-123/sample_tiles_viz/tile_scores_and_labels_visualization_0.png')
Out[13]:
Congratulations on completing the inference and visualization notebook! To view the end-to-end pipeline of the tiling workflow, please checkout the end-to-end notebook.