Dataset Preparation Tutorial¶
Welcome to the dataset preparation tutorial! In this notebook, we will download the toy data set for the tutorial and prepare the necessary tables used for later analysis. Here are the steps we will review:
- Verify prerequisites
- Create a new project workspace
- Review sample dataset
- Build the proxy table
- Run regional annotation ETL
NOTE: All of the configuration files for this tutorial have been provided in the container. The host and port values in the configuration files are dynamically set based on your system.
NOTE: The current working directory is '~/vmount/notebooks'. All file and directory paths specified in the configuration files are relative to the current working directory.
1. Verify prerequisites¶
Here are the software prerequisites for executing tasks with luna packages. These prerequisites have already been baked into this docker container. To view the setup, please see the corresponding dockerfile.
In [1]:
Copied!
!python3 --version
!echo LUNA_HOME: $LUNA_HOME
import luna.pathology
print(luna.pathology.__path__)
!python3 --version
!echo LUNA_HOME: $LUNA_HOME
import luna.pathology
print(luna.pathology.__path__)
Python 3.9.16 LUNA_HOME: /home/pollardw/vmount ['/opt/conda/lib/python3.9/site-packages/luna/pathology']
Verify that the dask scheduler is running in the docker container
In [2]:
Copied!
# !ps -ewo pid,ppid,args
!echo "LUNA_DASK_SCHEDULER: $LUNA_DASK_SCHEDULER"
# !ps -ewo pid,ppid,args
!echo "LUNA_DASK_SCHEDULER: $LUNA_DASK_SCHEDULER"
LUNA_DASK_SCHEDULER: tcp://192.168.176.4:8786
2. Create a new project workspace¶
Next, we create a luna home space and place the configuration files there. Using a manifest file, we will create a project workspace where your configurations, data, models, and outputs will go for this tutorial.
In [3]:
Copied!
%%bash
mkdir -p ~/luna
cp -R ~/vmount/conf ~/luna
cat ~/luna/conf/manifest.yaml
cp ~/luna/conf/manifest.yaml "${LUNA_HOME}/PRO-12-123/"
%%bash
mkdir -p ~/luna
cp -R ~/vmount/conf ~/luna
cat ~/luna/conf/manifest.yaml
cp ~/luna/conf/manifest.yaml "${LUNA_HOME}/PRO-12-123/"
# project manifest template # MIND project id PROJECT: PRO-12-123 # IRB IRB: # project title TITLE: pathology-tutorial # project description DESCRIPTION: End-to-end pathology analysis tutorial DATA_MODALITIES: pathology ROOT_PATH: ../
You should now see the manifest.yaml file in your vmount/PRO-12-123 directory. This will be your project workspace.
3. Review sample dataset¶
The data we'll be using for this tutorial is a set of 5 whole slide images (WSI) of ovarian cancer H&E slides, available in the svs file format. Whole slide imaging refers to the scanning of conventional glass slides for research purposes; in this case, these are slides that oncologists have used while inspecting cancer samples.
The slides have been downloaded by the script vmount/provision_girder.py, which ...
Creates an admin user and default assetstore
Downloads sample data from public kitware site. to
~/vmount/PRO-12-123/data/toy_data_set/Creates a collection and adds the slides to your local DSA
If this was successsful, the downloaded svs files will be listed by the tree command, below ...
In [4]:
Copied!
!tree "${LUNA_HOME}/PRO-12-123/data/toy_data_set"
!tree "${LUNA_HOME}/PRO-12-123/data/toy_data_set"
/home/pollardw/vmount/PRO-12-123/data/toy_data_set ├── 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs ├── 01OV002-ed65cf94-8bc6-492b-9149-adc16f.svs ├── 01OV007-9b90eb78-2f50-4aeb-b010-d642f9.svs ├── 01OV008-308ad404-7079-4ff8-8232-12ee2e.svs ├── 01OV008-7579323e-2fae-43a9-b00f-a15c28.svs └── table ├── ANNOTATIONS │ ├── 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.annotation.geojson │ ├── 01OV002-ed65cf94-8bc6-492b-9149-adc16f.annotation.geojson │ ├── 01OV007-9b90eb78-2f50-4aeb-b010-d642f9.annotation.geojson │ ├── 01OV008-308ad404-7079-4ff8-8232-12ee2e.annotation.geojson │ ├── 01OV008-7579323e-2fae-43a9-b00f-a15c28.annotation.geojson │ ├── metadata.yml │ └── slide_annotation_dataset_TCGA collection_ov_regional.parquet └── SLIDES └── slide_ingest_PRO-12-123.parquet 3 directories, 13 files
These files may also be viewed through the Minio instance. The URL for this instance is available in the docker-compose terminal log.
If you want to import your own data, you can do so from your local filesystem as well as an object store. For more details, refer to the girder user documentation
To import images from your local filesystem,
- Login to DSA with admin/password1
- Navigate to Collections and use the create collection interface to create a new collection
- Under the branch icon on the right, create a folder within the collection
- In this folder, use the blue info button to access the Unique ID for the folder, and copy and store it somewhere for reference.
- Add images to your local computer at
vmount/assetstore - Navigate to Admin Console -> Assetstores
- From the default assetstore, click on Import data
- Specify the path to the images you wish to import using absolute path, e.g.
/home/<user>/vmount/assetstore/yourimage, and specify the destination type as 'Folder' and the destination ID as the ID copied earlier, and click import
As the /assetstore mount is available to DSA, this import should be much faster than uploading the image through the Upload files button in the UI.
4. Build the proxy table¶
Now, we will run the Whole Slide Image (WSI) ETL to build a meta-data catalog of the slides in a proxy table.
For reference, ETL stands for extract-transform-load; it is the method that often involves cleaning data, transforming data types, and loading data into different systems.
In [5]:
Copied!
!slide_etl "${LUNA_HOME}/PRO-12-123/data/toy_data_set" \
--project_name PRO-12-123 --comment "Example Ingestion Job" \
--no-copy --output-urlpath "${LUNA_HOME}/PRO-12-123/data/toy_data_set/table/SLIDES"
!slide_etl "${LUNA_HOME}/PRO-12-123/data/toy_data_set" \
--project_name PRO-12-123 --comment "Example Ingestion Job" \
--no-copy --output-urlpath "${LUNA_HOME}/PRO-12-123/data/toy_data_set/table/SLIDES"
2023-08-03 01:50:46.648 | INFO | luna.pathology.cli.slide_etl:cli:95 - id ... properties.tiffslide.mpp-y 0 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 ... 0.2501 1 01OV002-ed65cf94-8bc6-492b-9149-adc16f ... 0.2501 2 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 ... 0.2501 3 01OV008-308ad404-7079-4ff8-8232-12ee2e ... 0.2501 4 01OV008-7579323e-2fae-43a9-b00f-a15c28 ... 0.2501 [5 rows x 70 columns] 2023-08-03 01:50:46.702 | INFO | luna.pathology.cli.slide_etl:cli:107 - Writing to parquet file 2023-08-03 01:50:46.726 | DEBUG | luna.common.utils:wrapper:146 - cli ran in 4.82s
This step may take a while. At the end, your proxy table should be generated!
Note code for the following text has been handled elsewhere, but it is still good to read the following to familiarize yourself with some potential issues with overfitting data.
Before we view the table, we must first update it to associate patient IDs with the slides. This is necessary for correctly training and validating the machine learning model in the coming notebooks. Once the slides are divided into "tiles" in the next notebook, the tiles are split between the training and validation sets for the ML model. If the tiles do not have patient ID's associated with them, then it is possible for tiles from one individual to appear in both the training and validation of the model; this would cause researchers to have an exaggerated interpretation of the model's accuracy, since we would essentially be validating the model on information that is too near to what it has already seen.
Note that we will not be using patient IDs associated with MSK. Instead, we will be using spoof IDs that will suffice for this tutorial. When running this workflow with real data, make sure to include the IDs safely and securely. Run the following block of code to add a 'patient_id' column to the table and store it using Spark (DEPRECATED).
Next, we may view the WSI table! This table should have the metadata associated with the WSI slides that you just collected, including the patient IDs.
This table may also be viewed through the Dremio instance. The URL for this instance is available in the docker-compose terminal log.
In [6]:
Copied!
import os
LUNA_HOME = os.environ["LUNA_HOME"]
TABLE_DIR = f"{LUNA_HOME}/PRO-12-123/data/toy_data_set/table"
import pandas as pd
pd.read_parquet(f"{TABLE_DIR}/SLIDES/slide_ingest_PRO-12-123.parquet")
import os
LUNA_HOME = os.environ["LUNA_HOME"]
TABLE_DIR = f"{LUNA_HOME}/PRO-12-123/data/toy_data_set/table"
import pandas as pd
pd.read_parquet(f"{TABLE_DIR}/SLIDES/slide_ingest_PRO-12-123.parquet")
Out[6]:
| id | project_name | comment | slide_size | uuid | url | channel0_R | channel0_G | channel0_B | channel1_R | ... | properties.tiffslide.series-index | properties.tiffslide.level[1].downsample | properties.tiffslide.bounds-y | properties.aperio.Top | properties.tiffslide.level[3].tile-height | properties.tiffslide.comment | properties.tiffslide.vendor | properties.aperio.Time Zone | properties.aperio.OriginalHeight | properties.tiffslide.mpp-y | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | PRO-12-123 | Example Ingestion Job | 237047223 | cd8c82fb-69d4-3ef2-8cb7-1eb0919c8d36 | file:///home/pollardw/vmount/PRO-12-123/data/t... | 0.620722 | 0.651500 | 0.436179 | 0.344821 | ... | 0 | 4.000000 | None | 19.529366 | 256.0 | Aperio Image Library vFS90 01\r\n56000x54940 [... | aperio | GMT-05:00 | 54940 | 0.2501 |
| 1 | 01OV002-ed65cf94-8bc6-492b-9149-adc16f | PRO-12-123 | Example Ingestion Job | 240691747 | e60d567c-a5d6-3030-b890-49f454eeee6c | file:///home/pollardw/vmount/PRO-12-123/data/t... | 0.628632 | 0.613612 | 0.477809 | 0.386583 | ... | 0 | 4.000121 | None | 17.250914 | 256.0 | Aperio Image Library vFS90 01\r\n53000x49887 [... | aperio | GMT-05:00 | 49887 | 0.2501 |
| 2 | 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 | PRO-12-123 | Example Ingestion Job | 262796337 | 71a882f0-b708-3a57-b152-b5ddfffe84ba | file:///home/pollardw/vmount/PRO-12-123/data/t... | 0.571893 | 0.659842 | 0.487388 | 0.268574 | ... | 0 | 4.000067 | None | 17.753954 | 256.0 | Aperio Image Library vFS90 01\r\n114000x60210 ... | aperio | GMT-05:00 | 60210 | 0.2501 |
| 3 | 01OV008-308ad404-7079-4ff8-8232-12ee2e | PRO-12-123 | Example Ingestion Job | 207479411 | 93604598-ede6-312b-b060-5375a9823e4b | file:///home/pollardw/vmount/PRO-12-123/data/t... | 0.585755 | 0.675688 | 0.447589 | 0.343329 | ... | 0 | 4.000000 | None | 18.320700 | NaN | Aperio Image Library vFS90 01\r\n55000x38728 [... | aperio | GMT-05:00 | 38728 | 0.2501 |
| 4 | 01OV008-7579323e-2fae-43a9-b00f-a15c28 | PRO-12-123 | Example Ingestion Job | 215796305 | ae824866-e870-32b9-90d2-b261fe78ce94 | file:///home/pollardw/vmount/PRO-12-123/data/t... | 0.618213 | 0.666116 | 0.417255 | 0.267862 | ... | 0 | 4.000034 | None | 20.990768 | 256.0 | Aperio Image Library vFS90 01\r\n44000x59309 [... | aperio | GMT-05:00 | 59309 | 0.2501 |
5 rows × 70 columns
If the table loads, then you have successfully run the Whole Slide Image (WSI) ETL to database the slides.
Run the regional annotation ETL¶
The whole slide images that you downloaded are images of ovarian cancer, but not every pixel on each slide is a tumor. In fact, the images typically include tumor cells, normal ovarian cells, lymphoctyes, and more. Note: A non-expert annotated this slide for demo purposes only.
The regional annotation ETL performs the following steps
- Download DSA json annotations
- Convert DSA jsons to GeoJSON format, which is compatible with downstream applications
- Save configs in your
~/vmount/PRO-12-123/configs/REGIONAL_METADATA_RESULTS - Save parquet table in your
~/vmount/PRO-12-123/tables/REGIONAL_METADATA_RESULTS
To run the regional annotation ETL, we use the dsa_annotation CLI. For more details on the dsa_annotation tool, and the annotations we support, please checkout the 7_dsa-annotation.ipynb notebook.
In [7]:
Copied!
!mkdir -p "{TABLE_DIR}/ANNOTATIONS"
!dsa_annotation http://girder:8080/api/v1 \
--collection-name "TCGA collection" \
--annotation-name "ov_regional" \
--username admin --password password1 \
--output-urlpath "{TABLE_DIR}/ANNOTATIONS"
!mkdir -p "{TABLE_DIR}/ANNOTATIONS"
!dsa_annotation http://girder:8080/api/v1 \
--collection-name "TCGA collection" \
--annotation-name "ov_regional" \
--username admin --password password1 \
--output-urlpath "{TABLE_DIR}/ANNOTATIONS"
2023-08-03 01:51:01.237 | INFO | luna.pathology.dsa.dsa_api_handler:system_check:447 - Successfully connected to DSA 2023-08-03 01:51:01.263 | DEBUG | luna.pathology.dsa.dsa_api_handler:get_collection_uuid:32 - Found collections _accessLevel ... updated _id ... 64c2d70a780496a4e184cb16 2 ... 2023-07-27T20:43:54.029000+00:00 [1 rows x 9 columns] 2023-08-03 01:51:01.264 | INFO | luna.pathology.dsa.dsa_api_handler:get_collection_uuid:44 - Found collection id=64c2d70a780496a4e184cb16 for collection=TCGA collection 2023-08-03 01:51:01.273 | INFO | luna.pathology.dsa.dsa_api_handler:get_slide_df:524 - Found 5 slides! 2023-08-03 01:51:01.275 | INFO | luna.pathology.cli.dsa_annotation_etl:dsa_annotation_etl:146 - Dashboard: http://192.168.176.4:8787/status _id ... annotation_name slide_id ... 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 64c2d70a780496a4e184cb18 ... ov_regional 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 64c2d70a780496a4e184cb18 ... ov_regional 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 64c2d70a780496a4e184cb18 ... ov_regional 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 64c2d70a780496a4e184cb18 ... ov_regional 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 64c2d70a780496a4e184cb18 ... ov_regional 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 64c2d70a780496a4e184cb18 ... ov_regional 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 64c2d70a780496a4e184cb18 ... ov_regional 01OV002-ed65cf94-8bc6-492b-9149-adc16f 64c2d70c780496a4e184cb24 ... ov_regional 01OV002-ed65cf94-8bc6-492b-9149-adc16f 64c2d70c780496a4e184cb24 ... ov_regional 01OV002-ed65cf94-8bc6-492b-9149-adc16f 64c2d70c780496a4e184cb24 ... ov_regional 01OV002-ed65cf94-8bc6-492b-9149-adc16f 64c2d70c780496a4e184cb24 ... ov_regional 01OV002-ed65cf94-8bc6-492b-9149-adc16f 64c2d70c780496a4e184cb24 ... ov_regional 01OV002-ed65cf94-8bc6-492b-9149-adc16f 64c2d70c780496a4e184cb24 ... ov_regional 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 64c2d70e780496a4e184cb2e ... ov_regional 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 64c2d70e780496a4e184cb2e ... ov_regional 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 64c2d70e780496a4e184cb2e ... ov_regional 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 64c2d70e780496a4e184cb2e ... ov_regional 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 64c2d70e780496a4e184cb2e ... ov_regional 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 64c2d70e780496a4e184cb2e ... ov_regional 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 64c2d70e780496a4e184cb2e ... ov_regional 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 64c2d70e780496a4e184cb2e ... ov_regional 01OV008-308ad404-7079-4ff8-8232-12ee2e 64c2d710780496a4e184cb3a ... ov_regional 01OV008-308ad404-7079-4ff8-8232-12ee2e 64c2d710780496a4e184cb3a ... ov_regional 01OV008-308ad404-7079-4ff8-8232-12ee2e 64c2d710780496a4e184cb3a ... ov_regional 01OV008-308ad404-7079-4ff8-8232-12ee2e 64c2d710780496a4e184cb3a ... ov_regional 01OV008-308ad404-7079-4ff8-8232-12ee2e 64c2d710780496a4e184cb3a ... ov_regional 01OV008-308ad404-7079-4ff8-8232-12ee2e 64c2d710780496a4e184cb3a ... ov_regional 01OV008-308ad404-7079-4ff8-8232-12ee2e 64c2d710780496a4e184cb3a ... ov_regional 01OV008-308ad404-7079-4ff8-8232-12ee2e 64c2d710780496a4e184cb3a ... ov_regional 01OV008-7579323e-2fae-43a9-b00f-a15c28 64c2d711780496a4e184cb46 ... ov_regional 01OV008-7579323e-2fae-43a9-b00f-a15c28 64c2d711780496a4e184cb46 ... ov_regional 01OV008-7579323e-2fae-43a9-b00f-a15c28 64c2d711780496a4e184cb46 ... ov_regional 01OV008-7579323e-2fae-43a9-b00f-a15c28 64c2d711780496a4e184cb46 ... ov_regional 01OV008-7579323e-2fae-43a9-b00f-a15c28 64c2d711780496a4e184cb46 ... ov_regional 01OV008-7579323e-2fae-43a9-b00f-a15c28 64c2d711780496a4e184cb46 ... ov_regional 01OV008-7579323e-2fae-43a9-b00f-a15c28 64c2d711780496a4e184cb46 ... ov_regional 01OV008-7579323e-2fae-43a9-b00f-a15c28 64c2d711780496a4e184cb46 ... ov_regional [37 rows x 41 columns] 2023-08-03 01:51:01.913 | INFO | luna.pathology.cli.dsa_annotation_etl:dsa_annotation_etl:178 - Created 5 geojsons, 0 points, and 32 polygons 2023-08-03 01:51:01.931 | DEBUG | luna.common.utils:wrapper:146 - cli ran in 1.0s
To check that the regional annotation ETL was correctly run, you can examine the regional annotations table. This table contains the metadata saved by the ETL. It includes paths to the bitmap files, numpy files, and geoJSON files that were mentioned before. To load the table, run the following cell,
In [8]:
Copied!
from pyarrow.parquet import read_table
annotations_table = f"{TABLE_DIR}/ANNOTATIONS/slide_annotation_dataset_TCGA collection_ov_regional.parquet"
pd.read_parquet(annotations_table)
from pyarrow.parquet import read_table
annotations_table = f"{TABLE_DIR}/ANNOTATIONS/slide_annotation_dataset_TCGA collection_ov_regional.parquet"
pd.read_parquet(annotations_table)
Out[8]:
| _id | baseParentId | baseParentType | created | creatorId | description | folderId | largeImage | lowerName | name | ... | xmax | ymin | ymax | bbox_area | x_coords | y_coords | slide_geojson | collection_uuid | collection_name | annotation_name | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| slide_id | |||||||||||||||||||||
| 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 64c2d70a780496a4e184cb18 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:54.271000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70b780496a4e184cb1a', 'source... | 01ov002-bd8cdc70-3d46-40ae-99c4-90ef77.svs | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs | ... | 28661.0 | 40529.0 | 44372.0 | 13108473.0 | [28211, 28328, 28587, 28630, 28655, 28661, 286... | [42225, 42546, 43126, 43261, 43379, 43607, 437... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 64c2d70a780496a4e184cb18 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:54.271000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70b780496a4e184cb1a', 'source... | 01ov002-bd8cdc70-3d46-40ae-99c4-90ef77.svs | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs | ... | 33932.0 | 35713.0 | 39793.0 | 9792000.0 | [32252, 32252, 32220, 32164, 32140, 32108, 320... | [37097, 37001, 36897, 36745, 36713, 36689, 366... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 64c2d70a780496a4e184cb18 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:54.271000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70b780496a4e184cb1a', 'source... | 01ov002-bd8cdc70-3d46-40ae-99c4-90ef77.svs | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs | ... | 25542.0 | 18922.0 | 21735.0 | 5583805.0 | [24180, 24136, 24063, 23980, 23874, 23813, 237... | [18972, 18978, 19017, 19078, 19161, 19217, 192... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 64c2d70a780496a4e184cb18 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:54.271000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70b780496a4e184cb1a', 'source... | 01ov002-bd8cdc70-3d46-40ae-99c4-90ef77.svs | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs | ... | 30651.0 | 23202.0 | 26990.0 | 14015600.0 | [30133, 30073, 30005, 29951, 29904, 29857, 298... | [26411, 26465, 26506, 26532, 26553, 26566, 265... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 64c2d70a780496a4e184cb18 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:54.271000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70b780496a4e184cb1a', 'source... | 01ov002-bd8cdc70-3d46-40ae-99c4-90ef77.svs | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs | ... | 17365.0 | 26776.0 | 29672.0 | 11097472.0 | [16525, 16597, 16701, 16781, 16845, 16901, 169... | [27088, 26992, 26896, 26840, 26816, 26808, 267... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 64c2d70a780496a4e184cb18 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:54.271000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70b780496a4e184cb1a', 'source... | 01ov002-bd8cdc70-3d46-40ae-99c4-90ef77.svs | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs | ... | 23500.0 | 16457.0 | 19848.0 | 6921031.0 | [23389, 23435, 23463, 23481, 23500, 23500, 234... | [18929, 19021, 19103, 19186, 19250, 19425, 194... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | 64c2d70a780496a4e184cb18 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:54.271000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70b780496a4e184cb1a', 'source... | 01ov002-bd8cdc70-3d46-40ae-99c4-90ef77.svs | 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77.svs | ... | NaN | NaN | NaN | NaN | None | None | /home/pollardw/vmount/PRO-12-123/data/toy_data... | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV002-ed65cf94-8bc6-492b-9149-adc16f | 64c2d70c780496a4e184cb24 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:56.498000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70d780496a4e184cb26', 'source... | 01ov002-ed65cf94-8bc6-492b-9149-adc16f.svs | 01OV002-ed65cf94-8bc6-492b-9149-adc16f.svs | ... | 25783.0 | 4477.0 | 8874.0 | 24473702.0 | [24143, 23829, 23663, 23611, 23532, 23436, 233... | [4477, 4477, 4503, 4521, 4556, 4591, 4608, 464... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV002-ed65cf94-8bc6-492b-9149-adc16f | 64c2d70c780496a4e184cb24 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:56.498000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70d780496a4e184cb26', 'source... | 01ov002-ed65cf94-8bc6-492b-9149-adc16f.svs | 01OV002-ed65cf94-8bc6-492b-9149-adc16f.svs | ... | 19778.0 | 4925.0 | 8728.0 | 19376285.0 | [19647, 19691, 19735, 19778, 19778, 19770, 197... | [5125, 5370, 5605, 5876, 6076, 6190, 6268, 634... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV002-ed65cf94-8bc6-492b-9149-adc16f | 64c2d70c780496a4e184cb24 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:56.498000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70d780496a4e184cb26', 'source... | 01ov002-ed65cf94-8bc6-492b-9149-adc16f.svs | 01OV002-ed65cf94-8bc6-492b-9149-adc16f.svs | ... | 12487.0 | 28794.0 | 34122.0 | 11972016.0 | [10388, 10374, 10334, 10294, 10253, 10240, 102... | [30395, 30328, 30207, 30059, 29938, 29830, 296... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV002-ed65cf94-8bc6-492b-9149-adc16f | 64c2d70c780496a4e184cb24 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:56.498000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70d780496a4e184cb26', 'source... | 01ov002-ed65cf94-8bc6-492b-9149-adc16f.svs | 01OV002-ed65cf94-8bc6-492b-9149-adc16f.svs | ... | 17692.0 | 10030.0 | 16216.0 | 37821204.0 | [15745, 15904, 16062, 16235, 16351, 16452, 165... | [10073, 10059, 10030, 10030, 10073, 10117, 101... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV002-ed65cf94-8bc6-492b-9149-adc16f | 64c2d70c780496a4e184cb24 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:56.498000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70d780496a4e184cb26', 'source... | 01ov002-ed65cf94-8bc6-492b-9149-adc16f.svs | 01OV002-ed65cf94-8bc6-492b-9149-adc16f.svs | ... | 15544.0 | 29416.0 | 33936.0 | 14138560.0 | [12928, 12872, 12808, 12776, 12728, 12704, 126... | [29472, 29504, 29568, 29608, 29672, 29712, 298... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV002-ed65cf94-8bc6-492b-9149-adc16f | 64c2d70c780496a4e184cb24 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:56.498000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70d780496a4e184cb26', 'source... | 01ov002-ed65cf94-8bc6-492b-9149-adc16f.svs | 01OV002-ed65cf94-8bc6-492b-9149-adc16f.svs | ... | NaN | NaN | NaN | NaN | None | None | /home/pollardw/vmount/PRO-12-123/data/toy_data... | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 | 64c2d70e780496a4e184cb2e | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:58.287000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70f780496a4e184cb30', 'source... | 01ov007-9b90eb78-2f50-4aeb-b010-d642f9.svs | 01OV007-9b90eb78-2f50-4aeb-b010-d642f9.svs | ... | 14698.0 | 38558.0 | 42478.0 | 30419200.0 | [13850, 13762, 13682, 13626, 13570, 13490, 134... | [41438, 41478, 41518, 41542, 41582, 41614, 416... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 | 64c2d70e780496a4e184cb2e | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:58.287000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70f780496a4e184cb30', 'source... | 01ov007-9b90eb78-2f50-4aeb-b010-d642f9.svs | 01OV007-9b90eb78-2f50-4aeb-b010-d642f9.svs | ... | 90028.0 | 17627.0 | 20103.0 | 5407584.0 | [88952, 88912, 88848, 88792, 88752, 88704, 886... | [17667, 17647, 17631, 17627, 17627, 17639, 176... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 | 64c2d70e780496a4e184cb2e | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:58.287000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70f780496a4e184cb30', 'source... | 01ov007-9b90eb78-2f50-4aeb-b010-d642f9.svs | 01OV007-9b90eb78-2f50-4aeb-b010-d642f9.svs | ... | 87377.0 | 27279.0 | 29663.0 | 3869232.0 | [86674, 86674, 86668, 86633, 86621, 86598, 865... | [27291, 27326, 27402, 27724, 27771, 27894, 279... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 | 64c2d70e780496a4e184cb2e | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:58.287000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70f780496a4e184cb30', 'source... | 01ov007-9b90eb78-2f50-4aeb-b010-d642f9.svs | 01OV007-9b90eb78-2f50-4aeb-b010-d642f9.svs | ... | 97400.0 | 26598.0 | 28177.0 | 2006909.0 | [96999, 97030, 97061, 97098, 97141, 97178, 972... | [26604, 26610, 26635, 26659, 26696, 26739, 268... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 | 64c2d70e780496a4e184cb2e | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:58.287000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70f780496a4e184cb30', 'source... | 01ov007-9b90eb78-2f50-4aeb-b010-d642f9.svs | 01OV007-9b90eb78-2f50-4aeb-b010-d642f9.svs | ... | 15718.0 | 14948.0 | 17225.0 | 4378671.0 | [15605, 15605, 15605, 15616, 15627, 15632, 156... | [16231, 16285, 16446, 16575, 16661, 16688, 167... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 | 64c2d70e780496a4e184cb2e | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:58.287000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70f780496a4e184cb30', 'source... | 01ov007-9b90eb78-2f50-4aeb-b010-d642f9.svs | 01OV007-9b90eb78-2f50-4aeb-b010-d642f9.svs | ... | 13059.0 | 16566.0 | 18370.0 | 1405316.0 | [12688, 12721, 12764, 12791, 12817, 12844, 128... | [16850, 16888, 16936, 16974, 17017, 17049, 171... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 | 64c2d70e780496a4e184cb2e | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:58.287000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70f780496a4e184cb30', 'source... | 01ov007-9b90eb78-2f50-4aeb-b010-d642f9.svs | 01OV007-9b90eb78-2f50-4aeb-b010-d642f9.svs | ... | 7571.0 | 14199.0 | 15951.0 | 2298624.0 | [7039, 7015, 6963, 6859, 6739, 6679, 6639, 660... | [14211, 14211, 14231, 14275, 14335, 14371, 144... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 | 64c2d70e780496a4e184cb2e | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:43:58.287000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d70f780496a4e184cb30', 'source... | 01ov007-9b90eb78-2f50-4aeb-b010-d642f9.svs | 01OV007-9b90eb78-2f50-4aeb-b010-d642f9.svs | ... | NaN | NaN | NaN | NaN | None | None | /home/pollardw/vmount/PRO-12-123/data/toy_data... | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-308ad404-7079-4ff8-8232-12ee2e | 64c2d710780496a4e184cb3a | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:00.156000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d711780496a4e184cb3c', 'source... | 01ov008-308ad404-7079-4ff8-8232-12ee2e.svs | 01OV008-308ad404-7079-4ff8-8232-12ee2e.svs | ... | 26787.0 | 7561.0 | 10402.0 | 3096690.0 | [26095, 25925, 25819, 25786, 25754, 25721, 257... | [7561, 7569, 7634, 7683, 7740, 7813, 7862, 794... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-308ad404-7079-4ff8-8232-12ee2e | 64c2d710780496a4e184cb3a | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:00.156000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d711780496a4e184cb3c', 'source... | 01ov008-308ad404-7079-4ff8-8232-12ee2e.svs | 01OV008-308ad404-7079-4ff8-8232-12ee2e.svs | ... | 21863.0 | 8532.0 | 11072.0 | 2377440.0 | [21846, 21863, 21863, 21846, 21822, 21806, 217... | [9151, 9224, 9273, 9371, 9452, 9501, 9542, 959... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-308ad404-7079-4ff8-8232-12ee2e | 64c2d710780496a4e184cb3a | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:00.156000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d711780496a4e184cb3c', 'source... | 01ov008-308ad404-7079-4ff8-8232-12ee2e.svs | 01OV008-308ad404-7079-4ff8-8232-12ee2e.svs | ... | 38687.0 | 18301.0 | 19409.0 | 620480.0 | [38411, 38419, 38439, 38459, 38469, 38481, 385... | [18675, 18637, 18573, 18533, 18519, 18507, 184... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-308ad404-7079-4ff8-8232-12ee2e | 64c2d710780496a4e184cb3a | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:00.156000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d711780496a4e184cb3c', 'source... | 01ov008-308ad404-7079-4ff8-8232-12ee2e.svs | 01OV008-308ad404-7079-4ff8-8232-12ee2e.svs | ... | 38443.0 | 21701.0 | 23747.0 | 6187104.0 | [38390, 38362, 38326, 38229, 38197, 38145, 379... | [23685, 23701, 23712, 23735, 23739, 23747, 237... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-308ad404-7079-4ff8-8232-12ee2e | 64c2d710780496a4e184cb3a | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:00.156000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d711780496a4e184cb3c', 'source... | 01ov008-308ad404-7079-4ff8-8232-12ee2e.svs | 01OV008-308ad404-7079-4ff8-8232-12ee2e.svs | ... | 30451.0 | 14502.0 | 26079.0 | 100279974.0 | [29241, 28934, 28506, 27955, 27741, 27557, 274... | [24267, 24536, 24804, 25071, 25190, 25309, 254... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-308ad404-7079-4ff8-8232-12ee2e | 64c2d710780496a4e184cb3a | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:00.156000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d711780496a4e184cb3c', 'source... | 01ov008-308ad404-7079-4ff8-8232-12ee2e.svs | 01OV008-308ad404-7079-4ff8-8232-12ee2e.svs | ... | 16345.0 | 15568.0 | 16623.0 | 1446405.0 | [15784, 15840, 15909, 15949, 15973, 15993, 160... | [15684, 15673, 15662, 15658, 15651, 15647, 156... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-308ad404-7079-4ff8-8232-12ee2e | 64c2d710780496a4e184cb3a | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:00.156000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d711780496a4e184cb3c', 'source... | 01ov008-308ad404-7079-4ff8-8232-12ee2e.svs | 01OV008-308ad404-7079-4ff8-8232-12ee2e.svs | ... | 24735.0 | 15599.0 | 17067.0 | 2058136.0 | [24707, 24700, 24634, 24544, 24508, 24485, 244... | [16018, 15990, 15821, 15656, 15620, 15607, 155... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-308ad404-7079-4ff8-8232-12ee2e | 64c2d710780496a4e184cb3a | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:00.156000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d711780496a4e184cb3c', 'source... | 01ov008-308ad404-7079-4ff8-8232-12ee2e.svs | 01OV008-308ad404-7079-4ff8-8232-12ee2e.svs | ... | NaN | NaN | NaN | NaN | None | None | /home/pollardw/vmount/PRO-12-123/data/toy_data... | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-7579323e-2fae-43a9-b00f-a15c28 | 64c2d711780496a4e184cb46 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:01.860000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d712780496a4e184cb48', 'source... | 01ov008-7579323e-2fae-43a9-b00f-a15c28.svs | 01OV008-7579323e-2fae-43a9-b00f-a15c28.svs | ... | 28755.0 | 19860.0 | 21427.0 | 6387092.0 | [25997, 25965, 25932, 25876, 25856, 25836, 256... | [19938, 19923, 19899, 19872, 19864, 19860, 198... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-7579323e-2fae-43a9-b00f-a15c28 | 64c2d711780496a4e184cb46 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:01.860000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d712780496a4e184cb48', 'source... | 01ov008-7579323e-2fae-43a9-b00f-a15c28.svs | 01OV008-7579323e-2fae-43a9-b00f-a15c28.svs | ... | 15095.0 | 17451.0 | 19701.0 | 3636000.0 | [14583, 14511, 14384, 14256, 14074, 14026, 139... | [17451, 17460, 17501, 17551, 17658, 17690, 177... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-7579323e-2fae-43a9-b00f-a15c28 | 64c2d711780496a4e184cb46 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:01.860000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d712780496a4e184cb48', 'source... | 01ov008-7579323e-2fae-43a9-b00f-a15c28.svs | 01OV008-7579323e-2fae-43a9-b00f-a15c28.svs | ... | 20765.0 | 13531.0 | 15375.0 | 3444592.0 | [20485, 20513, 20549, 20577, 20601, 20621, 206... | [14427, 14431, 14451, 14471, 14495, 14523, 145... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-7579323e-2fae-43a9-b00f-a15c28 | 64c2d711780496a4e184cb46 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:01.860000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d712780496a4e184cb48', 'source... | 01ov008-7579323e-2fae-43a9-b00f-a15c28.svs | 01OV008-7579323e-2fae-43a9-b00f-a15c28.svs | ... | 17465.0 | 11547.0 | 13601.0 | 3037866.0 | [16776, 16766, 16756, 16730, 16704, 16678, 166... | [11851, 11812, 11772, 11726, 11695, 11656, 116... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-7579323e-2fae-43a9-b00f-a15c28 | 64c2d711780496a4e184cb46 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:01.860000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d712780496a4e184cb48', 'source... | 01ov008-7579323e-2fae-43a9-b00f-a15c28.svs | 01OV008-7579323e-2fae-43a9-b00f-a15c28.svs | ... | 12733.0 | 26212.0 | 27987.0 | 2254250.0 | [12443, 12462, 12494, 12514, 12542, 12599, 126... | [26686, 26654, 26642, 26637, 26641, 26668, 267... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-7579323e-2fae-43a9-b00f-a15c28 | 64c2d711780496a4e184cb46 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:01.860000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d712780496a4e184cb48', 'source... | 01ov008-7579323e-2fae-43a9-b00f-a15c28.svs | 01OV008-7579323e-2fae-43a9-b00f-a15c28.svs | ... | 22424.0 | 12780.0 | 13898.0 | 1357252.0 | [22006, 21967, 21911, 21859, 21826, 21821, 218... | [13625, 13621, 13587, 13539, 13492, 13480, 134... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-7579323e-2fae-43a9-b00f-a15c28 | 64c2d711780496a4e184cb46 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:01.860000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d712780496a4e184cb48', 'source... | 01ov008-7579323e-2fae-43a9-b00f-a15c28.svs | 01OV008-7579323e-2fae-43a9-b00f-a15c28.svs | ... | 21173.0 | 13423.0 | 14415.0 | 1126912.0 | [20741, 20725, 20677, 20621, 20577, 20561, 205... | [13671, 13651, 13615, 13579, 13555, 13539, 135... | None | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional | |
| 01OV008-7579323e-2fae-43a9-b00f-a15c28 | 64c2d711780496a4e184cb46 | 64c2d70a780496a4e184cb16 | collection | 2023-07-27T20:44:01.860000+00:00 | 64c2d709780496a4e184cb12 | 64c2d70a780496a4e184cb17 | {'fileId': '64c2d712780496a4e184cb48', 'source... | 01ov008-7579323e-2fae-43a9-b00f-a15c28.svs | 01OV008-7579323e-2fae-43a9-b00f-a15c28.svs | ... | NaN | NaN | NaN | NaN | None | None | /home/pollardw/vmount/PRO-12-123/data/toy_data... | 64c2d70a780496a4e184cb16 | TCGA collection | ov_regional |
37 rows × 41 columns
Lastly, lets get our geojsons and join on slide ID.
In [9]:
Copied!
pd.read_parquet(annotations_table) \
.query("type=='geojson'")[['slide_geojson']] \
.join(
pd.read_parquet(f"{TABLE_DIR}/SLIDES/slide_ingest_PRO-12-123.parquet")['id']
)
pd.read_parquet(annotations_table) \
.query("type=='geojson'")[['slide_geojson']] \
.join(
pd.read_parquet(f"{TABLE_DIR}/SLIDES/slide_ingest_PRO-12-123.parquet")['id']
)
Out[9]:
| slide_geojson | id | |
|---|---|---|
| slide_id | ||
| 01OV002-bd8cdc70-3d46-40ae-99c4-90ef77 | /home/pollardw/vmount/PRO-12-123/data/toy_data... | NaN |
| 01OV002-ed65cf94-8bc6-492b-9149-adc16f | /home/pollardw/vmount/PRO-12-123/data/toy_data... | NaN |
| 01OV007-9b90eb78-2f50-4aeb-b010-d642f9 | /home/pollardw/vmount/PRO-12-123/data/toy_data... | NaN |
| 01OV008-308ad404-7079-4ff8-8232-12ee2e | /home/pollardw/vmount/PRO-12-123/data/toy_data... | NaN |
| 01OV008-7579323e-2fae-43a9-b00f-a15c28 | /home/pollardw/vmount/PRO-12-123/data/toy_data... | NaN |
At this point, you have successfully set up your workspace, downloaded the data, and run both the pathology and regional annotation ETLs to prepare your data. You are ready to move on to the tiling notebook!