- Images organised in class folders.<\/li>
- Images data organised as CSV file.<\/li>
- Images data organised in a folder with image name as class.<\/li><\/ul>\n\n\n\n
We will be using example cat and dog datasets to code custom datasets.<\/p>\n\n\n\n
Images organised in class folders<\/h2>\n\n\n\n
\n![\"\"](\"https:\/\/code-ml.com\/wp-content\/uploads\/2021\/09\/undraw_Image__folder_re_hgp7.png\")
Ref<\/figcaption><\/figure><\/div>\n<\/div><\/div>\n\n\n\n Before actually jumping into Dataloader, let’s first take a look at how our folder structure should be organized for this part. As you can below, the dataset is divided into class folders. <\/p>\n\n\n\n
\n\n\n\n\n\u2500\u2500\u2500 train\n \u251c\u2500\u2500 cat\n \u2502 \u251c\u2500\u2500 cat1.png\n \u2502 \u251c\u2500\u2500 cat2.png\n \u2502 \u251c\u2500\u2500 ......\n \u2502 \u2514\u2500\u2500 cat100.png\n \u251c\u2500\u2500 dog\n \u251c\u2500\u2500 dog1.png\n \u251c\u2500\u2500 dog2.png\n \u251c\u2500\u2500 ......\n \u2514\u2500\u2500 dog100.png<\/code><\/pre>\n<\/div>\n\n\n\n\n\u2500\u2500\u2500 test\n \u251c\u2500\u2500 cat\n \u2502 \u251c\u2500\u2500 cat1.png\n \u2502 \u251c\u2500\u2500 cat2.png\n \u2502 \u251c\u2500\u2500 ......\n \u2502 \u2514\u2500\u2500 cat10.png\n \u251c\u2500\u2500 dog\n \u251c\u2500\u2500 dog1.png\n \u251c\u2500\u2500 dog2.png\n \u251c\u2500\u2500 ......\n \u2514\u2500\u2500 dog10.png<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div><\/div>\n<\/div>\n<\/div>\n\n\n\nPyTorch provides two data primitives:
torch.utils.data.DataLoader<\/strong><\/code> andtorch.utils.data.Dataset<\/strong><\/code> that allow you to use pre-loaded datasets as well as your own data.Dataset<\/code> stores the samples and their corresponding labels, and Data Loader wraps an iterable around the Dataset to enable easy access to the samples. Let us first write the code for custome Dataset Class and then we will explore each section:<\/p>\n\n\n\nimport os\nimport torch\nfrom PIL import Image\nfrom torchvision import transforms, datasets\n\nclass CustomDataset(torch.utils.data.Dataset):\n def __init__(self, root_dir):\n self.classnames=["cat", "dog"] # Class List\n self.root_dir = root_dir # Train or Test\n self.img_label_tuple = [] # Data list\n\n # Loop to extract image path and label for folder name\n for i in range(0, len(self.classnames)):\n class_dir = os.path.join(root_dir, self.classnames[i])\n all_files = [os.path.join(class_dir,f) for f in listdir(class_dir) if isfile(join(class_dir, f))]\n for j in all_files:\n item = tuple((j, i))\n self.img_label_tuple.append(item)\n\n # Simple transform\n self.transform = transforms.Compose([transforms.Resize((32,32)),\n transforms.ToTensor()])\n\n def __len__(self):\n # Returns lenght of the dataset.\n return len(self.img_label_tuple)\n\n # Contructur to process dataset, tranform it and returns (input image with label)\n def __getitem__(self, idx):\n path = self.img_label_tuple[idx][0]\n class_id = self.classnames.index(img_label_tuple[idx][1])\n img = Image.open(path).convert('RGB')\n img = self.transform(img)\n \n return (img, class_id)<\/pre><\/div>\n\n\n\nIn the above code, __init__()<\/strong> is known as a constructor which always gets executed when the object is called. In our CustomDataset, we initiate every Dataset with the class list, image_label_tuple list, and then we extract the dataset information from the given folder, In this way, we use the extracted data information to further process our image dataset. We have inherited the above custom class with the properties of torch.utils.data.Dataset, so that we can leverage features provided by this PyTorch’s class such as multiprocessing<\/em>. <\/p>\n\n\n\n
We also defined two other constructor __len__() <\/strong>and __getitem__()<\/strong>, here __len__()<\/strong> is use to get the length of the given dataset. Whereas __getitem__()<\/strong> is a little complex, here we can define specific pre-processing techniques to pre-process our images before passing them to our machine learning model. <\/p>\n\n\n\n
Now that we have created the custom Dataset class for the dataset, we can move ahead with creating an iterable using torch.utils.data.DataLoader<\/a><\/strong>. In order to create the iterable we need our custom Dataset class and a few arguments given below, learn about more arguments here<\/a>:<\/p>\n\n\n\n
- batch_size(int):<\/span><\/strong> Number of samples in each set of genearted batch.<\/li>
- shuffle(bool):<\/span><\/strong> If set True, the data will shuffle every time a batch is generated. Generally True for training.<\/li>
- num_worker:<\/span><\/strong> which denotes the number of processes that generate batches in parallel.<\/li><\/ul>\n\n\n\n
bs = 64 \t#Batch Size\n\nroot_dir = "Dataset folder path"\ndataset = CustomDataset(root_dir) # Dataset class initiation \ndataloader = torch.utils.data.DataLoader(dataset, bs, shuffle=True)<\/pre><\/div>\n\n\n\n
Images data organised as CSV file<\/h2>\n\n\n\n
![\"\"](\"https:\/\/code-ml.com\/wp-content\/uploads\/2021\/09\/undraw_Spreadsheet_re_cn18.png\")
<\/figure><\/div>\n\n\n\nIn many Kaggle competitions, we are given we data folder and one CSV file. This file often contains meta-data of the dataset, for this section, we are assuming only two columns are given 1.) image_file_path<\/strong> 2.) class_label<\/strong>.<\/p>\n\n\n\n
import os\nimport torch\nimport pandas as pd\nfrom PIL import Image\nfrom torchvision import transforms, datasets\n\nclass CustomDataset(torch.utils.data.Dataset):\n def __init__(self, root_dir):\n\n self.dataframe = pd.read_csv(root_dir) # Data frame from CSV file\n self.class_list = self.dataframe['class_label'].unique()\n # Simple transform\n self.transform = transforms.Compose([transforms.Resize((32,32)),\n transforms.ToTensor()])\n\n def __len__(self):\n # Returns lenght of the dataset.\n return len(self.dataframe)\n\n # Contructur to process dataset, tranform it and returns (input image with label)\n def __getitem__(self, idx):\n path = self.dataframe['img_path'].iloc[idx]\n class_id = self.class_list.index(self.dataframe['class_label'].iloc[idx])\n img = Image.open(path).convert('RGB')\n img = self.transform(img)\n \n return (img, class_id)<\/pre><\/div>\n\n\n\nSimilar to first section, torch.utils.data.DataLoader<\/strong> will be use to create dataloaders, shown in the code given below:<\/p>\n\n\n\n
bs = 64 \t#Batch Size\n\nroot_dir = "Dataset folder path"\ndataset = CustomDataset(root_dir) # Dataset class initiation \ndataloader = torch.utils.data.DataLoader(dataset, bs, shuffle=True)<\/pre><\/div>\n\n\n\n
Images data organised in a folder with image name as class.<\/h2>\n\n\n\n
![\"\"](\"https:\/\/code-ml.com\/wp-content\/uploads\/2021\/09\/undraw_folder_files_nweq.png\")
<\/figure><\/div>\n\n\n\nIn a case when we have the image name as their classes we can just extract the class label from the image name. Below is the folder structure of such a scenario.<\/p>\n\n\n\n
\u2500\u2500\u2500 train\n \u251c\u2500\u2500 cat_1.png\n \u251c\u2500\u2500 dog_2.png\n \u251c\u2500\u2500 cat_3.png\n \u251c\u2500\u2500 cat_4.png \n \u251c\u2500\u2500 dog_5.png \n \u251c\u2500\u2500 cat_6.png\n \u251c\u2500\u2500 ......\n \u2514\u2500\u2500 dog_100.png<\/code><\/pre>\n\n\n\nSimilar to the above approach we will be creating a CustomDataset for the above folder structure.<\/p>\n\n\n\n
import os\nimport torch\nimport pandas as pd\nfrom PIL import Image\nfrom torchvision import transforms, datasets\n\nclass CustomDataset(torch.utils.data.Dataset):\n def __init__(self, root_dir):\n self.classnames=["cat", "dog"] # Class List\n self.root_dir = root_dir # Train or Test\n self.img_path_list = [] # Data list\n\n # Loop to extract image path\n all_files = [os.path.join(self.root_dir,f) for f in listdir(self.root_dir) if isfile(join(self.root_dir, f))]\n for item in all_files:\n self.img_path_list.append(item)\n\n # Simple transform\n self.transform = transforms.Compose([transforms.Resize((32,32)),\n transforms.ToTensor()])\n\n def __len__(self):\n # Returns lenght of the dataset.\n return len(self.img_path_list)\n\n # Contructur to process dataset, tranform it and returns (input image with label)\n def __getitem__(self, idx):\n path = self.img_path_list[idx]\n class_id = self.classnames.index(path.split("_")[0])\n img = Image.open(path).convert('RGB')\n img = self.transform(img)\n \n return (img, class_id)<\/pre><\/div>\n\n\n\nIn __getitem__()<\/strong>, we simply extracted the class label name by splitting the pathname of the image. After creation of custom Dataset class. We are ready to create DataLoader again for the third time, <\/p>\n\n\n\n
bs = 64 \t#Batch Size\n\nroot_dir = "Dataset folder path"\ndataset = CustomDataset(root_dir) # Dataset class initiation \ndataloader = torch.utils.data.DataLoader(dataset, bs, shuffle=True)<\/pre><\/div>\n\n\n\n
Conclution<\/h2>\n\n\n\n
In this article, we learned about Custom Dataset and DataLoader and how to create them in PyTorch. We also explored three different ways in which our data could be organized and how to create a Custom dataset for each set of scenarios. Hope you like the article and learned something, do share with one who needs such content. Read more articles here<\/a>. <\/p>\n\n\n\n
References <\/h5>\n\n\n\n
- https:\/\/pytorch.org\/docs\/stable\/data.html <\/a><\/li>
- https:\/\/pytorch.org\/tutorials\/beginner\/basics\/data_tutorial.html<\/a><\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"
With the advance in Deep learning, working with huge datasets has increasingly become part of the Data Science pipeline. Particularly in computer vision problems, we have a visual dataset that takes a huge amount of memory space to process and augment. Also, loading such a huge dataset takes a lot of time to process the […]<\/p>\n","protected":false},"author":1,"featured_media":3196,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0},"categories":[1],"tags":[24,25,23,22],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":6}},"aioseo_notices":[],"featured_image_urls":{"full":["https:\/\/codeml.ai\/wp-content\/uploads\/2021\/09\/undraw_Image__folder_re_hgp7.png",1005,807,false],"thumbnail":["https:\/\/codeml.ai\/wp-content\/uploads\/2021\/09\/undraw_Image__folder_re_hgp7-150x150.png",150,150,true],"medium":["https:\/\/codeml.ai\/wp-content\/uploads\/2021\/09\/undraw_Image__folder_re_hgp7-300x241.png",300,241,true],"medium_large":["https:\/\/codeml.ai\/wp-content\/uploads\/2021\/09\/undraw_Image__folder_re_hgp7-768x617.png",768,617,true],"large":["https:\/\/codeml.ai\/wp-content\/uploads\/2021\/09\/undraw_Image__folder_re_hgp7.png",1005,807,false],"1536x1536":["https:\/\/codeml.ai\/wp-content\/uploads\/2021\/09\/undraw_Image__folder_re_hgp7.png",1005,807,false],"2048x2048":["https:\/\/codeml.ai\/wp-content\/uploads\/2021\/09\/undraw_Image__folder_re_hgp7.png",1005,807,false]},"post_excerpt_stackable":"
With the advance in Deep learning, working with huge datasets has increasingly become part of the Data Science pipeline. Particularly in computer vision problems, we have a visual dataset that takes a huge amount of memory space to process and augment. Also, loading such a huge dataset takes a lot of time to process the dataset, and programs often slow down. To meet the requirement of processing a huge dataset we need an efficient way to process our dataset so that we have better memory and time management. The PyTorch framework provides a well-defined schema to process our dataset more…<\/p>\n","category_list":"How to. ?<\/a>","author_info":{"name":"Mohammad Ahmad","url":"https:\/\/codeml.ai\/index.php\/author\/md-ahmad0652\/"},"comments_num":"0 comments","_links":{"self":[{"href":"https:\/\/codeml.ai\/index.php\/wp-json\/wp\/v2\/posts\/124"}],"collection":[{"href":"https:\/\/codeml.ai\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/codeml.ai\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/codeml.ai\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/codeml.ai\/index.php\/wp-json\/wp\/v2\/comments?post=124"}],"version-history":[{"count":16,"href":"https:\/\/codeml.ai\/index.php\/wp-json\/wp\/v2\/posts\/124\/revisions"}],"predecessor-version":[{"id":3224,"href":"https:\/\/codeml.ai\/index.php\/wp-json\/wp\/v2\/posts\/124\/revisions\/3224"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/codeml.ai\/index.php\/wp-json\/wp\/v2\/media\/3196"}],"wp:attachment":[{"href":"https:\/\/codeml.ai\/index.php\/wp-json\/wp\/v2\/media?parent=124"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/codeml.ai\/index.php\/wp-json\/wp\/v2\/categories?post=124"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/codeml.ai\/index.php\/wp-json\/wp\/v2\/tags?post=124"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- https:\/\/pytorch.org\/tutorials\/beginner\/basics\/data_tutorial.html<\/a><\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"
- shuffle(bool):<\/span><\/strong> If set True, the data will shuffle every time a batch is generated. Generally True for training.<\/li>
- batch_size(int):<\/span><\/strong> Number of samples in each set of genearted batch.<\/li>