Skip to content

B
Basedformer
Commit b1e335ec authored by novelailab's avatar novelailab
Browse files
Options

Don't check if identity is dict

parent dad308db
Hide whitespace changes
Inline Side-by-side
Showing with 0 additions and 2 deletions
basedformer/dataset.py
View file @ b1e335ec
...@@ -81,8 +81,6 @@ class ShardedImageDataset(data.Dataset): ...@@ -81,8 +81,6 @@ class ShardedImageDataset(data.Dataset):
self.pointer_lookup = {} self.pointer_lookup = {}
for t in self.index: for t in self.index:
offset, length, id = t offset, length, id = t
if type(id) == dict: # if id is identity
id = id['id']
self.pointer_lookup[id] = (offset, length) self.pointer_lookup[id] = (offset, length)
#make so metadata is shardable by world_size(num_gpus) #make so metadata is shardable by world_size(num_gpus)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment