KaneAI - World’s First E2E Software Testing Agent.Book a Demo Join TestMu Conference

How to use split_shard method in localstack

Best Python code snippet using localstack_python

dist_dataset.py

Source:dist_dataset.py

...38    def generate(self):39        if self.world_size == 1 or len(self.files) == 1:40            cur_dataloader_files = self.files41        else:42            cur_dataloader_files = split_shard(43                self.files, self.rank, self.world_size)44        while True:45            if self.shuffle:46                random.shuffle(cur_dataloader_files)47            worker_info = torch.utils.data.get_worker_info()48            if worker_info is not None:49                if len(cur_dataloader_files) % worker_info.num_workers != 0:50                    print('[DATA]--current dataloader %s file num %s cannot split to worker_num %s ' %51                             (self.rank, len(cur_dataloader_files), worker_info.num_workers))52                cur_worker_files = split_shard(53                    cur_dataloader_files, worker_info.id, worker_info.num_workers)54                if worker_info.id == 0:55                    print("[DataLoader] --> Rank:{}  Workers:[{} ~ {}][{}]  Size of process file:{}  ...".format(56                        self.rank, 0, worker_info.num_workers - 1, worker_info.id, len(cur_dataloader_files)))57            else:58                cur_worker_files = cur_dataloader_files59            if self.shuffle:60                random.shuffle(cur_worker_files)61            for filepath in cur_worker_files:62                if self.is_hdfs:63                    with hopen(filepath, 'r') as reader:64                        for line in reader:65                            yield line.decode()66                    continue67                with open(filepath, 'r') as reader:68                    for line in reader:69                        yield line70            if not self.repeat:71                break72    def __iter__(self):73        return self.generate()  74def split_shard(data: List[Any], shard_idx: int, shard_size: int):75    num = len(data)76    if num < shard_size:77        raise RuntimeError("num:{} < shard size:{}".format(num, shard_size))78    start_idx = (num * shard_idx) // shard_size79    end_idx = (num * (shard_idx + 1)) // shard_size...

Automation Testing Tutorials

Learn to execute automation testing from scratch with LambdaTest Learning Hub. Right from setting up the prerequisites to run your first automation test, to following best practices and diving deeper into advanced test scenarios. LambdaTest Learning Hubs compile a list of step-by-step guides to help you be proficient with different test automation frameworks i.e. Selenium, Cypress, TestNG etc.