TestMu 2025 - The Largest Virtual Software Testing Conference is Back | 50,000+ Attendees

How to use split_shard method in localstack

Best Python code snippet using localstack_python

dist_dataset.py

...38    def generate(self):39        if self.world_size == 1 or len(self.files) == 1:40            cur_dataloader_files = self.files41        else:42            cur_dataloader_files = split_shard(43                self.files, self.rank, self.world_size)44        while True:45            if self.shuffle:46                random.shuffle(cur_dataloader_files)47            worker_info = torch.utils.data.get_worker_info()48            if worker_info is not None:49                if len(cur_dataloader_files) % worker_info.num_workers != 0:50                    print('[DATA]--current dataloader %s file num %s cannot split to worker_num %s ' %51                             (self.rank, len(cur_dataloader_files), worker_info.num_workers))52                cur_worker_files = split_shard(53                    cur_dataloader_files, worker_info.id, worker_info.num_workers)54                if worker_info.id == 0:55                    print("[DataLoader] --> Rank:{}  Workers:[{} ~ {}][{}]  Size of process file:{}  ...".format(56                        self.rank, 0, worker_info.num_workers - 1, worker_info.id, len(cur_dataloader_files)))57            else:58                cur_worker_files = cur_dataloader_files59            if self.shuffle:60                random.shuffle(cur_worker_files)61            for filepath in cur_worker_files:62                if self.is_hdfs:63                    with hopen(filepath, 'r') as reader:64                        for line in reader:65                            yield line.decode()66                    continue67                with open(filepath, 'r') as reader:68                    for line in reader:69                        yield line70            if not self.repeat:71                break72    def __iter__(self):73        return self.generate()  74def split_shard(data: List[Any], shard_idx: int, shard_size: int):75    num = len(data)76    if num < shard_size:77        raise RuntimeError("num:{} < shard size:{}".format(num, shard_size))78    start_idx = (num * shard_idx) // shard_size79    end_idx = (num * (shard_idx + 1)) // shard_size...

Blogs

Check out the latest blogs from LambdaTest on this topic:

How To Use CSS Contain Property To Optimize Browsers

How to increase and maintain team motivation

The best agile teams are built from people who work together as one unit, where each team member has both the technical and the personal skills to allow the team to become self-organized, cross-functional, and self-motivated. These are all big words that I hear in almost every agile project. Still, the criteria to make a fantastic agile team are practically impossible to achieve without one major factor: motivation towards a common goal.

A Complete Guide To CSS Houdini

As a developer, checking the cross browser compatibility of your CSS properties is of utmost importance when building your website. I have often found myself excited to use a CSS feature only to discover that it’s still not supported on all browsers. Even if it is supported, the feature might be experimental and not work consistently across all browsers. Ask any front-end developer about using a CSS feature whose support is still in the experimental phase in most prominent web browsers. ????

Get A Seamless Digital Experience With #LambdaTestYourBusiness????

The holidays are just around the corner, and with Christmas and New Year celebrations coming up, everyone is busy preparing for the festivities! And during this busy time of year, LambdaTest also prepped something special for our beloved developers and testers – #LambdaTestYourBusiness

Digital transformation – part 1 – basics

Digital transformation - part 1 - basics

Automation Testing Tutorials

Learn to execute automation testing from scratch with LambdaTest Learning Hub. Right from setting up the prerequisites to run your first automation test, to following best practices and diving deeper into advanced test scenarios. LambdaTest Learning Hubs compile a list of step-by-step guides to help you be proficient with different test automation frameworks i.e. Selenium, Cypress, TestNG etc.