About the textdiffuser dataset

[Ruby-He]

Describe the bug
Model I am using textdiffuser, but I had problems when downloading the dataset laion-ocr

The problem arises when using:

img2dataset --url_list="XX/laion_ocr_image_url/laion-ocr-url.txt" --output_folder=laion_ocr --thread_count=64 --resize_mode=no

I found that the index of the downloaded image and the index of the label information in the compressed package laion-ocr.zip do not match.

As shown in the figure below:

why the same key corresponds to different image url ?

[JingyeChen]

Hi, thanks for your interest in TextDiffuser. Does this issue exist in few cases? Sometimes the image owner may replace the image, resulting in the mismatch of downloaded image.

[Ruby-He]

Hi, thanks for your interest in TextDiffuser. Does this issue exist in few cases? Sometimes the image owner may replace the image, resulting in the mismatch of downloaded image.

Finally, I have to use the index you provided in “laion-ocr-index-url.txt” and write a new script to move and rename the image, so that can match the annotaions.

Another question is could I resize the image directly in the "train.py/inference.py", such as
image = Image.open(image_path).convert("RGB")
image = image.resize((512,512))