Hi, may I know:

1) How many steps did you trained?

2) What is the learning rate that you used?

3) Make sure that you have loaded the newly trained model instead of the base model

4) What are the prompt that you used for inference?

In theory, it will work even if train_text_encoder is disabled. This setting is mainly used to prevent overfitting.