According to Jubilees, Er sinned by refusing to sleep with Tamar:

(1) And in the forty-fifth jubilee, in the second week, (and) in the second year, Judah took for his first-born Er, a wife from the daughters of Aram, named Tamar. (2) But he hated, and did not lie with her, because his mother was of the daughters of Canaan, and he wished to take him a wife of the kinsfolk of his mother, but Judah, his father, would not permit him. (3) And this Er, the first-born of Judah, was wicked, and the Lord slew him. ( Jubilees 41:1-3 )

According to Rabbinic sources, Er did sleep with Tamar, but avoided impregnating her, in order to maintain her beauty:

ויהי ער בכור יהודה רע בעיני ה', שהיה חורש בגנות ומערה לאשפות. (Bereshit Rabbah 85:4)

ער ואונן שמשו שלא כדרכן מיתיבי כל עשרים וארבעה חדש דש מבפנים וזורה מבחוץ דברי ר' אליעזר א"ל הללו אינו אלא כמעשה ער ואונן כמעשה ער ואונן ולא כמעשה ער ואונן כמעשה ער ואונן דכתיב (בראשית לח, ט) והיה אם בא אל אשת אחיו ושחת ארצה ולא כמעשה ער ואונן דאילו התם שלא כדרכה והכא כדרכה בשלמא אונן דכתיב ביה ושחת ארצה אלא ער מנלן אמר רב נחמן בר יצחק דכתיב (בראשית לח, י) וימת גם אותו אף הוא באותו מיתה מת בשלמא אונן משום לא לו יהיה הזרע אלא ער מ"ט עבד הכי כדי שלא תתעבר ויכחיש יפיה. (Bavli Yevamot 34b)

Er and Onan indulged in unnatural intercourse. An objection was raised: During all the twenty-four months one may thresh within and winnow without; these are the words of R. Eliezer. The others said to him: Such actions are only like the practice of Er and Onan! -Like the practice of Er and Onan, and yet not [exactly] like the practice of Er and Onan: 'Like the practice of Er and Onan', for it is written in Scripture, And it came to pass, when he went in unto his brother's wife, that he spilt it on the ground; and 'not [exactly] like the practice of Er and Onan', for whereas there it was an unnatural act, here it is done in the natural way. [The source for] Onan's [guilt] may well be traced, for it is written in Scripture, That he spilt it on the ground; whence however, [that of] Er? — R. Nahman b. Isaac replied: It is written, And He slew him also, he also died of the same death. [The reason for] Onan's [action] may well be understood, because he knew That the seed would not be his; but why did Er act in such a manner? — In order that she might not conceive and thus lose some of her beauty. (Bavli Yevamot 34b)

וַהֲוָה עֵר בּוּכְרָא דִיהוּדָה בִּישׁ קֳדָם יְיָ עַל דְלָא הֲוָה יָהִיב זַרְעֵיהּ לְאִנְתְּתֵיהּ וּתְקֵיף רוּגְזָא דַיְיָ עֲלֵיהּ וְקַטְלֵיהּ יְיָ. (Targum Pseudo-Jonathan Bereshit 38:7)

But Er the firstborn of Jehuda was evil before the Lord because he had not given his seed unto his wife, and the anger of the Lord prevailed against him, and the Lord slew him. (Targum Pseudo-Jonathan Bereshit 38:7)

רע בעיני ה' – כרעתו של אונן משחית זרעו שנאמר באונן וימת גם אותו כמיתתו של ער מיתתו של אונן ולמה היה ער משחית זרעו כדי שלא תתעבר ויכחיש יפיה. (Rashi Bereshit 38:7)

Was evil in the eyes of the Lord – [His evil was] like the evil of Onan, viz. that he wasted his semen, as it is written in connection with Onan: “and He put him to death also,” meaning that, as Er’s death, so was Onan’s death. Now, why should Er waste his semen? So that she (Tamar) would not become pregnant and her beauty be impaired. (Rashi Bereshit 38:7)