$\begingroup$

We can solve this integral using only substitutions and integration by parts, as follows: $$I:=\int_0^1 \frac{\operatorname{arctanh} (\beta t) }{t\sqrt{(1-t)(1- \alpha t)}}dt=\int_0^1 \frac{\operatorname{arctanh}(\beta t)}{t(1-t)}\sqrt{\frac{1-t}{1-\alpha t}}dt$$ $$\overset{\large \frac{1-t}{1-\alpha t}=x}=\int_0^1 \frac{\operatorname{arctanh}\left(\beta \frac{1-x}{1-\alpha x}\right)}{\sqrt x(1-x)}dx\overset{x=y^2}=2\int_0^1 \frac{\operatorname{arctanh}\left(\beta \frac{1-y^2}{1-\alpha y^2}\right)}{1-y^2}dy$$ $$\overset{\large y=\frac{1-x}{1+x}}=\int_0^1 \operatorname{arctanh}\left( \frac{4\beta x}{(1+x)^2-\alpha (1-x)^2}\right)\frac{dx}{x}=\frac12 \int_0^1 \ln\left(\frac{\left(ab+x\right)\left(\frac{1}{ab}+x\right)}{\left(\frac{a}{b}+x\right)\left(\frac{b}{a}+x\right)}\right)\frac{dx}{x}$$ $$\overset{IBP}=\frac12 \int_0^1 \ln x \left(\frac{1}{\frac{a}{b}+x}+\frac{1}{\frac{b}{a}+x}-\frac{1}{ab+x}-\frac{1}{\frac{1}{ab}+x}\right)dx$$

In each of the integral from above we will simplify the denominator using the substitution $x\to kx$, where $k$ is the constant found in each denominator.

$$\Rightarrow I=\frac12 \left(\int_0^\frac{b}{a}\frac{\ln\left(\frac{a}{b}x\right)}{1+x}dx+\int_0^\frac{a}{b}\frac{\ln\left(\frac{b}{a}x\right)}{1+x}dx-\int_0^\frac{1}{ab}\frac{\ln\left(ab x\right)}{1+x}dx-\int_0^{ab}\frac{\ln\left(\frac{x}{ab}\right)}{1+x}dx\right)$$ $$\small =\color{red}{\frac12} \left(\ln\left(\frac{a}{b}\right)\ln\left(1+\frac{b}{a}\right)+\ln\left(\frac{b}{a}\right)\ln\left(1+\frac{a}{b}\right)-\ln(ab)\ln\left(1+\frac{1}{ab}\right)-\ln\left(\frac{1}{ab}\right)\ln\left(1+ab\right)\right)$$ $$+\color{chocolate}{\frac12}\left(\int_0^\frac{b}{a}\frac{\ln x}{1+x}dx+\int_0^\frac{a}{b}\frac{\ln x}{1+x}dx-\int_0^\frac{1}{ab}\frac{\ln x}{1+x}dx-\int_0^{ab}\frac{\ln x}{1+x}dx\right)$$ We can also rewrite the four integrals from above as: $$\color{blue}{\int_\frac{1}{ab}^\frac{b}{a}\frac{\ln x}{1+x}dx}+\int_{ab}^\frac{a}{b}\frac{\ln x}{1+x}dx\overset{\color{blue}{x\to \frac{1}{x}}}=\color{blue}{\int_{ab}^\frac{a}{b}\frac{\ln x}{x}dx-\int_{ab}^\frac{a}{b}\frac{\ln x}{1+x}dx}+\int_{ab}^\frac{a}{b}\frac{\ln x}{1+x}dx$$ $$=\int_{ab}^\frac{a}{b}\frac{\ln x}{x}dx=\frac{\ln^2 x}{2}\bigg|_{ab}^\frac{a}{b}=-2\ln a\ln b$$ So with some algebra for the first term we finally get: $$I=\color{red}{\frac12}\left(4\ln a \ln b\right)+\color{chocolate}{\frac12}\left(-2\ln a \ln b\right)=\boxed{\ln a\ln b}$$

An alternative approach using Feynman's trick can be found here, which shows: $$\int_0^1 \ln\left(\frac{\left(ab+x\right)\left(\frac{1}{ab}+x\right)}{\left(\frac{a}{b}+x\right)\left(\frac{b}{a}+x\right)}\right)\frac{dx}{x}=2\ln a\ln b$$ It might be useful in the future so I'll also mention that, since $\int_0^1 \frac{\ln x}{t+x}dx=\operatorname{Li}_2\left(-\frac{1}{t}\right)$ the following Dilogarithm identity arises from above: $$\boxed{\operatorname{Li}_2\left(-\frac{a}{b}\right)+\operatorname{Li}_2\left(-\frac{b}{a}\right)-\operatorname{Li}_2\left(-ab\right)-\operatorname{Li}_2\left(-\frac{1}{ab}\right)=2\ln a\ln b;\ a,b>0}$$