The term "janky" is common in specific gaming communities and refers to using tactics that are bad or subpar. A specific example from Reddit:

So Reynad just climbed about 800 ranks in legend with this Echo of Medivh Mage deck... Janky as hell but he went 12-1 with it tonight.

A comment explaining the meaning of the term:

"Jank" in [Magic: The Gathering] has come to mean a card or deck which either isn't or doesn't appear to be good. (From "janky", which I think was borrowed from computer science slang.)

The computer science usage is typically something like, "This setup is janky" which is the example I have used in the title of the question. I have personally encountered this usage when referring to computers and various machines that seem to be falling apart or built from subpar pieces.

But where did this word come from?

"Janky" doesn't have an entry in all dictionaries and etymonline doesn't show anything interesting for "janky" or "jank".

Wikitionary also mentions a connection to computer science slang:

(jargon, computing, rare) Unresponsive (of a software application’s user interface), sluggish.

But I am still not seeing anything particularly revealing about this word's etymology.