Fraudulent retail transactions are on the rise, especially as more shoppers turn to the internet to ring up purchases, making tracking consumers more onerous and costly.

The level of fraud as a percentage of retailers' revenues has climbed to 1.58 percent year to date from 1.47 percent a year ago, according to a study by LexisNexis Risk Solutions, an Atlanta-based data analytics provider. The group surveyed more than 650 risk and fraud executives from retail organizations.

"Merchants selling physical and digital goods often apply a one-size-fits-all program to fight fraud, and they use a limited set of solutions," Kimberly Sutherland, a senior director of LexisNexis' fraud and identity management strategy, said in a statement.

"These less advanced and less sophisticated legacy solutions do not appear to be working, given the sharp rise in costs and volume of successful fraud attempts."

Driving the increases in fraud for many retailers this year are more international transactions and new payment options, such as mobile wallets, the study found.

Retailers say it's still difficult to verify a shopper's identity online, and there's a lag between the time an order is placed and when that transaction is confirmed, which opens the door for more errors or fraudulent activity.

LexisNexis has found more retailers are investing in fraud monitoring, but the technology isn't always "optimal" or effective.

For example, companies must also take into consideration their trustworthy customers, whom they don't want to turn away when being overly protective, said Lindsay Sakraida, a marketing director at DealNews.

This is especially true as retailers craft their own procedures for returns, where crime can skyrocket around the holiday shopping season.

"Return fraud is a major problem for the retail industry, but [stores] still want to encourage shoppers to shop with them, and they don't want to make it too difficult to make a return," Sakraida said. "Return policies become a way for customers to know if they want to be a loyal shopper."

A 2017 survey by the National Retail Federation has found that retail return fraud losses will cost companies as much as $15 billion this year.

This type of crime often involves an individual first stealing merchandise, and then trying to return those items for a gift card or other currency. Some criminals have even targeted major retailers, including Home Depot and Target, to take advantage of lenient return policies to fuel their drug addictions, a CNBC investigation found.

"Self checkouts are one area where thieves love to go," said Bob Moraca, NRF's vice president of loss prevention. "There is a price of doing business. ... Retailers just have to continue to work with law enforcement and prosecutors to protect their assets and brands."

Moraca's team found that retailers are expecting, on average, 11 percent of their total sales to be returned this year. And 11 percent of those returns are then expected to be fraudulent, which is slightly up from 2015, according to NRF's survey of 63 retailers.

Moraca said criminals are also getting more creative in counterfeiting store receipts, which can then be used to process a fraudulent return more seamlessly, if an employee doesn't notice the difference.

While less prevalent than other methods, Moraca said this tactic is on the rise.

In turn, retailers are looking for solutions that aren't too aggressive. This could include making a receipt necessary for a return, compressing the window of time within which a return can be made, or requiring the item's original packaging to still be intact. Demanding a form of identification is another tactic most retailers use to keep track of frequent violators.

This past year, nearly 30 percent of retailers have altered their return policies to address organized retail crime, more than in 2015 and 2016, NRF said.

Home Depot told CNBC the company has changed its policy to crack down on return fraud, only handing out "store credits," which can't be used online, in place of gift cards. Target has taken a similar approach in offering store credits, or solely an exchange, when a person doesn't present a receipt.

Other companies, including Lowe's and Walmart, have said they look closely at returns where no original receipt is presented.

On the whole, technological advancements should allow for better fraud prevention in the future, NRF's Moraca said. For example, facial recognition is already being tested to keep track of shoppers who frequent certain stores, in place of a license or other ID, he said.

"Imagine a world where you walk in [a store], and you don't even have to show ID — even if they don't know [your] name — they know this customer has been in this number of times."