Aptitive has designed many solutions that automate processes to collect, transform, and combine data for analysis. There are a variety of technologies and approaches, but today I thought I would explore the Azure App Service. For this example, I decided to collect Twitter data into a cloud-based SQL database (I’m sure you’ll notice there are lots of source and target options for collecting your data).

1. Create Landing Area for the Data

To start, I created a SQL Server and a blank SQL database.

Next, using my local SSMS, I connected to the SQL Server by selecting “Connect > Database Engine…” and then connecting to “<servername>.database.windows.net”. Finally, I used SQL to create a basic table for collecting my twitter data (I used the “string all the things” approach for this lab and created only a few of the available attributes):

2. Create a Logic App

After creating the landing, I added a new Logic App (in the Web + Mobile section). Then, clicked on the “Logic App Designer” under Development Tools and selected the “When a new tweet is posted” starter trigger.

I then added a condition to filter out tweets from accounts with less than 50 users:

Finally, I mapped the data to the corresponding columns:

The resulting trigger will extract the desired attributes from the tweet containing the word “aptitive” and, if the user has > 50 followers, insert the row into my Azure SQL db:

3. Combine with other data and analyze

The advantage of pulling the twitter data into a structured staging table is that I can now combine and manipulate the data for analysis. From here, we could connect to a visualization tool, use the data in a predictive model, apply data cleansing, and automate data-driven business processes.

There are a variety of approaches to integrating your data. Aptitive is experienced at using everything from traditional tools like SSIS to newer app services in the cloud (Azure, GCP, AWS). If you’re looking to discuss a solution for improving your data flows, please reach out!