Summary

In the first Exploring DAP blog of 2024, our focus shifts towards a more technical aspect of the DNSRF’s DAP.LIVE platform—specifically, the utilization of DAP for defining Application Programming Interfaces (APIs) intended for non-DAP applications. This involves configuring DAP APIs to seamlessly integrate the robust data feeds and analytical functionalities of the DAP into existing applications, services, and systems, thereby enhancing the local application data. The article defines the DAP data enrichment lifecycle, a framework where external data is transmitted to the DAP, undergoes enrichment using its data and analytics capabilities, and is subsequently made accessible to the application through one or more customized APIs.

This strategic use of the DAP empowers existing applications to swiftly and effortlessly harness the full spectrum of DAP capabilities, encompassing a diverse array of data feeds and extensive analytical tools. By applying this enriched data, local applications can ensure that business processes and decisions are founded on the best information available, ultimately leading to the generation of new insights and an overall enhancement of decision-making processes not previously available.

The Power of DAP APIs

Over the past year, the Exploring DAP series has introduced the many analytical capabilities, and diverse data feeds available on the DNSRF’s DAP.LIVE platform. As the New Year begins, I wanted to focus on a more technical aspect of the DNSRF’s DAP.LIVE platform: Using the DAP to define Application Programming Interfaces (APIs) for use by non-DAP platforms, systems, and applications. This powerful feature allows local systems to use DAP capabilities and data feeds by exposing simple request and response interfaces and allowing them to “enrich” internal data with data hosted on the DAP.

In a previous article (Blog: Protecting Your Brand: A Comprehensive Guide to DAP for Brand Monitoring), I described how users can use the DAP for simple brand/trademark monitoring purposes. In this article, I’ll riff on that idea and explain how an existing local application can leverage the data available on the DAP. 

This article will define a simple use case with data we want the DAP to enhance and then explain how to enrich that data using the DAP.LIVE analytics capabilities This data enrichment cycle includes 1) making the data available to the DAP within a user’s private account,  2) describing the queries and analytics used to enhance the data, and 3) showing how the DAP can make those queries available to local applications via APIs

Defining the Use Case Requirements

Let's assume that a given application manages a collection of brands and trademark strings and would like to enrich this collection with data from the DAP. First, we must define an API enabling the app to upload a list of strings to the DAP.

Manage a List of Strings

• Add string(s), Update string(s), Delete String(s), List Strings

Next, we need to define an API that returns the following data for every string: 

 Zone File Registrations Hits

• Return details of all new zone file registrations that contain and are similar to the string  
• Return a summary of new zone file registrations 

Blockchain Registration Hits 

• Return details of all new blockchain registrations that contain and are similar to the string
• Return a summary of new blockchain registrations

Malware, Phishing, and Scam Security List Hits

• Return recent security list entries that contain the string
• Indicate if the string is in the URL, hostname, or domain name
• Return a summary of hits

Obtaining this enriched data from the DAP enables the user’s own local application to understand how and where their brands are being used. The data returned to the local application from DAP allows the user to filter and prioritize the potential risk of each “hit” based on their own evolving business rules. Once prioritized, the DAP-enriched data allows the local application to build custom workflows to investigate potential issues and determine the best mitigation method.

Importing Data into DAP

To enrich data for our example application, we first need to configure a DAP API so the local application can deliver the data to the DAP.  Note that data made available to the DAP is secured, so it is only available to users of their private DAP account. For our example, this API will allow the application to manage a list of strings (brands and trademarks). Standard REST-style operations are supported by the DAP API, allowing the management of this list, including 

• Create/Add Entries
• Read Entries
• Update Entries
• Delete Entries

The strings we used as examples are

amazon, apple, facebook, google, instagram, microsoft

Configuring the DAP Queries

To configure DAP APIs that meet the requirements outlined above, we need to build a series of DAP stored queries using the following DAP data feeds

• Zone Files: Domains - List of all available domains obtained by processing zone files from the CZDS and other ccTLD providers
• DAP: Blockchain Domains - A summary of all blockchain domains obtained from all providers available on DAP.LIVE
• DAP: Malware Combined - Combined Malware data obtained from all current providers on DAP.LIVE
• DAP: Phishing Combined - Combined Phishing data obtained from all current providers on DAP.LIVE
• ScamAdvisor: Complete Dataset - The complete dataset from ScamAdvisor containing data on all domain names detected

Step 1 - Optimize Queries

DAP data feeds contain hundreds of thousands, and in many cases millions, of rows of data. As we are only interested in the most recent data for our use case, we start by creating project-based snapshots of only the data we need. To give the API some flexibility, we take a snapshot of the last 7-days of data for each data feed. Next, we configure the DAP to take a snapshot of each of our five data feeds daily at 08:00 UTC (00:00 Pacific time).

Step 2 - Configuring Queries

Next, we configure the queries to return the desired results. 

Starting with the three abuse feeds, we use the imported list of brand strings to find exact matches for each in the URL, hostname, or domain name. Then, we use the domain name to query for the relevant WHOIS data from the WhoisXML data feed, which allows the API to return additional information associated with the domain name, including the registration date and registrar name, to name a few. Finally, we devise a formula to create a new column that indicates where the string exists.  i.e., in the domain name, hostname, or URL.

The query for the zone file and blockchain registration data is different. In this query, we use the imported list of brand strings to find exact matches for the string and leverage the Levenshtein distance algorithm to look for matches that are “similar” to the string, which gives the user an indication of the existence of possible typo-squatting attacks. Finally, we devise a formula to create a new column that indicates if the match found was exact, similar, or contained somewhere in the domain name.

Step 3 - Adding Parameters

By default, each query defined in Step 2 will return results (rows) for all brands seen in the last seven days. To give users of the API flexibility and control over which data they might desire, we add two parameters to these queries 

• Search String: Filter responses to a single brand MUST be one of the strings known to the DAP. 
• Time Period: Indicate how many days in the past to query. MUST be between 1 and 7 days. 

Creating APIs

Creating APIs for the queries configured earlier is simply accomplished using the DAPs API Connections feature. See Figure 1 for an example configuration screen. 

^{Figure 1 - DAP API Configuration Dialog}

A list of all APIs configured for the project is available in the API Connections area of the DAP(Figure 2)

^{Figure 2 - DAP API Connections List}

Project Diagram

Figure 3 below depicts how we used the DAP in our example. At the top, it shows the DAP Feeds we leveraged. Below that, we show the locally stored queries described above. Finally, the project APIs that allow the application to interface with the DAP are depicted at the bottom.

^{Figure 3 - Project Diagram for our Example}

API Definitions

Once defined, the DAP creates documentation for each API. Each API requires an API-KEY and API-SECRET. We have not included those values in the examples below for security purposes.

Zone File Summary

A summary of domain name registrations for the brand(s) and the time period specified

Request

Method:

• GET

Endpoint:

• /feed/zone-file-summary

Parameters

• searchString: One of the known brands or %*% for all
• timePeriod: X_DAYS_AGO (X is an integer between 1 and 7)

Example Request

• https://webservices.dap.live/feed/zone-file-summary?searchString=google&timePeriod=1_DAY_AGO

Response Description

• May be NULL
• Sorted by Brand, Match Type, and count

Match Type is 

• Exact: The exact string (and only the string) was found in the domain name. (example.com)
• Contains: The domain name contains the string (someexample.com)
• Similar: The domain name contains a similar string (exomple.com)

[

  {

    "brand": "amazon",

    "matchType": "Contains",

    "count": 29

  },

  {

    "brand": "apple",

    "matchType": "Contains",

    "count": 27

  }

]

Zone File Details

List of domain name registrations for the brand(s) and time period specified

Request

Method

• GET

Endpoint

• /feed/zone-file-details

Parameters

• searchString: One of the known brands or %*% for all
• timePeriod: X_DAYS_AGO (X is an integer between 1 and 7)

Example Request

• https://webservices.dap.live/feed/zone-file-details?searchString=*%*&timePeriod=1_DAY_AGO

Response Description

• May be NULL
• Each entry contains the following information

Match Type is

• Exact: The exact string (and only the string) was found in the domain name. (example.com)
• Contains: The domain name contains the string (someexample.com)
• Similar: The domain name contains a similar string (exomple.com)

{

    "event_date": "2023-12-07 01:01:56",

    "tld": "email",

    "tld_type": "gTLD",

    "prefix": "microsofts",

    "domain_name": "microsofts.email",

    "brand": "microsoft",

    "matchType": "Contains",

    "matchPosition": "In 2nd Level",

    "createdDateNormalized": "2023-12-05 01:27:46",

    "updatedDateNormalized": "2023-12-05 01:27:46",

    "expiresDateNormalized": "2024-12-05 01:27:46",

    "registrarName": "GMO Internet Group, Inc. d/b/a Onamae.com",

    "registrarIANAID": "49",

    "estimatedDomainAge": 3,

    "registrantName": "REDACTED FOR PRIVACY",

    "registrantOrganisation": "GMO-Z.com RUNSYSTEM JSC",

    "status": "addPeriod"

  }

Blockchain Summary

A summary of blockchain registrations for the brand(s) and time period specified

Request

Method

• GET

Endpoint

• /feed/blockchain-summary

Parameters

• searchString: One of the known brands or %*% for all
• timePeriod: X_DAYS_AGO (X is an integer between 1 and 7)

Example Request

• https://webservices.dap.live/feed/blockchain-summary?searchString=%*%&timePeriod=7_DAY_AGO

Response Description

• May be NULL
• Sorted by Brand, Match Type and count

Match Type is

• Exact: The exact string (and only the string) was found in the domain name. (example.com)
• Contains: The domain name contains the string (someexample.com)
• Similar: The domain name contains a similar string (exomple.com)

[

  {

    "brand": "amazon",

    "matchType": "Similar",

    "count": 4

  },

  {

    "brand": "apple",

    "matchType": "Similar",

    "count": 6

  },

  {

    "brand": "facebook",

    "matchType": "Contains",

    "count": 1

  },

  {

    "brand": "google",

    "matchType": "Similar",

    "count": 3

  },

  {

    "brand": "instagram",

    "matchType": "Similar",

    "count": 1

  },

  {

    "brand": "microsoft",

    "matchType": "Contains",

    "count": 1

  }

]

Blockchain Details

List of blockchain name registrations for the brand(s) and time period specified.

Request

Method

• GET

Endpoint

• /feed/blockchain-details

Parameteres

• searchString: One of the known brands or %*% for all
• timePeriod: X_DAYS_AGO (X is an integer between 1 and 7)

Example Request

• https://webservices.dap.live/feed/blockchain-details?searchString=instagram&timePeriod=3_DAY_AGO

Response Description

• May be NULL
• Each entry contains the following information

Match Type is

• Exact: The exact string (and only the string) was found in the domain name. (example.com)
• Contains: The domain name contains the string (someexample.com)
• Similar: The domain name contains a similar string (exomple.com)

{

    "domain_name": "instagrok.eth",

    "prefix": "instagrok",

    "prefix_unicode": "instagrok",

    "tld": "eth",

    "created_date": "2023-12-07 17:55:35",

    "expiry_date": "2026-12-06 17:55:35",

    "blockchain": "ETH",

    "name_service": "ENS",

    "registrant_id": "0xd4416b13d2b3a9abae7acd5d6c2bbdbe25686401",

    "domainNameRaw": "<pre class=\"py-2 bg-gray-100 inline\">instagrok.eth</pre>",

    "brand": "instagram",

    "matchType": "Similar"

  }

Malware Summary

A summary of reported malware URLs for the brand(s) and time period specified

Request

Method

• GET

Endpoint

• /feed/malware-summary

Parameters

• searchString: One of the known brands
• timePeriod: X_DAYS_AGO (X is an integer between 1 and 7)

Example Request

• https://webservices.dap.live/feed/malware-summary?searchString=facebook&timePeriod=1_DAY_AGO

Response Description

• May be NULL
• Sorted by Brand, Match position, and count

Match Position is

• In 2nd Level: the string was found in the domain name
• In 3rd Level: the string was found in the hostname
• In URL: The string was found in the URL

[

  {

    "brand": "microsoft",

    "matchPosition": "In URL",

    "count": 4

  }

]

Malware Details

List of reported Malware URLs for the brand(s) and time period specified

Request

Method

• GET

Endpoint

• /feed/malware-details

Parameters

• searchString: One of the known brands
• timePeriod: X_DAYS_AGO (X is an integer between 1 and 7)

Example Request

• https://webservices.dap.live/feed/malware-details?searchString=facebook&timePeriod=1_DAY_AGO

Response Description

• May be NULL

Match Position is

• In 2nd Level: the string was found in the domain name
• In 3rd Level: the string was found in the hostname
• In URL: The string was found in the URL
• Each entry contains the following information

  {

    "url": "http://66.228.43.8/wlc/microsoftdecidedtodeleteentirethingsfromthepccookiecachehistoryeverything.doc",

    "hostname": "66.228.43.8",

    "domain_name": null,

    "tld": null,

    "urlhaus_discover_time": "2023-12-07 17:17:07",

    "urlhaus": "1",

    "window_time": "2023-12-07 00:00:00",

    "brand": "microsoft",

    "matchPosition": "In URL",

    "createdDateNormalized": null,

    "updatedDateNormalized": null,

    "expiresDateNormalized": null,

    "registrarName": null,

    "registrarIANAID": null,

    "estimatedDomainAge": 0,

    "registrantName": null,

    "registrantOrganisation": null,

    "status": null

  }

Phishing Summary

A summary of reported phishing URLs for the brand(s) and time period specified

Request

Method

• GET

Endpoint

• /feed/phishing-summary

Parameters

• searchString: One of the known brands
• timePeriod: X_DAYS_AGO (X is an integer between 1 and 7)

Example Request

• https://webservices.dap.live/feed/phishing-summary?searchString=%*%&timePeriod=1_DAY_AGO

Response Description

• Sorted by Brand, Match position and count
• May be NULL

Match Position is

• In 2nd Level: the string was found in the domain name
• In 3rd Level: the string was found in the hostname
• In URL: The string was found in the URL

[

  {

    "brand_2": "amazon",

    "matchPosition": "In 3rd level",

    "count": 1

  },

  {

    "brand_2": "apple",

    "matchPosition": "In URL",

    "count": 7

  }

]

Phishing Details

List of reported Phishing URLs for the brand(s) and time period specified

Request

Method

• GET

Endpoint

• /feed/phishing-details

Parameters

• searchString: One of the known brands
• timePeriod: X_DAYS_AGO (X is an integer between 1 and 7)

Example Request

• https://webservices.dap.live/feed/phishing-details?searchString=%*%&timePeriod=1_DAY_AGO

Response Description

• May be NULL

Match Position is

• In 2nd Level: the string was found in the domain name
• In 3rd Level: the string was found in the hostname
• In URL: The string was found in the URL
• Each entry cointans the following information

 {

    "url": "https://facebookbarnacle.com/A8324K213/7193175735.html",

    "hostname": "facebookbarnacle.com",

    "domain": "facebookbarnacle.com",

    "tld": "com",

    "brand": "Facebook",

    "sector": null,

    "country_code": null,

    "country_name": null,

    "ip": "172.67.223.26",

    "asn": null,

    "asn_name": null,

    "apwg_discover_time": "2023-12-06 09:08:36",

    "openphish_discover_time": null,

    "apwg": "1",

    "openphish": null,

    "window_time": "2023-12-06 00:00:00",

    "brand_2": "facebook",

    "matchPosition": "In 2nd Level",

    "createdDateNormalized": "2023-12-06 10:39:50",

    "updatedDateNormalized": "2023-12-06 10:45:02",

    "expiresDateNormalized": "2024-12-06 10:39:50",

    "registrarName": "Network Solutions, LLC",

    "registrarIANAID": "2",

    "estimatedDomainAge": 1,

    "registrantName": "Legaspi, Jet",

    "registrantOrganisation": "JLCG",

    "status": "clientTransferProhibited"

  }

Scam Summary

A summary of reported scam URLs for the brand(s) and time period specified

Request

Method

• GET

Endpoint

• /feed/scam-summary

Parameters

• searchString: One of the known brands
• timePeriod: X_DAYS_AGO (X is an integer between 1 and 7)

Example Request

• https://webservices.dap.live/feed/scam-summary?searchString=amazon&timePeriod=6_DAY_AGO

Response Description

• May be NULL
• Sorted by Brand, Match position and count

Match Position is

• In 2nd Level: the string was found in the domain name
• In 3rd Level: the string was found in the hostname
• In URL: The string was found in the URL

[

  {

    "brand": "amazon",

    "matchPosition": "In 2nd Level",

    "count": 934

  },

  {

    "brand": "amazon",

    "matchPosition": "In 3rd Level",

    "count": 10

  },

  {

    "brand": "apple",

    "matchPosition": "In 2nd Level",

    "count": 78

  },

  {

    "brand": "apple",

    "matchPosition": "In 3rd Level",

    "count": 17

  },

  {

    "brand": "facebook",

    "matchPosition": "In 2nd Level",

    "count": 7

  },

  {

    "brand": "facebook",

    "matchPosition": "In 3rd Level",

    "count": 7

  }

]

Scam Details

List of reported scam hostnames for the brand(s) and time period specified

Request

Method

• GET

Endpoint

• /feed/scam-details

Parameters

• searchString: One of the known brands
• timePeriod: X_DAYS_AGO (X is an integer between 1 and 7)

Example Request

• https://webservices.dap.live/feed/scam-details?searchString=amazon&timePeriod=6_DAY_AGO

Response Description

• May be NULL

Match Position is

• In 2nd Level: the string was found in the domain name
• In 3rd Level: the string was found in the hostname
• In URL: The string was found in the URL
• Each entry contains the following information

  {

    "hostname": "instagram-com-marianamoreira.blogspot.com",

    "domain_name": "blogspot.com",

    "tld": "com",

    "last_updated": "2023-12-06 16:40:58",

    "first_checked": "2023-12-06 16:40:58",

    "score": 50,

    "blacklisted": "No",

    "brand": "instagram",

    "matchPosition": "In 3rd Level",

    "createdDateNormalized": "2000-07-31 21:38:58",

    "updatedDateNormalized": "2023-06-29 10:00:58",

    "expiresDateNormalized": "2024-07-31 21:38:58",

    "registrarName": "MarkMonitor, Inc.",

    "registrarIANAID": "292",

    "estimatedDomainAge": 8529,

    "registrantName": "name",

    "registrantOrganisation": "Google LLC",

    "status": "clientDeleteProhibited clientTransferProhibited clientUpdateProhibited serverDeleteProhibited serverTransferProhibited serverUpdateProhibited"

  }

Brand List Management API

Add/Create

Insert a set of rows into the data source

Method

• POST

Endpoint

• https://webservices.dap.live/api/tabularData/brands

Example of payload and curl command

• [{"brand": "string_one"},{"brand": "string_two"}]

% curl -H "API-KEY:" -H "API-SECRET:" -X POST https://webservices.dap.live/api/tabularData/brands -d '[{"brand": "string_one"},{"brand": "string_two"}]

Update

Update a set of rows to the data source

Method

• PUT

Endpoint

• https://webservices.dap.live/api/tabularData/brands

Example payload and curl command

• [{"brand": "string"}]

% curl -H "API-KEY:" -H "API-SECRET:" -X PUT https://webservices.dap.live/api/tabularData/brands -d "[{"brand": "updated_string"}]"

Delete

Delete a set of rows to the data source

Method

• DELETE

Endpoint

• https://webservices.dap.live/api/tabularData/brands

Example payload and curl command

• ["primary_key_1", "primary_key_2", "primary_key_3"]

% curl -H "API-KEY:" -H "API-SECRET:" -X DELETE https://webservices.dap.live/api/tabularData/brands -d '[{"string_one"}]'