Stream Routing API

Suggest Edits

"WHERE" Clauses --> Route Rules

V2 Conductor Schema does not have a direct analog for WHERE clauses in V1 schema, or LQL. In V1, for example, it was possible to describe a clause to filter data that is mapped for a stream, such as the following, which ignores events whose _url field contains the string localhost:

WHERE     not(contains(`_url`, "localhost"))

To achieve the same result in accounts with Conductor Schema turned on, we need to use the Stream Route Rules API, whose endpoints include the following:

  • GET /v2/stream/rule retrieves a list of all route rules defined in the account
  • GET /v2/stream/rule/:id retrieves a single route rule by its ID
  • POST /v2/stream/rule creates a new route rule
  • POST/PUT /v2/stream/rule/:id upserts an existing route rule by its ID
  • DELETE /v2/stream/rule/:id deletes an existing route rule by its ID

Route rules define data to route from one input stream to another output stream. On ingress, the stream on the record is overwritten from input to output if the rule is active and evaluates to true when executed against the incoming record.

The mechanics of this imply one important distinction relative to V1 WHERE clauses: since route rules describe which records to redirect away from the input stream, rather than which records to evaluate for the input stream, they are in effect the inverse of WHERE clauses. If, for example, we want to ignore urls containing the string localhost from our input stream app in the WHERE clause above, we would need to define a route rule containing the following expression (which is the precise inverse of our WHERE clause):

WHERE     contains(`_url`, "localhost")

This tells the system to redirect events matching this expression to a new output stream we define on the route rule object. By convention, if we don't care about mapping records that match our route rule, we call that output stream {input-stream}_divert, and define the following route rule object (note the need to escape double quotes inside the expression so that our object is valid JSON):

{
    "account_id": "4f7b525bdba058fc096757a6",
    "active": true,
    "aid": 12,
    "expression": "contains(`_url`, \"localhost\")",
    "input": "app",
    "name": "Ignore events from localhost",
    "output": "app_divert",
    "priority": 1
}

📘

Note: If there is a use case to take all events and route them to the output stream, the expression can hold the string value of "true" instead of a logic condition. i.e. "expression": "true"

Our object includes several properties which should be self-explanatory: account_id, aid, and name. The rest are as follows:

  • active defines whether the rule should be activated against incoming data. Marking a rule as "active": false turns it off.
  • priority indicates the order in which the rules are evaluated (higher priority first) against an event. secondary sorting according the age (newer rules first) is performed when priorities match

📘

Note that because these route rules are cached, it will take up to 10 minutes for any updates you make to take effect.

Example

To create our sample route rule, we make the following call. If successful, the response body will include the full route rule object, including a system-generated ID which can be used for subsequent calls against the object:

echo '{
    "account_id": "4f7b525bdba058fc096757a6",
    "active": true,
    "aid": 12,
    "expression": "contains(`_url`, \"localhost\")",
    "input": "app",
    "name": "Ignore events from localhost",
    "output": "app_divert",
    "priority": 1
}' | http POST $LIOAPIPROD/v2/stream/rule Authorization:$LIOTOKEN account_id==4f7b525bdba058fc096757a6

Routing Tips and Tricks

Exporting Routed Data

When data is routed from the input stream to the output stream, Lytics will associate the raw routed data to the output stream. This means when a raw activity data is exported, the data that was routed will be associated with the output stream only.

Behavior Scoring on Streams

An optional feature with all streams is to add Behavioral Scoring to selected streams. If a behavioral scores are desired for a specific stream make sure the stream that is being targeted is the routing output stream and not the input stream.

Updated 7 months ago