web
You’re offline. This is a read only version of the page.
close
Skip to main content
News and Announcements icon
Community site session details

Community site session details

Session Id :
Dynamics 365 Community / Blogs / Dynamics 365 FastTrack Blog / Leveraging CI-J interaction...

Leveraging CI-J interaction data without Fabric

Views (140)
Peter Krause Profile Picture Peter Krause 5
Based on the feedback received from our partners and customers, we have determined that a solution for data export without using Fabric requires additional investigations and refinements. We will publish an updated guidance after we fully understand the needs and expectations of our users so the proposed solution can meet their needs. Currently, we do not have a published roadmap for this solution in Customer Insights – Journeys or the underlying platform (Dataverse).

Comments

  • Scott Moore Profile Picture Scott Moore 17
    Posted at
    Leveraging CI-J interaction data without Fabric
    1. Discover the MDL container location for the specific Dataverse organization through “/api/data/v9.2/RetrieveAnalyticsStoreDetails”
    2. From the Dataverse datalakefolders table, fetch the folder named “Customer Insights Journeys” to determine the path in the MDL container through “/api/data/v9.2/datalakefolders?$filter=name%20eq%20%27Customer%20Insights%20Journeys%27&$select=path”
    3. Call the RetrieveAnalyticsStoreAccess OData function to retrieve URL and access token for direct access to MDL. The parameters of the function are “/api/data/v9.2/RetrieveAnalyticsStoreAccess(Url=@p1,ResourceType='Folder',Permissions='Read,List',SasTokenValidityInMinutes=30,UseBlobProxy=true)?@p1=%27{urlEncode(container+path)}%27”:
      • Url: MDL container URL (from step 1) + path (from step 2)
      • ResourceType: 'Folder'
      • Permissions: 'Read,List'
      • SasTokenValidityInMinutes: Validity of the access token, in minutes. Maximum of 60 minutes is allowed.
      • UseBlobProxy: true
    4. The URL returned can be targeted via the ADLS Gen2 REST APIs as follows.
    5. Download the default.manifest.cdm.json manifest file first by calling “https://{endpointID}.environment.api.powerplatformusercontent.com/storage/aeth-{accountID}/{folderName}/default.manifest.cdm.json?{SAStoken}” (endpointID, accountID, folderName and SAStoken need to be preserved from the URL obtained in step 3).
    6. The default.manifest.cdm.json contains the list of interaction (entity) types in the entities array. Interaction types are identified by the entityName property, while the location property (within the dataPartitions array) denotes their relative folder path in the file system:
    1. List files for the chosen interaction type using “https://{endpointID}.environment.api.powerplatformusercontent.com/storage/aeth-{accountID}?resource=filesystem&recursive=true&directory={folderName}/{interactionLocation}&(remainingSASparameters)”
      • Please note: folderName found in the URL from step 3 and location found from step 6 need to be concatenated to determine the relative path required for the directory parameter.
      • Please note: The path listing API uses continuation tokens for paging results. Observe the x-ms-continuation response header and if present, call the list API repeatedly by passing the continuation token from the x-ms-continuation response header to the continuation query string parameter of the subsequent request. Further details on this API can be found in Path - List - REST API (Azure Storage Services) | Microsoft Learn
    2. Use the response from path listing API to determine where individual files are located and whether they were modified since the last data download:
    1. Download individual files using “https://{endpointID}.environment.api.powerplatformusercontent.com/storage/aeth-{accountID}/{fileName}?{SAStoken}”
     
    The result of this process is that a bunch of files is provided as a download. Implementing such a program and running it on a local computer would result in a file structure like this:

     
    Please note that the files received by this procedure are not human-readable. In the next section, we look at ways on how to process this data. A sample .NET console application is provided here on GitHub.
    Process data
    Since the interaction data is downloaded in the Delta lake format, you’ll need to use a tool which can accept Delta lake format as input for further processing. One of those technologies is Apache Spark which can be utilized from within Python and / or Jupyter notebooks.
    Important note: The data needs to be consumed as Delta Lake format. Consuming just Parquet files while disregarding Delta log files will result in consuming duplicated and deleted records.
    During the processing stage, the data is read, eventually transformed and stored so that the visualization technology can make use of it, for example a CSV file, database tables or any other format.
    Processing steps could look like:
    1. Read the Delta files using PySpark:
    from pyspark.sql import SparkSession
    spark = SparkSession.builder.appName("APP_Name").getOrCreate()
    df = spark.read.format("delta").load("path_to_downloaded_data")
    df.show()
    1. Transform the data (filter, aggregate, or join with other datasets from Dataverse).
    2. Store the processed data in a more accessible format, such as:
      • CSV (for simple reporting)
      • Database tables (SQL, Azure Synapse)
      • Parquet files (for efficient storage and querying)
     
    By following these steps, you can efficiently process Customer Insights - Journeys interaction data for downstream analysis. Using Apache Spark and Delta Lake enables scalable data transformations, ensuring the data is structured in a meaningful way for reporting. Once processed, the refined dataset can be stored in a suitable format (CSV, SQL database, or Parquet) for easy access and visualization.
     
    Next, we will explore how to take this processed data and build insightful Power BI dashboards to drive customer engagement analysis and business decision-making.
     
    Visualize data
    Once the data has been extracted and processed, the next step is to create visualizations that provide insights into interactions with customers. Power BI is the recommended tool for this, as it can natively connect to Delta Lake files via Azure Synapse Analytics or Azure Data Lake Storage Gen2.
     
    Power BI Integration Steps:
    1. Connect to Processed Data
      1. If using Azure Data Lake, connect via Azure Data Lake Storage Gen2.
      2. If stored in Azure Synapse Analytics, use the Azure Synapse connector.
      3. If in CSV/Parquet format, import directly into Power BI.
    2. Build Visualizations by creating custom reports
      1. Display engagement rates, email opens, click-throughs.
      2. Blend CI-J interaction data with Dataverse data (e.g., customer profiles, sales data) to provide deeper insights.
    3. Automate & Refresh: Schedule automatic refresh in Power BI Service to keep dashboards updated.

     
    By leveraging Power BI, you can transform the processed Customer Insights – Journeys interaction data into actionable insights. Using pre-built Power BI connectors or custom queries, you can visualize key engagement metrics, customer behavior trends, and campaign performance. With interactive dashboards, teams can monitor real-time data, drill down into specific segments, and make data-driven decisions to optimize marketing efforts.
     
    Retrieving only data updates
    The Customer Insights – Journeys interaction data is, in general, slow moving and vast majority of the historical data remains unchanged day to day. To reduce the download time and the data volume required for daily updates, we recommend that customers observe the following best practices:
    • After the initial data download, persist the data in durable storage (i.e., not deleted after each download run).
    • For subsequent downloads, retrieve only files that have changed since the previous iteration. Use the modified time properties on individual files (both parquet and delta log) to determine whether the file needs to be downloaded.
    • Download only the interaction (entity) types required for reporting. Each interaction type is stored in a separate folder, so we recommend selecting only those relevant to your reporting needs. You can find descriptions of individual interaction types in the Overview of CustomerInsightsJourneys - Common Data Model - Common Data Model | Microsoft Learn.
     
    Please note that data downloads should be run no more than once per day to ensure optimal performance and efficient use of the system.
    To ensure service availability and maintain fair usage for all customers, Microsoft reserves the right to limit or suspend access in cases of sustained excessive usage or patterns inconsistent with the intended use of the feature.
     
    Summary
    While it's possible to process Customer Insights – Journeys interaction data without using Microsoft Fabric, doing so requires additional development effort to access and incorporate the data into your own reporting strategy, as outlined in this article.
    For customers seeking a streamlined and scalable solution, integration with Microsoft Fabric remains the recommended and most future-proof approach. Once set up, using Fabric provides built-in capabilities for managing, transforming, and visualizing data with minimal overhead.
  • MS-10071903-0 Profile Picture MS-10071903-0
    Posted at
    Leveraging CI-J interaction data without Fabric
    Hi,
    I was quite surprised when I learned that the only supported way to access interaction data in CI-J is through fabric.
    Was the previously described way to extract the data without Fabric an officially supported one? Or was it using some Feature which can be turned off any time?
    What I am asking, can a reliable solution be built using this now removed approach.
     
    Best Regards
     
  • okohler Profile Picture okohler
    Posted at
    Leveraging CI-J interaction data without Fabric
    Hi Peter. Until very recently this page contained a step by step description on how to retrieve the marketing interaction data in json and parquest files. Is that description no longer viable, or could I still use that route to retrieve the interaction data?
     
    Thanks, and with kind regards,
    Oskar
  • CU10071306-0 Profile Picture CU10071306-0
    Posted at
    Leveraging CI-J interaction data without Fabric
    Are there any updates on this topic? Are there alternatives to using Fabric to access the Customer Insights – Journeys?
     
    Thank you in advance!