Cosmos Db Data Explorer Download |WORK|
Demarcus Smith
Azure Data Explorer supports data ingestion from Azure Cosmos DB for NoSql using a change feed. The Cosmos DB change feed data connection is an ingestion pipeline that listens to your Cosmos DB change feed and ingests the data into your Data Explorer table. The change feed listens for new and updated documents but doesn't log deletes. For general information about data ingestion in Azure Data Explorer, see Azure Data Explorer data ingestion overview.
Each data connection listens to a specific Cosmos DB container and ingests data into a specified table (more than one connection can ingest in a single table). The ingestion method supports streaming ingestion (when enabled) and queued ingestion.
Before you create a data connection, create a table where you'll store the ingested data and apply a mapping that matches schema in the source Cosmos DB container. If your scenario requires more than a simple mapping of fields, you can use update policies to transform and map data ingested from your change feed.
Update policies are a way to transform data as it's ingested into your table. They're written in Kusto Query Language and are run on the ingestion pipeline. They can be used to transform data from a Cosmos DB change feed ingestion, such as in the following scenarios:
Specify the Event retrieval start date. This is the time from which the connector will start ingesting data. If you don't specify a time, the connector will start ingesting data from the time you create the data connection. The recommended date format is the ISO 8601 UTC standard, specified as follows: yyyy-MM-ddTHH:mm:ss.fffffffZ.
Grant the data connection permission to access your Cosmos DB account. Providing the data connection access to your Cosmos DB allows it to access and retrieve data from your database. You'll need your cluster's principal ID, which you can find in the Azure portal. For more information, see Configure managed identities for your cluster.
Azure Data Explorer has an aggregation (batching) policy for queued data ingestion designed to optimize the ingestion process. The default batching policy is configured to seal a batch once one of the following conditions is true for the batch: a maximum delay time of 5 minutes, total size of one GB, or 1000 blobs. Therefore, you may experience a latency. For more information, see batching policy. To reduce latency, configure your table to support streaming. See streaming policy.
The change feed API is polled by the data connector at regular intervals, typically every few seconds. Each poll contains changes that occurred in the container between calls, but only the latest version of change per document.
Because of this scenario, the data connector may miss some intermediate document changes. For example, some events may be missed if the data connection service is down for a few minutes, or if the frequency of document changes is higher than the API polling frequency. However, the latest state of each document is captured.
Azure Data Explorer keeps track of the change feed by checkpointing the "position" it is at in the feed. This is done using continuation token on each physical partitions of the container. When a container is deleted/recreated, those continuation token are invalid and aren't reset: you must delete and recreate the data connection.
Azure Cosmos DB Explorer is a standalone web-based interface that allows you to view and manage the data stored in Azure Cosmos DB. Azure Cosmos DB Explorer is equivalent to the existing Data Explorer tab that is available in Azure portal when you create an Azure Cosmos DB account. The key advantages of Azure Cosmos DB Explorer over the existing Data explorer are:
Customers receiving HTTP-401 errors may be due to insufficient Azure RBAC permissions for your Azure account, particularly if the account has a custom role. Any custom roles must have Microsoft.DocumentDB/databaseAccounts/listKeys/* action to use Data Explorer if signing in using their Microsoft Entra credentials.
The Data explorer given in the Azure Cosmos DB account is a web-based interface that allows us to view, access, and manage cosmos DB elements like databases, containers, and data stored in Cosmos DB items. We can access this option only with Cosmos DB account creation. We can perform all activities using data explorer. I will show you step by step process about performing various cosmos DB activities using data explorer in this article. Do not confuse Data Explorer given in Cosmos DB account with Azure Data Explorer Clusters. Both are different and designed for different purposes.
Creating databases and collections will require an Azure cosmos DB account first, so first, we will create a cosmos DB account and then will create its database followed by the collections using data explorer.
Once you have entered all details on all tabs then hit the Review + create button to start the validation process. Finally, click on Create button to proceed with creation. Deployment of cosmos DB account creation will start processing after that. You will be notified once the deployment will be completed. We have successfully created a cosmos DB account named cosmosdb19oct20211 in this section. Next, click on Go to resource tab to jump to the cosmos DB account overview page.
Now, we have a Cosmos DB account named cosmosdb19oct20211. Click on the Overview page of this account. You can see all details of this account from the right-side pane. You can also access data explorer from the left side pane as highlighted in the below image. Click on this option to open the data explorer page.
The below screen will appear after accessing the Data Explorer tab from the left side pane. You can focus on dark red arrows to see important links. I have created this Azure Cosmos DB account using Core SQL API so SQL API is showing in the right side pane. You can click on this option to expand it; all databases and containers can be seen as organized under this API. Have a look at the other dark red arrows which are showing links and options using which we can create a new database or containers in this cosmos DB account.
Once you will click on the Start with Sample rectangle from the above image, sample database creation will start processing and can be tracked its progress in the bottom side of the above image which is highlighted in yellow color. I have taken a screenshot of that notification bar which is shown in the below image.
Now, we will create a new database by accessing the New Database link given in the right-side pane of the Data Explorer. You can also click on the drop-down given near New Container to choose a new database creation window. A database in cosmos DB is a logical container for one or more containers.
Below New Database screen will appear to enter details for the new database. Enter database if as I have entered as cosmosdb1. Provision throughput checkbox will be enabled as it was selected during creating the cosmos DB account. If you want this database to Autoscale in case of increased workload or throughput then you can choose this option given under Database throughput (autoscale). The provisioned throughput at the database level will be shared among all collections of that database. Next, configure the database max request units per second and then finally click on the OK button given bottom side of the below image.
If you want to choose database throughput as Manual, then you need to select this option and you need to carefully enter the required request units per second as capacity throughput for this database. Note, the database will not be auto-scaled in case of increased work as you have chosen Manual mode for database throughput.
Your database will be created after a few minutes of executions post clicking the OK button of the New Database window. You can see and validate your newly created database by accessing the SQL API tree in data explorer as shown in the below image near the dark red arrow.
Once you choose the existing database, other options will be disappeared from the database Id section. Next, you need to fill in details for Container Id. Enter the id for the container, enter the partition key based on which data partitioning will work. You can also add unique keys by accessing this option. Click the OK button once you filled in all details. You can leave options as it is to create this container with default configurations.
Once you will click on the OK button of the above screen, a new container will be created under your selected database. Have a look at the dark red arrows to verify this newly created container cosmoscontainer1.
You can click on Items under the container to see its data or add new items to this container. Click on the New Item to add a new item or you can also upload the saved JSON file to upload items to this container by accessing their respective tabs from the below image.
We have created new containers for the new sample database and a new user database in the above steps. We can access these containers using the Azure portal as well. You can click on the Browse tab given under Containers from the left side pane and all containers and their details like associated database, their provisioned throughput, and other details will appear in the right-side pane.
Azure Cosmos DB is a fully managed, highly secure, and globally distributed NoSQL database. We can manage its databases and elements using the data explorer utility given inside the cosmos DB account. I have explored data explorer and steps to create a sample database, new container, and database using data explorer in the Azure Cosmos DB account. You can manage other configuration details of cosmos databases and containers using this tool.
df19127ead