Category Archives: Database

Create Cosmos DB , failover options,data replication options from azure subscription

This article outlines the steps to create Cosmos DB from the azure subscription.

  1. login to azure portal – Click on Azure Cosmos DB – Create Cosmos DB
  2. Type the document ID – keep this in mind this document ID is the URL we will be using  as connection string in the application.
  3. Select the preferred API according to your requirement.
  4. Choose the azure subscription, select the resource group
  5. Choose the primary location where the data needs to be replicated.There is an option to enable geo redundancy which can be done later as well.

Picture9

To Enable Geo-Redundancy-

Click on – Enable Geo Redundancy – and choose the  preferred region.

Picture10

Replicate data globally in few clicks –

Picture11

Failover options –

There are 2 failover options Manual and automatic.

Picture12
Manual can be triggered any time – we just need to select the disclaimer and initiate failover.

Picture13

Add new regions any time and replicate your data in few minutes-

Picture14

Failover options – Automatic

we need to go and enable the automatic failover as below

Picture15

Also there is an option to change the priorities of the failover in few clicks.Good part is can be done any time and we do not need to change them on the code.

Picture16

Consistency levels:

Can be modified and altered any time. The default consistency type is session as below.

Picture17

Network Security for the Database:

We have an option to access the database only from few subnets. This gives a complete security to the document. A VAPT can be initiated after setting up this security which eases the database admin job on considerations on data security.

Picture18

Endpoint and Keys to integrate with your code:

We need to use the URI and the primary key to integrate with the code. This can be seen by clicking on the keys section on the left side.

Picture19.png

Summary:

Now a Cosmos Database is created  –  now create new Collection-  create documents –  Then they are stored in JSON rows. Try to have most of the  documents under one collection, because the pay model is per collection.

Create collection:

Click on Add collection

Picture21.png

Create new Database ID – then Collection ID

Remember the collections in Cosmos DB are created in the below order.

Picture23

Now we need to choose the throughput and the storage capacity. They will be charged according to the selection. Also there is an option to choose unique keys which adds more data integrity.

Picture22

Example of a new document

Picture25

Better to define a document ID and collection ID.

Picture24

Once the above is done, we can connect via preferred available API  to your document and the developer do not need to worry about data schema , indexing, security.

More sample codes in GitHub:

https://github.com/Azure-Samples/azure-cosmos-db-documentdb-nodejs-getting-started

Example below:

Before you can run this sample, you must have the following prerequisites:

◦An active Azure Cosmos DB account.
◦Node.js version v0.10.29 or higher.
◦Git.

1) Clone the repository.

Picture26
2) Change Directories.
3) Substitute Endpoint with your primary key and endpoint.

Picture27
4) Run npm install in a terminal to install required npm modules
5) Run node app.js in a terminal to start your start your node application

Thanks & Regards
Sathish Veerapandian
MVP – Office servers & Services.

Microsoft Cosmos DB features,options and summary

This article gives an introduction on Microsoft Cosmos DB, features available in them and options to integrate with application.

Introduction:

CosmosDB is the next Generation of Azure DB,its a enhanced version of document db.
Document DB customers, with their data, are automatically Azure Cosmos DB customers.
The transition is seamless and you now have access to all capabilities offered by Azure Cosmos DB.

Cosmos DB is a planet scale database.It is a good choice for any server less application that needs low order-of-millisecond response times, and needs to scale rapidly and globally. They are more transparent to your application and the config does not need to change.

How It was derived:

Microsoft Cosmos DB isn’t entirely new: It grew out of a Microsoft development initiative called Project Florence that began in 2010. Project Florence is a speculative glimpse into our Future where both our Natural and Digital worlds could co-exist in harmony through enhanced communication.

Picture1

  • It was first commercialized in 2015 with the release of a NoSQL database called Azure DocumentDB
  • Cosmos DB was introduced in 2017.
  • Cosmos DB expands on it by adding multi-model support, global distribution capabilities and relational-like guarantees for latency, throughput, consistency and availability.

Why Cosmos DB?

  • It has no Data Scheme and schema-free. indexes all the data without requiring you to deal with schema and index management.
  • It’s also multi-model, natively supporting document, key-value, graph, and column-family data models.
  • Industry first Globally distributed, horizontally scalable, multi-model database service. Azure Cosmos DB guarantees single-digit-millisecond latencies at the 99th percentile anywhere in the world, offers multiple well-defined consistency models to fine-tune performance, and guarantees high availability.
  • No need to worry about instances, servers, CPU , Memory. Just select the throughput , required storage and create collections. CosmosDB works based only on throughputs. It has integrations with Azure functions. Serverless event driven solution.
  • API’s and Access Methods- Document DB API,Graph API (Gremlin),MongoDB API,RESTful HTTP API & Table API. This gives more flexibility to the developer.
  • They are elastic Globally scalable and with HA , Automatically indexes all your data.
  • 5 Consistency concepts – Bounded Staleness, Consistent Prefix,Session Consistency,Eventual Consistency,Immediate Consistency.  Application owner has now  more options to choose between consistency and performance.

Summary on Cosmos DB:

Picture2

Example without Cosmos DB:

  • Data Geo replication might be a challenge for the developer.
  • Users  from remote locations might experience latency and inconsistency in their data’s.
  • Providing an automatic failover is a real challenge.

Picture3.png

Example with Cosmos DB:

  • Data Can be Geo-Distributed in few Clicks.
  • Developer do not need to worry about the data replication.
  • Strong consistency can be given to the end users across geo-distributed location.
  • Web-Tier application can be  changed anytime between primary and secondary in few clicks.
  • Failover can be initiated any  time manually and automatic failover is present.

Picture5

Data Replication Methods:

  • Replicate Data with a single click – we can add/remove them by a single click.
  • Failover can be customized any time in few clicks(automatic/manual).
  • Application does not need to change.
  • Easily move web tier and it will automatically find the nearest DB.
  • Write/Read Regions can be modified any time.
  • New Regions can be added/removed any time.
  • Can be accessed with different API’s.

Existing data can be migrated:

  • For Example if we already have a mongo app we can just import and move them over.
  • Just copy the mongo data into the cosmos and replacing the URL in the code.
  • We can use Data migration Tool for the migration.

5 Consistency Types:

There are 5 consistency types where the developer can choose according to  the requirement.

  • Synchronous – eventual consistent End users get the best performance.(but data will not be consistent)
  • Strong – will only commit the database to the write/read regions after the copy is successful.(consistent data across all regions)
  • Bounded –  Option to set Bounded staleness to 2 hour. If it is set to 0 then it becomes strong consistency.(We can select few interval up to which the consistency can be strong till the replication is completed to read regions)
  • Session – It is synchronous but not consistent for all users. Clients who commits the data can see the fresh data.
  • Consistent Prefix – Copy of the data order will be maintained and they will see the uniform data.

Based on these 5 consistency concepts, the application developer can  decide to choose either to give the best performance or a consistent data to the end users.

Example of Eventual Replication:

The data is not consistent for the read region and users in write region alone can see the fresh data.

Picture6

Replicate Data with a single click:

Provides more regions to replicate just in few clicks which are more than Amazon and Google combined.

Picture7.png

Available API Methods:

Picture8

Recommendations from Microsoft:

  • According to Microsoft, Cosmos DB can be used for “any Web, mobile, gaming and IoT applications that need to handle massive amounts of reads and writes on a global scale with low response times.
  • ” However, Cosmos DB’s best use cases might be those that leverage event-driven Azure functions, which enable application code to be executed in a serverless environment.
  • Its not a relational database.Its not a SQL server not good at random joins . Does not matter what value of data it is as long as you don’t do joins.
  • Minimum is 400RU per collection, which would be around 25 USD / month. Each Collections are charged individually, even if they contain small amounts of data. Need to change your code to put all of documents into one collection.
  • It’s a “NoSQL” platform with SQL on top of it for SQL operations better not to do multiple joins.

Thanks & Regards
Sathish Veerapandian
MVP – Office servers & services

Troubleshooting rapid Log Files and Database growth in Exchange 2010

In this article let me discuss about troubleshooting unusual growth in log files and database in Exchange 2007 and Exchange 2010.

It’s always better to check and validate first before we jump into any troubleshooting. It’s better to check first from what time  the log files and the database starts growing rapidly.

We can probably filter out events for any related information about logs and database in the application logs in the affected mailbox server.

Also we need to collectively gather all information about the list of third party software’s installed and running on the affected mailbox server.This could be the problem as well if the handshake between Exchange and third party agent is broken.

I have classified  few troubleshooting steps which would help us in narrowing down and rectifying these kind of issues.Below are the list of troubleshooting that can be possibly done .

 

EXMON

The first step that would easily help us in identifying the problem is using EXMON tool to see if there are any user’s unusual activity which causes the log files to grow rapidly.

You can download the exmon tool from the below location

http://www.microsoft.com/en-us/download/details.aspx?id=11461

Run exmon tool and sort the the value by % cpu and look for high CPU consuming users. Also you can check the log bytes column to monitor the log growth.

If you identify any potential users then you can see the following things

  • See if there are any email with a large attachment which is stuck on the outbox.
  • Also you can monitor if there are any spam mails circulating on the affected user’s mailbox.

 

Exclude FILE LEVEL AV SCANNING

If the AV scanning running on mailbox servers is not aware of exchange databases and log files then this will definitely cause the transaction logs to grow rapidly.

Following things can be checked

  • Check if there are any recent updates that happened on the AV scanner on the mailbox servers which might remove  exchange databases and log files exclusions.
  • Ensure that AV exclusions are set for Exchange databases and Log files on the AV scanner in the mailbox servers.
  • Disable AV scanning on the affected mailbox server where transaction logs and DB are growing rapidly. Monitor for few hours and see the log files and database growth and compare the results.

 

 

OPEN RELAY

Check if the server is an open relay to the internet, there will be tons of transaction logs. You will also usually see a bunch of items in the junk mail folder. So ensure that the environment is not open for relay as huge amount of spam mail circulated also will cause the server performance and server to send out more number of spam messages internally as well as externally.

It’s better to have a look at all the queues in all hub servers to ensure that no spam messages are been sent out from our organization in such kind of scenarios.

 

 

PUBLIC FOLDER REPLICATION

  • Check if there are any PF replicas initiated recently on the affected mailbox server recently as it could cause the problem.
  • Check in the message queue if there is more number of public folder replication messages.

 

 

CHECK BACKUP CONFIGURATION

If you have any backup running in the environment ensure that the backup is scheduled properly. Ensure that you are running only full and incremental backups as only these two types of backups will truncate the logs and the rest wouldn’t have the capability to truncate them.

Imp Note:

If a server hosting the data being backed up is a member of a database availability group (DAG) and hosts both active and passive database copies, you must disable the Microsoft Exchange Replication service VSS writer. If the Microsoft Exchange Replication service VSS writer is enabled, the backup operation will fail.

 

To disable the Microsoft Exchange Replication service VSS writer, perform the following steps:

  • Log on to the server by using an account that has local administrator access, and then start Registry Editor (regedit).
  • Navigate to HKEY_LOCAL_MACHINE\Software\Microsoft\ExchangeServer\v14\Replay\Parameters.
  • Add a new DWORD value named EnableVSSWriter, and set its value to 0.

Exit Registry Editor and then restart the Microsoft Exchange Replication service.

 

 

Bulk Mailbox Move

If there are any recent bulk mailbox move that is happening that could generate lot of log files if few of the mailboxes are larger in size and if it has more corrupted items. Probably for this as a temporary fix until the mailbox move completes you can enable circular logging on the source and the destination mailbox database. Ensure that you need to disable circular logging once the move is completed as enabling circular logging always is not a good choice.

Hope this helps in scenarios where we come across scenarios  in troubleshooting unusual  rapid growth in database and log files.

Cheers

Sathish Veerapandian

%d bloggers like this: