Tagged: azure storage upload

Sitecore Media Library on Microsoft Azure Cloud Storage with language version

This blog guides to configure the AzureCDN Media Upload. Media files are uploaded to azure at the time of publishing and hence the workflow can also be attached to media files. Also, it works with Media having Language versions When user renders the site, media items are fetched from the Azure but when content editor renders the site in page editor it will fetch the media from Sitecore so that it can be edited easily.

So starting with the Azure account first,

  1. You need to create Azure account. Create a new blob storage and a container. Please choose the name appropriate. Storage Account Name and Primary Access Key is required for connection string.azure account setup
    You are now done with the Azure part.
  2. Now in Visual Studio, Install the NuGet package “WindowsAzure.Storage” in your project.WindowsAzure.StorageOnce the NugetPackage is installed, we are to jump into the code.
  3. Let’s do some config changes. In web.config, add following keys under <appSettings>
    <appSettings>
        <add key="StorageConnectionString" value="DefaultEndpointsProtocol=https;AccountName=AZURE-ACCOUNT-NAME;AccountKey=PRIMARY-ACCOUNT-KEY" />
        <add key="AzureContainer" value="AZURE-CONTAINER-NAME" />
        <add key="OrignalPrefix" value="AZURE-URL-WITH-CONTAINER-NAME" />
    

    OrignalPrefix contains the value like https://testAccount.blob.core.windows.net/ContainerName.

  4. Let’s start with the AzureStorageUpload Code.AzureStorageUploadCS
    Create a class AzureStorageUpload.cs. This class contains the logic to upload/update/delete media to/from Azure Storage.

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Web;
    using Microsoft.WindowsAzure;
    using Microsoft.WindowsAzure.Storage;
    using Microsoft.WindowsAzure.Storage.Auth;
    using Microsoft.WindowsAzure.Storage.Blob;
    using System.Configuration;
    using Sitecore.Data.Items;
    using System.Diagnostics;
    using Sitecore.Diagnostics;
    
    namespace Website.Custom
    {
        public class AzureStorageUpload
        {
            CloudStorageAccount storageAccount;
            CloudBlobClient blobClient;
            CloudBlobContainer container;
            public AzureStorageUpload()
            {
                //use ConfigurationManager to retrieve the connection string
                 storageAccount = CloudStorageAccount.Parse(ConfigurationManager.AppSettings["StorageConnectionString"].ToString());
    
                //create a CloudBlobClient object using the storage account to retrieve objects that represent containers and blobs stored within the Blob Storage Service
                 blobClient = storageAccount.CreateCloudBlobClient();
    
                // Retrieve a reference to a container.
                 container = blobClient.GetContainerReference(ConfigurationManager.AppSettings["AzureContainer"]);
    
                // Create the container if it doesn't already exist.
                container.CreateIfNotExists();
    
                //By default, the new container is private and you must specify your storage access key to download blobs from this container. If you want to make the files within the container available to everyone, you can set the container to be public
                container.SetPermissions(
                    new BlobContainerPermissions
                    {
                        PublicAccess = BlobContainerPublicAccessType.Blob
                    });
            }
           public void uploadMediaToAzure(MediaItem mediaItem, string extension = "", string language = "")
            {
              
                // Retrieve reference to a blob.            
                CloudBlockBlob blockBlob = container.GetBlockBlobReference(mediaItem.MediaPath.TrimStart('/').Replace(mediaItem.DisplayName, mediaItem.ID.ToString().Replace("{", "").Replace("}", "").Replace("-", "")) + "-" + language + "." + extension);
                blockBlob.DeleteIfExists();
                if (string.IsNullOrEmpty(mediaItem.Extension))
                    return;
                // Create or overwrite the "myblob" blob with contents from a local file.
                if (mediaItem.HasMediaStream("Media"))
                {
                    using (var fileStream = (System.IO.FileStream)mediaItem.GetMediaStream())
                    {
                        blockBlob.UploadFromStream(fileStream);
                    }
                }
                else
                {
                    blockBlob.DeleteIfExists();
                }            
            }
    
    
    
            public void deleteMediaFromAzure(MediaItem mediaItem, string extension = "", string language = "")
            {   
                //Get the reference of the blob and delete it if exist.
                CloudBlockBlob blockBlob = container.GetBlockBlobReference(mediaItem.MediaPath.TrimStart('/').Replace(mediaItem.DisplayName, mediaItem.ID.ToString().Replace("{", "").Replace("}", "").Replace("-", "")) + "-" + language + "." + extension);
                blockBlob.DeleteIfExists();
            }
    
            public void replaceMediaFromAzure(MediaItem mediaItem, string extension="", string language = "")
            {         
                CloudBlockBlob blockBlob = container.GetBlockBlobReference(mediaItem.MediaPath.TrimStart('/').Replace(mediaItem.DisplayName, mediaItem.ID.ToString().Replace("{", "").Replace("}", "").Replace("-", "")) + "-" + language + "." + extension);
                blockBlob.DeleteIfExists();
                if (string.IsNullOrEmpty(mediaItem.Extension))
                    return;
                using (var fileStream = (System.IO.FileStream)mediaItem.GetMediaStream())
                {
                    blockBlob.UploadFromStream(fileStream);
                }
                
            }
        }
    }
    
  5. Now we are going to write the code of Publishing, What will do is, when the item get published, we will upload the media to AZURE account. And at the end, we will use this class as a processor and patch it in pipeline.
    using Sitecore;
    using Sitecore.Data.Items;
    using Sitecore.Diagnostics;
    using Sitecore.Globalization;
    using Sitecore.Jobs;
    using Sitecore.Publishing;
    using Sitecore.Publishing.Pipelines.PublishItem;
    using System;
    using System.Collections.Generic;
    using System.IO;
    using System.Linq;
    using System.Threading.Tasks;
    using System.Web;
    
    namespace Website.Custom
    {
        public class CdnPublishing : PublishItemProcessor
        {
            public string Enabled { get; set; }
            
            public override void Process(PublishItemContext context)
            {
                //Check the configuration to run the processor or not
                if (this.Enabled.ToLower() != "yes")
                    return;
                Log.Debug("Performing CDN validations", (object)this);
                Assert.ArgumentNotNull((object)context, "context");
                //Get the Context Item
                Item sourceItem = context.PublishHelper.GetSourceItem(context.ItemId);
                //If the source item is null, get the target item (specifically used for deleted item)
                if (sourceItem == null || !sourceItem.Paths.IsMediaItem)
                {
                    Item webSourceItem = context.PublishHelper.GetTargetItem(context.ItemId);
                    if (webSourceItem == null || !webSourceItem.Paths.IsMediaItem)
                    {
                        return;
                    }
                    else
                    {
                        sourceItem = webSourceItem;
                    }
    
                }
                MediaItem mediaItem = (MediaItem)sourceItem;
                string mediaExtension = mediaItem.Extension;
              //Get the Media Stream
                Stream mediaStream = mediaItem.GetMediaStream();            
                if (mediaStream == null || mediaStream.Length == 0L)
                {
                    if (((MediaItem)context.PublishHelper.GetTargetItem(context.ItemId)).GetMediaStream() == null || ((MediaItem)context.PublishHelper.GetTargetItem(context.ItemId)).GetMediaStream().Length == 0L)
                        return;
                    else
                        mediaExtension = ((MediaItem)context.PublishHelper.GetTargetItem(context.ItemId)).Extension;                    
                }
                AzureStorageUpload azureStorageUpload = new AzureStorageUpload();
                Log.Debug("Starting CDN synchonization", (object)this);
                try
                {
                    //Get Version Information
                    Item versionToPublish = context.VersionToPublish;
                    if (versionToPublish == null)
                    {
                        if (context.PublishHelper.GetTargetItemInLanguage(context.ItemId, sourceItem.Language) != null)
                            versionToPublish = context.PublishHelper.GetTargetItemInLanguage(context.ItemId, sourceItem.Language);
                    }
    
                    if (versionToPublish != null)
                    {
                        //Parameters to upload/replace/delete from on Azure
                        object[] args = new object[] { mediaItem, mediaExtension,versionToPublish.Language.Name };                    
                        Sitecore.Jobs.JobOptions jobOptions = null;
                        Context.Job.Status.State = JobState.Initializing;                    
                        if (context.Action == PublishAction.None)
                        {
    
                            jobOptions = new Sitecore.Jobs.JobOptions(
                                mediaItem.ID.ToString(),                     // identifies the job
                                "CDN Upload",                 // categoriezes jobs
                                Sitecore.Context.Site.Name,         // context site for job
                                azureStorageUpload,                  // object containing method
                                "uploadMediaToAzure",                  // method to invoke
                                args)                               // arguments to method
                                {
                                    AfterLife = TimeSpan.FromSeconds(5),  // keep job data for one hour
                                    EnableSecurity = false,             // run without a security context
                                };
                            Context.Job.Status.State = JobState.Finished;
                            Sitecore.Jobs.Job pub = Sitecore.Jobs.JobManager.Start(jobOptions);                        
                        }
                        if (context.Action == PublishAction.PublishSharedFields || context.Action == PublishAction.PublishVersion)
                        {
                            jobOptions = new Sitecore.Jobs.JobOptions(mediaItem.ID.ToString(), "CDN Upload", Sitecore.Context.Site.Name, azureStorageUpload, "replaceMediaFromAzure", args) { AfterLife = TimeSpan.FromSeconds(5), EnableSecurity = false, };                        
                            Context.Job.Status.State = JobState.Finished;
                            Sitecore.Jobs.Job pub = Sitecore.Jobs.JobManager.Start(jobOptions);                                          
                        }
                        //If the publish action is delete target item, get all the language versions of the item and delete it from Azure
                        if (context.Action == PublishAction.DeleteTargetItem)
                        {
                            foreach(Language lang in context.PublishOptions.TargetDatabase.GetLanguages())
                            {
                                mediaItem = context.PublishHelper.GetTargetItemInLanguage(mediaItem.ID, lang);
                                args = new object[] { mediaItem, mediaItem.Extension, lang.Name };
                                jobOptions = new Sitecore.Jobs.JobOptions(mediaItem.ID.ToString(), "CDN Upload", Sitecore.Context.Site.Name, azureStorageUpload, "deleteMediaFromAzure", args)
                                {
                                    AfterLife = TimeSpan.FromSeconds(5),
                                    EnableSecurity = false,
                                };
                                Context.Job.Status.State = JobState.Finished;
                                Sitecore.Jobs.Job pub = Sitecore.Jobs.JobManager.Start(jobOptions);
                            }                        
                        }                 
                    }
                }             
                catch (Exception ex)
                {
                    Exception exception = new Exception(string.Format("CDN Processing failed for {1} ({0} version: {2}). {3}", (object)sourceItem.ID, (object)sourceItem.Name, (object)context.VersionToPublish.Language.Name, (object)ex.Message));
                    Log.Error(exception.Message, exception, (object)this);
                    context.Job.Status.Failed = true;
                    context.Job.Status.Messages.Add(exception.Message);
                }
                Log.Debug(" CDN synchronization finished ", (object)this);
            }
        }
    }
    

    Now let’s patch this class in pipeline.

  6. Instead of changing the configuration, we can patch the processor from a different file to keep the web.config intact.Create a .config file under App_Config\Include\AzurePublishing.config:
    <configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
      <sitecore>
        <pipelines>
          <publishItem>
              <processor type="Website.Custom.CdnPublishing,Website"
            patch:before="processor[@type='Sitecore.Publishing.Pipelines.PublishItem.PerformAction, Sitecore.Kernel']">          
                <!--If yes, this custom CDN publishing processor will be executed otherwise not. Values: yes|no-->
              <Enabled>yes</Enabled>            
            </processor>
          </publishItem>
        </pipelines>
      </sitecore>
    </configuration>
    
  7. Now, to fetch the media from Azure, we need to make changes in Media Provider. But also we need to make sure that for Page Editor and Preview , the media must be fetched from Sitecore only. And to render the site, media must be fetched from Azure. So here we go,
    using Sitecore.Data.Items;
    using Sitecore.Events.Hooks;
    using Sitecore.Resources.Media;
    using System;
    using System.Collections.Generic;
    using System.IO;
    using System.Linq;
    using System.Text;
    using System.Web;
    
    namespace Website.Custom
    {
        public class MediaProvider : Sitecore.Resources.Media.MediaProvider, IHook
        {
            public void Initialize()
            {
                MediaManager.Provider = this;
            }
    
            public override string GetMediaUrl(MediaItem item)
            {
                string mediaUrl = base.GetMediaUrl(item);
                return mediaUrl;
            }
    
            public override string GetMediaUrl(MediaItem item, MediaUrlOptions options)
            {            
                string mediaUrl = base.GetMediaUrl(item, options);            
                return GetCstmMediaUrl(mediaUrl, item);
            }
    
            /// <summary>
            /// Determines if media should be pulling from the CDN or not
            /// </summary>
            /// <param name="mediaUrl"></param>
            /// <param name="item"></param>
            /// <returns></returns>               
            public string GetCstmMediaUrl(string mediaUrl, MediaItem item)
            {
                //verify the domain was set in the config
                if (string.IsNullOrEmpty(OriginPrefix))
                {
                    return mediaUrl;
                }
    
                //Condition to fetch the Azure url only if the site is rendered and not for Preview or editing mode.            
                if(Sitecore.Context.GetSiteName().ToLower() != "website" || Sitecore.Context.PageMode.IsPageEditor||Sitecore.Context.PageMode.IsPreview||Sitecore.Context.PageMode.IsSimulatedDevicePreviewing)
                {
                    return mediaUrl;
                }
                mediaUrl = mediaUrl.Replace("~/media/", "/");            
                //this happens while indexing unless the proper site is set
                mediaUrl = mediaUrl.Replace("/sitecore/shell/", "/");
                mediaUrl = mediaUrl.Replace("//", "/");
                //reference the file in the cdn by the actual extension
                mediaUrl = mediaUrl.Replace(".ashx", "." + item.Extension);
                mediaUrl = string.Format("{0}{1}", OriginPrefix,mediaUrl);           
                string Language = string.Empty;
                if (mediaUrl.Contains("la="))
                {
                    Language = mediaUrl.Substring(mediaUrl.IndexOf("la=") + 3);
                    if (Language.Contains("&"))
                        Language = Language.Substring(0, Language.IndexOf("&"));
                }
                else
                {
                    Language = Sitecore.Configuration.Settings.DefaultLanguage;
                }
                //create the media url accoriding to the naming convention of Azure-media-file name
                mediaUrl= mediaUrl.Replace(item.DisplayName + "." + item.Extension, item.ID.ToString().Replace("{", "").Replace("}", "").Replace("-", "")+"-"+Language + "." + item.Extension);            
                if (HttpContext.Current != null && HttpContext.Current.Request.IsSecureConnection)
                {
                    //if we are on a secure connection, make sure we are making an https url over to the cdn
                    mediaUrl = mediaUrl.Replace("http://", "https://");
                }
               return mediaUrl;
            }
    
            
            public string OriginPrefix
            {
                get
                {
                    return System.Configuration.ConfigurationManager.AppSettings["OrignalPrefix"];
                    
                }
                //set;
            }
    
          
        }
    }
    
  8. Now we need  to replace the Sitecore default MediaProvider with our custom Media Provider.
    In web.config, replace the default MediaProvider with this one.

    <!--<mediaProvider type="Sitecore.Resources.Media.MediaProvider, Sitecore.Kernel" />-->
    <mediaProvider type="Website.Custom.MediaProvider, Website" />
    

 

So, that’s it..  Publish any media item, and you will find it in your Azure storage account.

Things to notice here in Publishing processor , we have used Jobs to publish the items. Reason is, uploading a media synchronously on Azure takes time, and if there are around 1000 + items it will take long time and using Jobs – we can reduce it into seconds.