March 2008 - Posts

• SharePoint - Get to know the directory structure in the 12 Hive

  The 12 hive is where everything in SharePoint happens and is where all the binaries, configuration, web pages, web controls, help files and other resources reside.  As a developer writing your own features for SharePoint, it is vital that you learn where things go and where you should be targeting your resources.  Without having an understanding of this, you have lost the battle before you have even started. This article forms part on the "Series of posts on the art of provisioning in SharePoint". A while back I posted an article SharePoint - The 12 Hive which pointed you to the location of the 12 hive, however the article was for general users and not aimed at administrators or developers.  This article however is aimed at an administrator and developer with a basic overview of what you can find where.   I will follow up on this post with a post on the TEMPLATE folder as this is the heart of the site and developers should really get familiar with the Template folder. Below is a list of core folders in the 12 hive. \ADMISAPI The directory contain the web service used used by the SharePoint Central Administration and appears as a virtual directory. \BIN The directory contains all the core binary files, utilities that are used by Windows SharePoint Services.  Your command line tools such as STSADM.EXE reside in this folder \BIN\LCIDD A directory will be created for each language will be created that contains language specific binary files. \CONFIG This directory contains a set of configuration, binary and resource files used by SharePoint.  Some files are the default values which will be copied to web site instances. \Data SharePoint uses this directory structure for the indexing services where content will be indexed. \HCCab\LCID This directory has a set of cab files containing manifest and content information used by the SharePoint help sytem \Help The folder contains a compiled html help file (.chm) used by the configuration wizard. \ISAPI This directory contains all the standard Web Services for SharePoint and some additional DLL's, resources and configuration files that the web services use.  Every web application provisioned in SharePoint will have a virtual directory strong>/_vti/_bin that points to this directory, thus giving each web application it's own set of web services. \ISAPI\HELP This directory contains all the help files used by SharePoint.  The folder also contains LCID sub directories for each language installed thus globalising the help system. \LOGS This is the directory that you will visiting frequently whilst doing development and administration as it contains the log files of what SharePoint did and what errors occurred. \Resources This directory contains the core.resx file used for creating language packs for SharePoint.  If you are going to be localising your SharePoint sites with different languages and cultures, this is the folder to do it in. \TEMPLATE This directory structure contains the core web site functionality in SharePoint, that is the features, templates, configurations, resources of a web site.  What is important to note about this directory structure is that the Virtual Path Provider hides and masks this directory structure, thus it appears under each web site, list in a completely different structure. This directory structure rather large and it fully warrants a complete post by itself.  As a SharePoint developer you should completely familiarise yourself with this folder structure. Tips and suggestions Open up the 12 Hive on your computer and spend the next five minutes looking through the directory structure. If you plan to be administrate or be a SharePoint development guru. Look at the content. On your server add the \bin directory to PATH environment variable, as it will search this folder when you are at a command prompt. As a developer you will be updating this directory structure with your own content, and it is important that you do it through the correct mechanisms. DO NOT CHANGE ANY FILE DIRECTLY in the 12 Hive, unless like pain, wasting time resolving bugs and having to reinstall SharePoint. Learn how to provision properly. Posted Mar 24 2008, 01:17 PM by Brett with 1 comment(s) Filed under: SharePoint

• What is BizTalk Server?

  Microsoft BizTalk Server is a server side product that is aimed to provide integration between system as well as business process automation.  The product is used in many organisations to integrate system ranging from internal systems to partner, vendor and customer system. It also allows you to implement a workflow (called an orchestration) to manage, automate the flow of data throughout the organisation. The business challenges BizTalk addresses Companies and their systems have evolved over the past decade and the solutions that these systems need to deliver are complex.  These systems have also introduced a whole new set of challenges that professional need to look at, these are System isolation Using an integration tool such as BizTalk allows you to keep your systems independent and disparate.  This allows you to focus on the functionality of a single system and not on trying to make a single ERP solution that does everything.  Large systems which I term as a "Mother" systems, try to do everything, changes normally are complex and require long development runs to implement simple changes. Business reengineering also becomes a nightmare as the impact of a business change may cripple the organisation due to  the fact the development may take too long and cost a far too much.  The reasons I call them mother systems, is they are normally the "mother of all stuff-ups". System integration Through BizTalk, you are able to isolate and then integrate systems by sharing data between systems.  BizTalk will be responsible for getting the data from system A, then passing it through an orchestration allowing you to process, route, apply business rules and finally deliver it to one or many systems.  This does require the organisation to adopt a Service Oriented Architecture (SOA) over a period of time. Supply chain management Integrating with many customers, suppliers and trading partners can be really tough, however one will typically find that the Business to business data transmitted between these companies is very similar, however normally in different formats and over different technologies.  To implement this end-to-end supply chain for each business partner takes a lot of development time, thus incurring large expenses especially if there are many partners.  Using BizTalk you will be able to reduce this time drastically and improve the time-to-market to the partner. Data integrity across systems Through routing of information between systems one is able to keep data synchronised where changes made will be reflected in all other systems.  Not only does this improve the quality of the organisations data, it assists the Business Intelligence and the relevant information workers to get a reporting across the organisation. Business Processes Many companies are exceptionally week in managing critical business procedures and typically work in on different steps of a procedure on different system.  This poses many business challenges to users especially with communication, timing and managing the process.  Using BizTalk one can route information across the organisation, thus assisting business adopt formalised process management for critical processes. This assists the business in formalising their business process and in doing so weaknesses will be identified and hopefully rectified.  This will enter the company into a new age of process management. Tracking With business processes spanning many departments across the organisation, information is hard to track.  Using BizTalk one is able to track the movement of data in a real time manner as well as report on the information and where it is within a process. Change management The only constant in the information technology industry is change, and this change is complicated by both changes in technology as well as within the business.   As an IT team, we need to be able to quickly adopt to these changes by implementing new systems and processes.  Using BizTalk one can achieve this through the use of adapters, maps, business rules and orchestrations. History of BizTalk BizTalk is definitely not a new product and it has gone through multiple releases, and as with all Microsoft products it has matured and stabilised over the past few years. Below is a short overview to the history of BizTalk. 2000 - BizTalk Server 2000 Microsoft released BizTalk late 2000.  The BizTalk editor had a Mapper for translating data, a management desk for tracking data.  Orchestrations were introduced to manage business processes. The product hade protocols for EDI, HTTP, HTTPS, MSMQ, SMTP, File transfer.  An enterprise and standard version was released. 2002 - BizTalk Server 2002 Several new features were introduced such as the integration with Visual Studio .NET.  SEED was introduced assist configuring with trading partners.  BizTalk integration with Microsoft Operations Manager (MOM).  There was a big drive was on web services.  Changes were made to orchestrations with transaction support.  The mappers were changed to support mixed XML.  Developers could create functiods.  There were also changes to many adapters with a few new adapters. 2004 - BizTalk Server 2004 Changes were made to how messaging and orchestrations integrated.  The orchestration designer was significantly changed.  Human Workflow Services, Business Activity Services added.  A welcome content-based routing mechanism introduced.  Security changed and SSO introduced.  The rules engine and integration of rules into orchestrations.  Pipeline designer for assembling and disassembling messages.  Better support for industry standards. Used Microsoft .NET 1.0 2006 - BizTalk Server 2006 Used Microsoft .NET 2.0.  The deployment model changed with a more modular approach.  Easier to configure.  Business Activity Monitoring was introduced.  Health and Activity Tracking introduced with improvements to monitor.  Development support in Visual Studio 2005  The processing of XML changed with some welcome changes should there be failures in messages.  Recoverable interchange processing.  Developers could create flat-file schemas with a wizard.  Changes to the orchestration designer.   BizTalk application introduced to allow solutions to be separated.  Monitoring was radically changed.  Many improvements were made to adapters with an introduction of new adapters.  BAM improved and enhanced, the BAM portal, alerts, notifications, Web Portals and BAM interceptors added. 2007 - BizTalk Server 2006 R2 Runs on Microsoft .NET 2.0 with Microsoft .NET 3.0 using Windows Communication Foundation and Windows Workflow Foundation.  RFID functionality added to BizTalk  EDI Adapters changed.  The BizTalk Adapter Pack introduced allowing adapters to be created and consumed with .NET Framework applications.  BAM Interceptors for WF and WCF introduced.  SSO 4.0 adopted. Tools and Services There are many tools, engines and services offered by BizTalk and I will breifly discuss each one as to give you a basic understanding to what the product entails. Messaging engine BizTalk processes data in the form of messages and communication between systems is based on the exchange of messages.  Thus the messages engine is essential to BizTalk Services.  The BizTalk messaging engine performs the following Uses adapters to physically send and receive data in the form of messages to and from different systems. Pass the messages through pipelines to encrypt\decrypt, compress\decompress the message Format the external data into XML for processing Map the data from one data structure to an internally understood data structure Extract key data from the messages that will be used for routing the messages. Track documents Deliver the message to the MessageBox. Route data to the necessary orchestrations for processing and finally to send adapters to be delivered to destination systems. Adapters BizTalk's prime objective is to route data between different systems which is controlled by a business process, this data is represented in the form of messages.  However these systems that need to be integrated run on different technologies, written by different vendors with different development tools and are transported with various different protocols.  BizTalk has an extensive library of adapters that allow communication between systems over well defined and common industry protocols, these include SMTP, POP3, FTP, Web Services, MSMQ, MQ-Series.  There are also product specific adapters such as Siebel, SAP, SQL Server which have been written to interact with specific software.  There are also adapter such as the EDI and SWIFT adapters allowing connectivity to the SWIFT and EDI networks.  This list of adapters is by no means extensive and is only for you to get an idea of what they are, there are many, many different adapters; some come with BizTalk others you need to pay additional licence fees to use and then there is a whole bunch of commercially available adapters.  Developers can then also write their own adapters. Mapping When dealing with data from different systems, there is always the need to translate data from one the source data structure to the destination data structure.  The mapping engine does exactly this, however it includes an extensive functoid library that allows you to do data manipulation, aggregation and transformation.  This too is customisable and developers are able to fully code against this mapping engine. Internally BizTalk uses data in an XML format and translation is done through mapping; basically this is BizTalk's way of doing XSLT.  External systems may not unnecessarily use XML and this is where the Pipelines and adapters come in. Orchestration services The BizTalk orchestration services are used to define, manage and execute business process.  These processes included looking at the data and making decisions on the data.  Typically the orchestration will engage in a process of getting data related from others systems, correlating the data, making decisions on the data and based on the outcomes of these decisions send messages to destination systems. A single BizTalk orchestration may engage with obtaining or updating data from many different systems.  Transactional integrity of this data can then be maintained by BizTalk across many different system. These orchestrations are also not time bound and may run for hours, weeks, months and even years.  Pipelines Pipelines are typically used to encrypt\decrypt messages, compress\decompress message and mime encode the messages.  Additional conversion may be done in the pipeline to get the data into a format the mappers understand for receive pipelines or to a format the adapters understand in a send pipeline. MessageBox Database The MessageBox is a SQL server database that BizTalk uses to store, route, pass to orchestrations and finally to send ports.  The MessageBox is thus used to track the entire life of the message over time.  The MessageBox also stores versions of the message as data may change through the orchestration.  This then allows information workers and administrators to troubleshoot data. Application development tools BizTalk development is development is done within the Visual Studio environment and when BizTalk is installed onto the developer's box a few project templates are added to Visual Studio. The first thing to do is to create a BizTalk project and then add the following items to your project Schema.  The BizTalk Editor allows you define XSL schemas which are used to describe your data. Map.  The BizTalk Mapper allows you to define transformations from one schema to another schema.  The conversion allows you to add functoids to the map to perform data manipulation, conversion and aggregation. Pipeline.  The Pipeline designer allows you to define the process for incoming and outgoing messages. Orchestrations.  The Orchestration designer allows you to model your message flow according to a business process. SDK.  An advanced developer can use the BizTalk SDK to define custom pipelines, orchestrations and adapters for BizTalk. Business Rule Engine The Business Rule Engine allows developers to define basic rules that can be implemented within orchestrations.  These rules can be used to define calculations, decision structures thus affecting the branching of an orchestration based on the rule conditions.  Rules are exposed to users through the Business Rule Composer, allowing non-developers to change and adjust the rules using non-technical terminology.  These rules then are applied and orchestrations are immediately adjusted. Business rules are ideal for giving users a way to control the execution and outcomes of orchestrations without having to engage a developer to change the orchestration. Message activity and tracking Health and Activity Tracking (HAT) allows administrators to monitor, track and resolve problems with business processes.  HAT provides a valuable and indispensable tool for administrators to manage and troubleshoot the BizTalk Serves. HAT includes the following benefits Tracking to see start, end times of messages as well as the current status Health allows you to view the current state of a process Monitoring live data allows you to see what is happening on the server, thus identifying issues Analysing live and archived data and tracking the complete message flow, changes to messages and the route the message whet through and orchestration. Orchestration debugging Business rule activity tracking Business Activity Monitoring (BAM) BAM provides information workers the ability to view running process and draw reports on process statistics and key information.  This information is near real and can be configured with alerts and notifications, the information worker then can react to different situations based on the activity.  In 2006 R2, developers can write their own components that can publish information to BAM.  The BAM information can be extracted and manipulated within excel or through the BAM portal running on SharePoint. BAM gives you the ability to Access views of data to understand what is happening Search for activities and query key data Summarise and aggregate data to draw statistics Drill down from summarised data to detailed data. Configure alerts and notifications to notify users when certain thresholds have been exceeded. Escalate issues to administrators from the BAM portal Business Activity Services (BAS) BizTalk provides a mechanism to manage relationships between trading partners.  Through trading partner management (TPM) and using the Business Activity Service, our are able to define the partners, define the protocols and formats of messages used in the relationship.  Developers use this information in orchestrations to define dynamic ports where information can be routed to and from the partners in different formats.  Through an easy interface, information workers can easily add, edit and manage partners.  BizTalk will apply these changes, thus preventing development and changes. RFID BizTalk RFID is an RFID device-level management set of components introduced BizTalk Server 2006 R2. Using RFID, you can eliminate many existing challenges faced with RFID devices as it allows a uniform way to discover, communicate and manage these devices. BizTalk RFID includes a SDK that developers need to build vertical applications that are reliant on RFID tag information -- everything from track-and-trace to asset tracking and inventory control.  Although independent of BizTalk Server, the RFID tag information can then be sent to BizTalk and then passed through orchestrations. Posted Mar 23 2008, 12:52 PM by Brett with 2 comment(s) Filed under: BizTalk

• Overview of features, templates, site definitions, solutions in SharePoint

  This article forms part on the "Series of posts on the art of provisioning in SharePoint".  In this post I will discuss the different ways you as a developer or administrator can package and deploy your content to SharePoint. When learning SharePoint, it is important that you understand the difference between features, templates, definitions, solutions and configurations.  If you are new to the product and doing customisation of sites, this may be tremendously confusing and I will try to demystify this in the post. Feature A feature provide a a developer to solve a business need by adding SharePoint content, such as page templates, lists, content types, web parts, work-flows and events into a self contained unit.  Typically a developer will create a feature to solve a common set business requirement.  In fact the default installation of WSS or MOSS installs many base features, such as the Team Site, My Site, Wiki and blog. Features can be installed and then activated within a site, when the feature is activated the lists, content types will be created on the site and the functionality provisioned.  Features can also be deactivated thus removing the functionality from the site. After a feature has been created, it is typically packaged into a solution file.  This solution file is then copied into the SharePoint configuration database and deployed to all servers in the farm.  Features are deployed to the 12 Hive\TEMPLATES\FEATURES folder. The benefit of developing features is that you do not have to build a huge and complex site in a site definition, rather create many smaller self contained features that are then activated by the site definition.  Not only does this break down the complexity of a site, it also facilitates reusability of features in other sites. Tips and points Create your custom functionality in a feature. Create many small features. Think reusability. Deploy your features in a solution package. Site Template   A site template allows you to create a template for a web site, once the template has been created it is easy for site administrators to create new web sites based on the template.  The template can contain many SharePoint contents, such as lists, pages, menus, enabled features.  The template is easy to create and can be done in the web front-end, through tool such as SharePoint designer or it can be developed by a programmer. There are many templates available and these can easily be downloaded off the web.  Most templates provide a starting point of a site allowing you to further customise it for your business needs. Tips and points Easy to create and copy a site from front-end. Can only use existing installed features. Site Templates are deployed to the configuration database. Cannot create application pages or site pages with code. As a developer, slap together a site, export the site and then open the cab and copy content into your own site definition. Templates A template is exactly the same as a site template, however you are able to create templates at a more granular level.  That is you can create a template just for a list or library. This too can easily bee created from the web front-end.   Once created these templates can be imported into a new site. Tips and points Easy to create and copy lists, content types and libraries from the front end As a developer, export a list, content type or library to a template and then open the cab file and copy the content to your site definition. Site Definition   Site definitions, sometimes referred to as admin templates, enable developers to create functionality using features, web parts and custom DLL's as well as SharePoint elements and lists.  The site definition is packaged into a .wsp file and installed by the administrator.  The site definition can then be deployed to all servers within the farm using a solution package. Site definitions for are identical to Site Templates, however they are created by a developer and not generated from the front-end.  The site definition can include custom Web Parts, Workflow's, DLLs, application pages as well as custom server side controls. Tips and points A site definition is a site template with more flexibility for custom development. Site definitions are deployed to the 12 hive. Can create multiple configurations of the site definition, thus giving reusability Can create application pages with code. Activates your custom features when deployed. Use site definitions, features to create your Line-of-business (LOB) applications. Deployed using a solution package. Solution Package A solution is a deployment mechanism introduced to Windows SharePoint Services 3.0.  All features, web parts, extra DLL's, ASP.NET pages and controls, resources and finally SharePoint elements defined in CAML are added to a CAB file.  The cab file also has a deployment manifest (manifest.xml) that defines what needs to be deployed to front-end servers. The power of packaging your custom components into a solution is that it is easily deployed to web servers in a farm.  This is done by an administrator who will add the solution to the configuration database.  The administrator will then deploy the solution to all web servers in the farm,  the powerful provisioning engine will then extract the contents of the solution on all web servers. Tips and points Always package your custom development into a solution package. Solution packages are copied to configuration database and then deployed to all web servers in the farm. Do NOT copy directly into the 12 hive, use a solution package to do it for you. Always thing server farm. Configuration A solution package may contain many different types SharePoint content which you may want to activate and reuse on different sites.  You are able to create a site template that is used to create a new site, however template within the site template definition, many different configuration can be created.  Each configuration can install and activate different features, pages and web parts. Using configurations you are able to create different variations of a  site template, with each configuration using different features and content.  The configurations thus give you the ability to create multiple templates using a common set of features, list definitions and other SharePoint content. For example you could create a template for IT project and in this template you would define lists, web parts and features.  Then you could create a "New development project", a "Maintenance project" and even a "Software deployment project".  When a site is created, the site administrator can create a new site based on one of the three "templates" Tips and points Think reusability. Create variations of your site definitions. Posted Mar 15 2008, 10:20 AM by Brett with 4 comment(s) Filed under: SharePoint

• A series of posts on the art of provisioning in SharePoint

  In the past year, I have come across many in-house custom solutions in SharePoint.  The most striking and common trend I notice is how the developers / administrators are provisioning their sites.  In almost all the cases they have completely missed the point of SharePoint and the power of its provisioning engine.  This prompted me to write a series of blogs on provisioning correctly to SharePoint. I will start off by painting a few scenarios of traditional manual provisioning and then move to describe the provisioning engine of SharePoint.  I will then follow this with templates, site definitions, solutions and configurations.  Following that I will give a few posts on creating different projects in order to deploy successfully to SharePoint from a developer and designers point of view.  Finally I will tackle some issues around taking solutions from development to test to live and then on how to do upgrades on the site. Traditional deployment To full appreciate SharePoint and its powerful provisioning engine, one needs to understand the traditional approach to deploying web solutions.  The process is lengthy and requires many different people set the web site up.  For the sake of simplifying this let us make a presumption that the following has been done Hardware procured. Operating system and other software installed and patched. Firewalls configured to secure traffic. Server connected to network and communication and routing work. Domain Name Service (DNS) has been set up and name resolutions to the server work. Security applied to server and all unnecessary services disabled. Backup and other operational procedures implemented. Now I will also presume that some custom line-of-business solution has been developed and tested. Scenario - Deploying a simple web application The Database Administrator (DBA) will need to install to create a database for the application and run any scripts to create tables, procedures and other database objects.  Security then needs to be applied.  Finally backup, maintenance and other operational tasks will need created to manage the database. The webmaster will need to create a web site within IIS followed an application pool to isolate the web application. set up authentication. set up other IIS settings, such as load balancing, home folder, initial files. Likely the web master will have to set up The webmaster then needs to copy over the files web site The web.config file may need to be modified for production settings. DLL's may need to be registered in the Global Assembly Cache. Scenario - Web farm Now the scenario above is a lot of work, however it becomes even more complex in a web farm.  As you are most probably aware, creating high available sites is just a matter of scaling out.  This becomes hard work for the web master has he\she has to deploy the site on every server on the farm.  This becomes a lot of work if you have say a farm of 10 and more servers. Scenario - Changing the content on the site in a farm Now this becomes a lot of work again, as each the web server needs to be installed with the upgrade. Timing becomes an issue when the site is in a web farm as you need to upgrade the entire site; it is very unlikely that the old and new version of the software is compatible and can run together.  This typically means down time or some cleaver tactics such as taking down a few servers, upgrading them and then brining the new site online, finally then upgrade the rest of the servers.  Timing is an issue especially when working with geographically dispersed sites. Scenario - Adding new servers to a farm As the load on the site increases, it is very likely that more servers will be added.  Now the web master needs to really be on top of her\his game and needs to keep track of every patch, version, hack and configuration change made to the web site.  The new server will have to be installed and every change made to the site applied to the site.  If the webmaster makes a mistake and forgets to do one thing, the serer is pretty much useless. Scenario - Multiple sites on each server With the power of the servers, it is so common that a single web server host many sites and each site has its own version, configuration requirements and patching that needs to be done.  This makes the webmasters life a hectic and inevitably mistakes will be made.  The situation is even exacerbated in a farm with multiple servers hosting the web sites. Why SharePoint provisioning At the heart of the WSS engine is a well thought through provisioning engine, in fact SharePoint is a provisioning engine with a few bells and whistle features added to make the product sell.  So what can the provisioning engine do for you It can create web sites within IIS. It can create virtual directories. It can configure load balancing for a web site. It can deploy DLL's to the web application or to the Global Assembly cache. It can configure the security of the web site. It can modify web.config files. It keeps track of all of the changes and in order of what change was made. Adding a new server to the farm, will automatically have these deployed. When a new solution is added to the farm, it is automatically copied and activated across the entire farm. Simply said, all of the above problems and issues that make deployment an web masters living hell simply vanish.  This takes a huge burden off his/her shoulders allowing other day-to-day issues to be solved.  The developer challenge SharePoint is a truly remarkable provisioning engine that allows for easy web site deployment over a huge farm of servers.  However, this utopia is fraught with challenges and issues.  Firstly, the development team need to build web sites with SharePoint functionality.  The solutions need to be developed in such as way that the are added to templates and features. Developers need to learn what they can and cannot change, especially the latter. There are many things in SharePoint that can be reused and leveraged in systems. Developers need to learn what and when to use these features. Many things can be customised in SharePoint, however there is the common wrong ways and then the right way. The last trick to learn is how to package and deploy the solution. Developers in general have a lot of learning to do, and with a bit of time and effort they will quickly get a respect for SharePoint.  I am sure many of my students that have attended my WSS development courses will agree to the power of the provisioning engine.  Please feel free to comment of your experiences with this power of provisioning. Posted Mar 08 2008, 05:19 PM by Brett with 2 comment(s) Filed under: SharePoint, SharePoint Designer