Defining the Business Workflow: Orchestration Designer

The whole point of BizTalk Server is application integration, and the reason why we want to integrate applications in the first place is to implement some workflow. A business supply chain is an example, specifying the interrelationships of customer orders, manufacturing scheduling, inventory management and purchasing software.

Consider how workflows are usually designed. They are designed at the highest level, which a manager or business analyst understands, although at a deeper level they involve specific technical issues such as transactions. Ideally, such a system is implemented by starting at the highest level too, drawing a flowchart of the individual activities needed to define and implement the workflow. Then the specific details of integrating each activity are added in.

Note that the activities themselves, however, are accomplished by the applications you wish to integrate. It may be necessary to write the odd utility or component, but by and large the idea is to integrate existing applications.

The process of designing and implementing a workflow is known as orchestration. Orchestration Designer is the tool provided by BizTalk Server for accomplishing the design task. It is highly graphical - in fact, it consists of VBA on top of Microsoft Visio (a package used for business diagram creation).

Analysts and programmers start the workflow design by drawing a flowchart of the business process. Systems in BizTalk are built by exchanging messages between applications, so the next step is to specify the messages to be sent or received at each step, and the locations from which messages are received or to which they are sent.

Finally, a designer provides Orchestration Designer with a high-level view of the workflow through the system. This is more important than it sounds at first. Since BizTalk Server uses a messaging model for applications, the workflow itself is viewed by BizTalk as a series of transformations or modifications of data. The flow of data, then, is the application. An example workflow flowchart is shown below. The data flow, not visible in the view below, details how data enters the system in a message, then flows to other messages which are sent out.

In fact, you will see a variation of the flowchart above in the next chapter when we begin our sample application. This flowchart will explain how a field agent sends a message describing a residential property back to a BizTalk Server at the home office. Rules in the flowchart are applied to the message to determine whether certain services should be ordered on behalf of the homeowner. If they are to be ordered, the flowchart directs messages be sent to the business partners whose supply those services.

Orchestration Designer permits you to specify message exchanges with Microsoft Message Queue, COM or script components, or its own Messaging Services. It supports transactions (several actions that must succeed or fail as a whole) and concurrency (two or more processes occurring at the same time). These features are so important to building systems with orchestration that we come back to them later, devoting Chapter 6 to their study.

When you have finished specifying a workflow, Orchestration Designer exports an XML document that defines the workflow. This document, termed a XLANG schedule, is later used by XLANG Scheduler to run your distributed system by controlling how and when your applications are sent messages.

It should be stressed that Orchestration Designer only specifies the high-level semantics of workflows. It orchestrates the activities of lower-level functions, chiefly Messaging Services and your individual applications and components, to compose a complete, distributed business application.

Orchestration Designer and the fundamentals of XLANG schedules are covered in Chapter 2, while advanced features of schedules are discussed in Chapter 6.

XLANG Scheduler

BizTalk Server installs a default COM+ application called XLANG Scheduler. Its function is to host the running instances of XLANG Scheduler Engine. In turn, this Engine is a service that manages and runs instances of XLANG schedules.

This somewhat confusing dichotomy - a COM+ application and a service with a similar name - allows any new COM+ application to host the Scheduler Engine. This permits great flexibility in how BizTalk systems are deployed, managed, and run, because writing a custom COM+ application that hosts the scheduling services allows you to take control of various aspects of scheduling. However, in general, and in this book, you will be using default XLANG Scheduler and its default support for the scheduling services.

The XLANG Scheduler Engine loads XLANG schedules - the XML documents produced by Orchestration Designer - and executes them. At its most basic, this involves calling Messaging Services in response to actions specified in the schedule.

There is a less obvious function of the Scheduler, however, and that is managing the resource utilization of schedules. Schedules can require a comparatively long time to execute. For example, executing the standard protocols of the Web over the public Internet can often require seconds or minutes to complete an operation. BizTalk schedules can wait for the arrival of a message, but since these messages are coming from other applications, even other organizations across the Internet, there is the potential to wait minutes, hours, or even days for a message. You might also be running hundreds of schedules in a production environment. Clearly, the scheduling services must ensure system resources are not consumed by schedules that are not actively executing a scheduled action.

To do this, XLANG Scheduler uses a process called hydration. A schedule is dehydrated when it is blocked waiting for an incoming message. The current state of the entire schedule is serialized to a database. When the Scheduler Engine detects the arrival of the message, unblocking the schedule, the schedule is rehydrated. Its serialized state is read and the schedule resumes where it left off.

An application causes XLANG Scheduler to load an XLANG schedule file by passing it a moniker. The moniker is a name that represents a particular schedule file or schedule instance. A moniker may be thought of roughly (very roughly) as being analogous to a URL. It is a sometimes arcane-looking string that locates a resource in the COM+ system. Here is a sample of a schedule moniker used by XLANG Scheduler:

sked://ibex!XLANG Scheduler/{770C13A4-AB4C-4470-821F-D1FAE5235AE0}/WaitForReply

XLANG Scheduler passes this off to the Scheduler Engine, which then produces an instance of the schedule. Since a schedule can be instantiated many times, on different servers, the moniker syntax not only identifies the schedule document but also the location and instance of the running schedule. This allows programmers to access any particular instance of the schedule for custom purposes.

Defining Message Specifications: BizTalk Editor

The process of providing the low-level details needed to implement a workflow starts at the bottom. Remember, the key to application integration using BizTalk is the exchange of structured messages. Whether those messages use XML or some other parsed text scheme, programmers must be able to specify the structure of a message file - the specification. This is the function of BizTalk Editor.

The Editor is a graphical tool that uses a tree-structure metaphor to build message specification files, or simply the specifications themselves. Editor is influenced by database terminology, which is a reasonable approach given that much of the information used to compose messages will come from databases anyway. Database programmers work with records and fields. A record corresponds to an object or entity, while fields are equivalent to the properties of the object. This works nicely with a database-style table structure in which the rows are individual objects, and the objects are described by the values contained in the row's fields.

Unlike a relational table, however, a specification can nest records within other records. This allows us to describe parent-child relationships like those represented by a database join. The tree that represents a specification in Editor is also analogous to a file directory structure, with records standing in for directories and fields replacing files.

In the Editor window shown below, the name of the specification is SiteServiceDescription. The pane on the left shows the tree structure of the specification, while the pane on the right displays some properties of SiteServiceDescription.

When writing an XML-based specification, a programmer may describe the records and fields in terms of the basic XML data types, as well as the derived types permitted under the XML - Data Reduced (XML-DR) Schemas technology implemented in the XML parser supplied with Internet Explorer 5.0. All the options available in a schema are available to specification builders. This includes being able to specify the cardinality (repetition) of specific records and fields.

In Chapter 3, we will use Editor to create three message specifications for our sample application. One will be XML. The other two, simulating legacy formats, will be non-XML in nature. These specifications will be used by BizTalk to tell the Server how to read our messages.

XML appears to be replacing other text formats for a variety of reasons. Editor permits programmers to take full advantage of XML without forcing them to become XML experts. Because Editor allows the use of formats other than XML, the interface is carefully constructed so that XML terminology does not dominate.

A complete discussion of XML, including its advantages with respect to other text file formats, is found in Professional XML (Wrox Press, 2000, ISBN 1-861003-11-0).

Right out of the box, BizTalk Server and Editor can handle the following text formats:

• XML
• Delimited flatfiles
• Positional flatfiles
• Hybrid delimited and positional flatfiles
• EDI (X12 and UN/EDIFACT)

With some custom coding, BizTalk can be made to understand your own custom formats. This may potentially include binary formats.

Using XML does not force programmers to start from scratch when defining specifications. For example, there are initiatives underway to transform EDI commerce message formats and Health Level 7 medical message formats to XML. BizTalk Server includes a collection of basic XML template specifications for common business messages such as purchase orders and invoices. ADO, the mainstream Microsoft database access technology, now allows database recordsets to be persisted as XML. Programmers can use this feature to model their specifications on existing database table schemas.

Many legacy message formats are in forms other than XML. Both EDI formats, X12 and EDIFACT, use delimited message formats. Versions of Health Level 7 prior to version three also used non-XML formats. Messages built around mainframe systems are frequently written in character delimited or fixed position formats. Editor supports the use of these formats; indeed, BizTalk includes specifications for both X12 and EDIFACT. A programmer can load these specifications and modify them as needed. When working with EDI formats, the Code List tab in the right-hand pane of the Editor window contains a list of all possible fields and records in the type of EDI message that is loaded.

More importantly, Editor permits programmers to specify all aspects of delimited and positional flatfile formats. Programmers may specify how to handle end-of-record delimiters in such files. Delimiting characters may be specified. Field sizes may be specified for positional files. This information is retained and used by BizTalk server to properly parse an incoming message.

The primary file format of a specification produced by Editor is an XML document that includes an XML-DR schema, but also has elements added by the Editor to record specification-specific information. These extra elements are especially important when working with non-XML specifications. For example, XML-DR schemas cannot capture delimiter and field position information. It is the extra elements that capture this information, and they are proprietary to BizTalk.

The BizTalk Framework schema repository works with XML-DR schemas, not BizTalk specifications. For this reason, Editor can export XML-DR schemas. A programmer can design a specification in the Editor, then export a schema and submit it to a repository. Going in the other direction, Editor can import an instance of a well-formed XML message and build a specification and schema from it.

The capabilities of BizTalk Editor, along with the specific syntax of BizTalk Server message specifications, are described in depth in Chapter 3. Microsoft has pledged to migrate to W3C standard XML schemas when they are approved as a formal W3C Recommendation. Schemas have now reached Candidate Recommendation status, one step away from formal approval.

Continued...