An Introduction to Metadata and Development as a Service
Abstract
Development as a Service (DaaS) is a suite of features that allows you to use traditional development practices for creating on-demand applications. DaaS expands the cloud computing development process to encompass external tools such as integrated development environments, source control systems and batch scripts to facilitate development and deployment.
The Force.com platform enables DaaS by providing a rich and powerful metadata model and API. The metadata provides a mechanism for describing your application and the API encompasses a set of web service methods that provide programmatic access to the configuration and source of your application. The Force.com IDE and Migration Tool are examples of client applications built upon this API, which deliver on the promise of DaaS.
After reading this article you will understand the role of metadata and the metadata API on the Force.com platform. You will also see how two popular tools, the Force.com IDE and Force.com Migration Tool, make use of the API. The final section of the article shows a few examples of programmatic access to the metadata API, hopefully motivating you to create your own tool based on the metadata API.
Metadata and the Metadata API
Metadata lies at the heart of the Force.com platform. The Force.com IDE, the Force.com Migration Tool, and even the user interface you use to add records in your application, are powered by metadata. The following sections define metadata, and show how it can be used.
Defining Metadata
Most applications manipulate data, which is often stored in a database. Typically this data is information such as Account details or Contact information. However, you can consider the source code for the application itself as data that provides the information on how your application looks, feels and functions. This information can be referred to as metadata.
Traditional development paradigms rely on software developers to take this code and configuration information, compile, package and deploy it before users could begin using the application. The Force.com platform however, delivers software as a service and immediately uses the metadata to provide an on-demand application directly in the cloud.
On The Force.com platform, metadata defines a rich set of components including all aspects of Force.com functionality, from the appearance of user interfaces through workflow. Examples of components of metadata include custom user interfaces created using Visualforce, generated page layouts, and even Apex classes for custom application functionality. In short, just about every aspect of an application developed on the Force.com platform, ranging from workflows to the configuration of database objects (that store data!) are metadata.
How Metadata Defines your Applications
To make this concrete, consider the way in which you can define a database object. One way in which you can do this is to log into your Developer Edition environment, and use the Setup menu. For example, you can navigate to Setup -> Create -> Objects, click on New and create a new object. As you can see from the following figure, an object called TodoItem was created, and this object has, among other things, a description, lookup and date field. A name and owner field is also present. Creating this object using the web interface is a matter of pointing and clicking using the Setup menu.
A result of creating the object is that you get a lot of functionality out of the box, including metadata which describes all of the logic and visual presentation that make up the TodoItem.
Here's a look at the data entry screen for this object, which is automatically generated:
You will notice that the page has a date picker for the due date field because the platform takes advantage of the metadata for the object and discovers that the field has a data type of "Date" and so, renders an appropriate user interface element. Likewise, the TodoList field is actually a lookup, so the user interface element provides a button that allows you to look up a related record. Read An Introduction to the Force.com Database to learn more about this functionality.
In other words, you can think of the Setup menu as providing a way for you to create and manipulate metadata that is used to create your applications, and the default user interface as being driven by that same metadata.
Some of the metadata that encapsulates your application is not manifested visually, but rather through functionality. For example, you may define a workflow rule through the Setup menu. It won't have a user interface that is automatically generated, as shown in the above example, but the workflow will add functionality to your application, based on the metadata that defines it. For this reason, metadata is not exactly the same as source code for your application—it's more. Metadata is used by the Force.com platform to configure and shape the default functionality of your specific applications.
Accessing Metadata through the Metadata API
The Setup menu is not the only way of creating and manipulating metadata. The platform exposes a metadata API—a SOAP-based web service—that lets you access metadata in the same way you can access any of your Force.com applications, from any location on the web. Since metadata controls the logic and presentation of your applications, you can develop or shape applications by creating and modifying the metadata directly. In other words, you can write programs that write programs.
The metadata API, discussed in more detail below, exposes two sets of methods via the web service interface. The first allows you to create, delete and update sets of metadata components. For example, you can use these methods to modify the schema of a database object to include an additional field. The second set of methods contains file-based metadata calls that allow you to retrieve and deploy file representations of metadata, which is especially important when migrating between organizations (described later). For example, using this metadata, I retrieved the file representation of the TodoItem object created above. Here is a snippet of the resulting XML file:
<CustomObject xmlns="http://soap.sforce.com/2006/04/metadata">
<deploymentStatus>Deployed</deploymentStatus>
<fields>
<fullName>Description__c</fullName>
<inlineHelpText>Some help text</inlineHelpText>
<label>Description</label>
<type>TextArea</type>
</fields>
<fields>
<fullName>Due_Date__c</fullName>
<defaultValue>TODAY() + 3</defaultValue>
<label>Due Date</label>
<type>Date</type>
</fields>
<!-- snip -->
<validationRules>
<fullName>Due_date_in_future</fullName>
<active>true</active>
<errorConditionFormula>Due_Date__c - TODAY() < 0</errorConditionFormula>
<errorMessage>Due date must be in the future</errorMessage>
</validationRules>
</CustomObject>
As you can see, this representation of the metadata describes the database object. It defines the name, default values, labels and type of each field, validation rules for data entry, and much more.
With an introduction to metadata and the metadata API out of the way, the rest of this article examines organizations and moving metadata, and development as a service. It culminates with a concrete example of interacting with the metadata API.
Organizations and Moving Metadata
The Developer Edition (DE) environment provides a home for your applications and data as you develop your functionality. This environment is referred to as an organization or org. You could begin creating an application by logging into this org using a web browser, accessing the Setup menu and creating your database objects.
You can have many such orgs. If you have deployed your application within your company, you probably did so in a production org, which may have license and data volume restrictions depending on your current salesforce.com subscription. Or you could be testing in a sandbox org, which mirrors some of the data found in your production org, but is separate from the production org and its operation.
One way to look at these orgs is as containers for your metadata. For you to copy an application from one org to another, all you need to do is download all of the metadata from the one, and upload it to the other. The metadata API enables this.
To access or move your data (as opposed to your metadata), you can use the Force.com Web services API or other tools and capabilities designed for this task, from data import tools built into the Setup environment to others such as the Data Loader.
As with other platforms, moving an application system from one org to another will probably involve moving both the source of the application, metadata on the Force.com platform, as well as the underlying data. Used in conjunction, the metadata API and web services API provide this functionality.
The following figure depicts these two APIs as facets of each org: the metadata API for retrieving the metadata for components of an application, and the Web services API for retrieving data stored in the database.
Development as a Service
In the preceding sections I discusssed metadata, the metadata API and the web services API as tools for manipulating applications built upon the Force.com platform. But what happens if we need to expand our development process beyond the browser? Development as a Service (DaaS) is comprised of a set tools and APIs that expand the development process from a browser environment to include external tools, such as code editors, source control systems and batch scripts to facilitate development and deployment. An important use of DaaS features is when migrating metadata components from one organization to another, for example, when making changes in a sandbox organization and then promoting those changes to a production environment.
The metadata API is key to the DaaS vision. It makes development in the cloud considerably richer than a browser-only environment - and provides the opportunities for novel applications that interface with the cloud beyond the data layer. The following two sections look at such applications: The Force.com IDE, and the Force.com Migration Tool.
The Force.com IDE
The Force.com IDE (see An Introduction to the Force.com IDE for a complete introduction) is an integrated development environment built on the Eclipse platform. The Force.com IDE is a client application that brings this familiar IDE environment to developers of applications on the Force.com platform.
Using the IDE, you will have access to editors with code completion and syntax highlighting, features for testing classes and exploring the schema of your objects, access to options that let you deploy metadata to any organization, and to advanced code sharing features that most IDEs support. As a result of the code sharing support, which is usually provided through the use of a source code repository such as Subversion, you can have a team of developers working on the same application.
All of these features of the IDE are enabled through the metadata API. In fact, the IDE supports all current metadata component types. If you save an Apex class, the IDE will save the class locally, as well as giving you the option to send the newly modified class code to the Force.com platform using the metadata API. The class will be compiled there, and any error messages sent back to the IDE. This process delivers a familiar programming environment, connected to the cloud.
As you can see from this figure, the IDE provides a directory structure for your metadata as well.
When creating projects using the Force.com IDE, you can determine which set of metadata objects are retrieved and kept synchronized.
This functionality is produced by the metadata file-based declarative calls, which we'll examine shortly.
The Force.com Migration Tool
The Force.com Migration Tool is an Ant library that lets you move metadata to an org using configuration files and a command line interface. After adding your credentials to a simple build configuration file, you can begin issuing commands to retrieve and deploy metadata. For example, here's a simple command:
ant retrieveUnpackaged
This will invoke the retrieveUnpackaged action in the build script, which will in turn perform a retrieve from the Force.com platform based on the contents of a default configuration file (package.xml). By default, this configuration file specifies (with wild cards) all classes, triggers and static resources— and as a result the retrieve action will bring all of the metadata associated with these objects and place them into a local directory called retrieveUnpackaged.
The output of the retrieveUnpackaged command in a directory structure containing the metadata components. For example:
classes triggers package.xml
Here the classes folder contains all Apex classes and the triggers folder all the triggers. The special package.xml file is an XML package deployment descriptor documented in the Metadata API Developer's Guide that describes the set of metadata components that are now available in this directory structure.
After retrieving the metadata, you can make modifications (for example adding a new trigger, or modifying the source code for a class) and redeploy the changes using a similar command:
ant deployUnpackaged
As you can imagine, by changing a few parameters you can deploy this code to a different org or environment, making it easy to migrate metadata from one organization to another.
Because this is a command line utility, you can also use it in scripted files - adding to your arsenal of DaaS tools. For example, you could script migration from staging to production organizations, script multistage release processes or more.
It's worth pointing out that what these calls do is retrieve a textual representation of the metadata utilizing the file-based metadata API calls discussed previously.
Summary
Metadata and the Metadata API provide a powerful mechanism for controlling logic, presentation and configuration information for applications built upon the Force.com platform. Metadata can be accessed either via the visual tools accessible in the Setup menu of the browser-based development environment or programmatically via the Metadata API.
Development as a Service, or DaaS, offers the ability for development to be extended beyond the browser-based environment to include external tools, such as code editors, source control systems and batch scripts to facilitate development and deployment. DaaS is enabled via the Metadata API.
The Force.com IDE and Migration tool are two examples of DaaS tools that allow the developer to complete typical development activities including source-control integration, automation of tasks and migration betweens orgs.
With a solid understanding of metadata and the Metadata API, developers can leverage the benefits of the Force.com platform with a better understanding of applications and powerful tools that support typical development scenarios and provide the opportunity to extend the platform beyond that which is provided out-of-the-box.
References
- The Metadata API Developer's Guide documentation describes the metadata API in detail, providing many useful examples and a full description of the package.xml deployment descriptor.
- Developer Force provides access to free developer edition accounts, which you can use to start programming immediately. It also provides links to documentation, forums, and more.
- An Introduction to Apex provides an introduction to the Apex language.
- The Force.com IDE page provides access to the Force.com IDE and its documentation, as well as the article An Introduction to the Force.com IDE
About the Author
Jon Mountjoy is the community manager and editor-in-chief at Developer Force. He gets kicks out of learning new things and communicating these to the community. You can find Jon on the Developer Force blog, Twitter, FriendFeed and more. Jon says "Thank you to the awesome Platform Documentation team for their great documentation. Thanks also to Quinton Wall, Jon Plax, Simon Fell, Peter Morelli and Chris Wall for their great suggestions and corrections. Please send me any feedback about the article!"