New report slams Office 365 compliance features unfairly

by Tony Redmond
Aug 18, 2015

Another report describing Office 365 compliance capabilities has appeared that needs some debate. This one is from Osterman Research, who wrote about Office 365 compliance features for Knowledge Vault. I hate to say it, but I found many holes in the report, some of which can be attributed to Microsoft releasing a slew of new compliance features recently. SharePoint Online, for instance, has picked up a lot of new capabilities that are found in the Office 365 Compliance Center. But in any case, the report sets out a structure that can be used to review the need for compliance within an organization, and that's always a useful tool.

Compliance is a pretty big deal for many companies, especially those who operate in regulated industries. Almost every jurisdiction mandates that companies preserve certain records related to their business dealings. Given the widespread use of Exchange and SharePoint for communications, it should come as no surprise that Microsoft has invested heavily in building compliance features to support their use in business operations.

At least, that statement is true in one aspect. Microsoft has delivered a wide range of compliance features in Exchange 2010 and subsequent releases, including Exchange Online running inside Office 365. The record is less impressive for SharePoint and Yammer, a point hammered home in a July 2015 white paper by Osterman Research, sponsored by Knowledge Vault, Good Technology, GWAVA, KeepIT, Mimecast, and Smarsh.

Any sponsored report has to be treated with a certain care. The sponsor has a view they wish to share with the market and the white paper serves as a vehicle to create a case to support that view. In this instance, vendors like Knowledge Vault have some reporting and auditing capabilities that it would like Office 365 tenants to consider if they’re in the market for compliance tools. Ideally, the text should lay out the case for supplementing whatever Microsoft provides inside Office 365 with third party software so that the reader sees the value contained in what the sponsor has to offer.

The document eventually landed on my desk through the efforts of some very persistent PR. It’s based on a relatively small survey sample of between 128 and 186 “decision makes and/or influencers” drawn from mid-sized and large organizations. Personally, I think this sample size is too small to draw any conclusions, especially when only 25% of those surveyed profess themselves fairly or very knowledgeable about the compliance features built into Office 365, possibly because only 33% use Exchange Online. This is an incredibly low percentage based on my experience as almost every Office 365 tenant I know uses email. Let’s dive into some of the findings to see how its content stacks up.

First, I think that the report highlights some issues that Microsoft still has to deal with when it comes to compliance within Office 365. Even though SharePoint Online has recently gained support for Data Loss Prevention policies, preservation policies, and document deletion policies (all managed through the new Compliance Center console) and has had an eDiscovery Center with the ability to search and hold across sites and mailboxes since SharePoint 2013, it’s true that SharePoint Online can’t archive documents because no equivalent of an archive mailbox exists. On the other hand, SharePoint documents can now be preserved in-place and can also be aged out automatically after a set period, so SharePoint is absorbing some compliance features originally debuted in Exchange.

Neither can Yammer conversations be preserved through retention policies or other means. The same is true for public folder content, which is only now slowly gaining some compliance capabilities.

It’s also true that Microsoft only deals with Office 365 data when it comes to compliance. Still, the current effort to ingest as much data from other sources into Office 365, including corporate communications such as Facebook and Twitter feeds and Jabber and BlackBerry IM, goes some way to addressing that problem, especially if third parties get on board to create more connectors for the Office 365 Import Service.

Another criticism leveled is that Microsoft delivers “good enough” compliance features. The report acknowledges that Office 365 has to service hundreds of millions of users, amounting to some 1.2 million tenants. A specific compliance requirement for one company might therefore not be found inside Office 365, especially if that requirement is specific to a certain industry or country. In any case, the success of Exchange and SharePoint in the on-premises arena is underpinned by an ecosystem of third party software that fill the gaps left by Microsoft.

And while gaps do exist, Microsoft can argue that they are in the process of building out their compliance suite, that the most fundamental features are present, and that others will come in time. In addition, Microsoft can also point to the work that they are doing to move away from application-specific functionality to policy-driven capabilities that can be used to preserve, remove, or find data across multiple applications.

Like any report, the text represents a certain snapshot in time. Those new Compliance Center capabilities seem to be overlooked, so there’s no mention of the ability of compliance searches to address the acknowledged performance problem with very large (more than 10,000 mailboxes) eDiscovery searches. My sources at Microsoft tell me that a compliance search is capable of handling hundreds of thousands of mailboxes and has done so for some of the largest Office 365 tenants. I think that the “wizards” available in the Compliance Center also address the criticism that “Many tasks in Office 365 require familiarity with and use of PowerShell to complete”, assuming that this point relates to compliance functionality.

The new unified auditing subsystem supported by the Management Activity API, designed to extract audit events from all of the Office 365 applications (SharePoint is already supported, Exchange will be soon, and the other applications thereafter) is also overlooked, and the report curiously worries about the 150 MB limit for sending large files, which apparently might cause employees to resort to consumer grade file sharing. The point here is that if you need to send large attachments that don’t fit under Exchange’s limit, they can be shared using OneDrive for Business, which is completely ignored in the report. Indeed, Microsoft is busy changing Outlook and Outlook Web App to encourage people to use “smart attachments” to address this issue. Changing user behavior is not something done overnight and it’s true that people will continue to use whatever they are accustomed to in order to get work done. In the case of DropBox, at least whatever documents are stored there can be retrieved and brought into Office 365.

There’s also a worry that policy-driven message encryption is not supported within Office 365. Apart from Exchange transport rules, it is true that there’s no policy-driven framework for applying encryption to email, but a number of other answers do exist. For example, Outlook Protection Rules can be used to apply Information Rights Management (IRM) templates to messages from the desktop on. And while those rules can be overridden (if permitted) by users, IRM templates can also be applied in transport rules. IRM is a royal pain to deploy on-premises, but it is so much easier to set up and manage within Office 365. And you can also use Office 365 Message Encryption to protect confidential email to external recipients, again enforced through transport rules. Users don’t have to do anything to apply encryption to messages as everything is done as email passes through the transport system.

I can’t find a source to back up the assertion that “Microsoft recommends against having more than one In-Place Legal Hold on a given mailbox at any one time, although Office 365 can handle up to five concurrently (with consequential impacts on performance).” My understanding of the situation is that after five holds are placed on a mailbox, Exchange places the entire mailbox on hold so as to avoid problems that might arise when resolving five different queries. I also understand that this is an implementation choice and five, six, or seven holds don’t really make much of a difference: it’s just easier to preserve everything when multiple holds are in place. I don’t understand why performance is an issue because Microsoft runs more than sufficient servers inside Office 365 to handle tasks of this nature. Besides, users aren’t impacted because the processing of retained documents occurs within a background mailbox assistant called the Email Lifecycle Assistant, which resolves the queries against items before they are removed from a mailbox. Asserting that this is a performance issue is curious.

Also, the assertion that “An eDiscovery search is likely to produce different results each time it is executed. In Office 365, the search query cannot be saved for repeated execution…” seems curious and is not borne out by my experience. Different results can be generated by a search, but that simply reflects the content of the Search Foundation indexes that are being constantly updated as new items arrive or items are removed. Search queries are saved in a search and can be repeated ad nausem.

I could go on but the point is made. I cannot make the case that Office 365 is perfect when it comes to compliance because it is not. Gaps exist as described in the report. It’s also fair criticism to say that the bulk of compliance features are only available to tenants running the E3 and E4 plans (and the new E5 plan when available) and their academic/government equivalents. A case can be made that more compliance features should be available to all tenants, but then you can argue that the folks running small Office 365 tenants probably don’t care too much about compliance because their business doesn’t require these features.

The report says that 59% of those surveyed expect compliance to get worse under Office 365. Well, I guess it all comes down to how a question is framed and the people who are asked the question. My view is that there are more compliance features to be exploited in Office 365 than are available in on-premises software, so I have a problem understanding how the issue could become worse, especially if time is freed up for administrators because they don’t have to do mundane server management tasks any more.

The Role of Third-Party Tools for Office 365 Compliance” is a curate’s egg: good in parts. It contains value in many points that should be considered by those charged with oversight of compliance within an organization. The development cadence within Office 365 and the speed at which new features appear (not all of which are fully baked when revealed to First Release tenants) makes it difficult to stay up to date in what can be a complex area. Some new functionality is overlooked and some existing functionality is simply ignored. In mitigation, I acknowledge that some time elapses between a report being commissioned and when it appears, and the frenetic pace of change that exists within Office 365 makes it terribly difficult for anything written to remain up-to-date.

Read the paper to see what you can get out of it, but remember, you know your business better than anyone, so take the points presented and put it into context with your knowledge to understand where it adds value for your organization.

Follow Tony @12Knocksinna

[Correction: I originally stated that survey data was used for three reports; Michael Osterman has told me that data was used solely for this report.]

Discuss this Blog Entry 2

on Aug 27, 2015

We appreciate Mr. Redmond's blog post on our recently published white paper, but wanted to offer our feedback on his critique of the paper:

The title, "New report slams Office 365 compliance features unfairly", doesn’t really reflect the tone of several of the statements we made in the white paper, including:

- “Microsoft has invested and continues to invest a significant amount of financial resources and effort to build compliance capabilities into Office 365.”

- “Microsoft offers a range of current [compliance] capabilities in these areas, and is evolving its capabilities to increase coverage.”

- “With a platform aimed at hundreds of millions of users, Microsoft recognizes that its compliance capabilities will not meet every need, nor address the requirements of every organization. The aim is to have sufficient systemic capabilities to address broad and general-purpose compliance requirements, in line with certain assumptions about the organization and its IT environment.”

- “Office 365 is a robust platform that offers a number of useful capabilities.”

Quite honestly, we don’t think that we slammed Microsoft or Office 365, both of which we hold in high regard as reflected in these quotes from the paper.

Moreover, we have publicly stated within the past couple of weeks: “Should organizations consider deploying Office 365? Absolutely, since Microsoft offers a robust feature set and continues to enhance Office 365 with new features and capabilities.”

In another multi-sponsor white paper we published earlier this year we stated, “There is no denying that Microsoft Office 365 is a robust offering that offers a wide range of capabilities. Microsoft has taken pains to ensure that Office 365 operates with reasonable reliability and that its features and functions meet the needs of a wide range of potential customers. However, as with any mass-market, technology-based offering there will be deficiencies in specific aspects of the features and functions that many customers require. Because no cloud-based offering can be all things to all customers, many – if not most – Office 365 customers will require third party products and services to supplement the native capabilities of the platform.”

In short, we like Office 365, we think it provides robust functionality, and we think it’s a good value for the money. What we’re not saying is that it can be all things to all users all the time.

Also, Mr. Redmond’s blog originally and inaccurately stated that that survey data cited in the report was used for three reports, but after our conversation he graciously and quickly corrected the statement – a survey was conducted specifically for this report, although we have conducted several Office 365-focused surveys this year. However, the fact that this white paper was also sponsored by five other companies in addition to Knowledge Vault (Good Technology, GWAVA, KeepIT, Mimecast and Smarsh) was not mentioned in the blog. As an aside, the link to “Knowledge Vault” in the blog is incorrect.

While we definitely do NOT think that Tony’s review of our white paper was in any way tainted by the fact that he is on the Advisory Board of a Knowledge Vault competitor, a footnote stating that would have been a useful addition.

Tony wrote that, “Another criticism leveled is that Microsoft delivers “good enough” compliance features. The report acknowledges that Office 365 has to service hundreds of millions of users, amounting to some 1.2 million tenants. A specific compliance requirement for one company might therefore not be found inside Office 365, especially if that requirement is specific to a certain industry or country. In any case, the success of Exchange and SharePoint in the on-premises arena is underpinned by an ecosystem of third party software that fill the gaps left by Microsoft.”

Yes, that was exactly our point, as stated in the paper: “As with any cloud-based offering, these [Office 365 compliance] limitations will necessitate the use of third-party compliance capabilities in order for organizations to fully satisfy their regulatory and legal compliance obligations.”

On balance, we appreciated Mr. Redmond’s blog, but take issue with a few of its points.

on Sep 9, 2015

Michael Osterman is entitled to his view on my review of his report and I accept what he is saying. The inaccurate link to Knowledge Vault (a cut and paste error - what can I say) was fixed immediately I was made aware of the problem. I should have noted that the report was sponsored by other companies (as is the standard operating model for Osterman Research reports) and am glad that Michael has listed them in his comment (I have inserted the names into the article).

I consider his reference to my membership of the advisory board of ENow Software to be a little over the top. I don't hide this fact (ever), as a glance at my LinkedIn profile demonstrates, and I never allow this fact to influence what I write as I understand that it would undermine what I report. I never commented on the functionality offered by Knowledge Vault, so it doesn't really make any difference here.

The real point is that the report overlooks many of the compliance features in Office 365 and that, in my mind, is justification for the "slamming" title. I don't agree that "most" Office 365 customers will need third-party products to meet their compliance needs. That hasn't been my experience in working with some very large organizations.

Please log in or register to post comments.