LibreOffice: SharePoint integration. A year of progress

The slides from my talk at the LibreOffice Conference in Rome (2017-10-13) (hybrid PDF; notes inside embedded ODP)

Advertisements

Windows Unicode API usage in LibreOffice

Windows still provides two sets of many of its Win32 API functions taking or returning strings: a legacy “Ansi” (functions named like fooA) and Unicode (named like fooW; available since Windows NT, and in Windows 95 with Layer for Unicode – and thus on any Windows OS supported by LibreOffice).

The “Ansi” functions take 8-bit strings in current codepage (single- or multibyte). The repertoire of characters representable in those strings is, naturally, limited to that codepage (that is either setup in system’s Language for non-Unicode programs, or explicitly set by running application). Unfortunately, unlike in other contemporary OSes, Windows doesn’t allow setting its locale to use UTF-8. If a string arrives to such a function that contains characters outside of that set, the string content will be altered, and functions’ behaviour might change unexpectedly.

“W” versions of those functions take UCS-2 strings, that are able to represent most of Unicode range (I am unsure if those strings are actually UTF-16, and so are able to represent the full Unicode repertoire, but anyway, even UCS-2 is much wider than most of single- or multi-byte codepages).

In last two weeks, we have replaced many places (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D) in LibreOffice codebase where legacy “A”-functions were still used, with explicit calls of their “W”-counterparts, removing redundant conversions of strings from LibreOffice internal UTF-16 string representation to 8-bit strings and back. One of most significant effects might be on file-management functions, where such conversions could alter paths/names containing Unicode characters not representable in currently selected 8-bit codepage, and lead to failed file operations. One example of such problems is tdf#103525.

The changes are included into master towards 6.0.

App compatibility manifest

Last week, we’ve added application compatibility manifests to LibreOffice libraries and executables. This is intended to allow the program to run in current OS context, instead of switching to a previous (namely, Windows Vista in case of absent manifest section) OS compatibility context. The effects of this are listed in a MS KB (this is alternative link just in case).

Currently we declare compatibility with Windows 7 through Windows 10. This change is integrated into current master towards LibreOffice 6.0.

LibreOffice MUFFIN

Recently, TDF has published its new MUFFIN concept. This publication has provoked a mixed response, with many negative reactions.

When we deal with UI, any possible change brings controversy, and it’s always very aggressive when someone finds that a function they are used to use some way had changed its position. OTOH, there is always a substantial part of community that aggressively promotes “modernization” of current UI.

We all remember how different projects made their UI changes. We remember cases when programs with huge user base made radical changes without an option to keep old UI for those who prefer it. And it forces many of us to resist any sign of possible UI changes in software we use.

Many believe that “design” and “marketing” are swearwords that are used when a company tries to make “purchasers” believe they need something they actually don’t. So, any use of these words in a press release makes yet another chunk of user base to get angry.

There are people that don’t care of UI changes, but have strong opinions that there are much higher-priority tasks to do, like improving stability, compatibility and so on. And when they hear of an effort in an area they regard as unimportant, they get upset. Some of them even have donated to TDF with hopes that donations would go to those goals, and now “see” that TDF wastes their money to pay some designers/marketologists instead of doing Right Things (TM).

Many see the news about UI and immediately realize that there’s indeed something new in UI that is announced. They don’t care to read (let alone comprehend), and so skip any text for screenshots/mockups, and get their views on the matter based on their perception of the graphics. And many of them are angry because they see something that disappoints their expectations of e.g. “ribbon-like” interface. Or they suppose that TDF fools them trying to make believe that there’s something new in the UI options (besides NotebookBar) when actually all that is presented is already possible for long time. “Ah, they try to show to you something old, and pretend that it recently have made a great work to create something new! Smart move!”

Some think that new UI elements are not ready for use and need to be polished in a different branch before inclusion into a LO release.

Many believe that UI changes must begin with UI implementation being rebased to some other library like QT, and that any effort that doesn’t do this is waste of time.

People seem to misunderstand what MUFFIN is.

Thankfully, there are some people in LO community who feel interested in design issues. They form our Design team. Those people are not necessarily developers, and if they are, they have their own interests and priorities, independent of TDF’s or some specific user’s. That’s a great feature of open source community, where each one is able to find an area of interests to apply their effort. And it doesn’t matter if you want LO to get a bug-fix/compatibility feature, and it doesn’t matter if I do some work on those fixes/features: those other people don’t need and aren’t forced to wait for me or help me; they volunteer to do their own contribution in their area in their spare time, and announce their results when they feel appropriate. So anyone who argues that some other things should have been done instead of this work, is misguided. If we would prevent the Design team from doing this, we wouldn’t get more man-power dedicated to your preferred tasks; rather, we would just had lost the man-power (and more important, we would loose those contributors, because there wouldn’t be any reason for them to stay with the project).

The Design team clearly sees the challenges that LO faces with regards to UI changes. On one hand, many want some “refresh”; on the other, many deny the very possibility of it and are afraid of it. There are issues in UI not following different OSes/DMs conventions (making LO to feel not native to these OSes/DMs). And with MUFFIN, the team had IMO made a very smart move.

MUFFIN isn’t a new interface itself. It is even not an interface concept in a usual sense of the term! What it actually is is a public statement, a promise made by LO to its users; and this statement and promise is that: LibreOffice believes that it should be pleasant and easy to use by anyone; and it will never take steps (like other softwares sometimes do) that will sacrifice one group of users’ preferences just to please another – instead, it promises to make its UI as much configurable and modular as it required to allow you, me and anyone to build their own UI of choice. The message is that LO will e.g. continue to provide toolbars: this is not a “first step after which some evil designer will inevitably drop support for good old UI”! And in the same time, LO will not ignore those numerous current and potential users that want another interface: their interests aren’t of lower importance.

If someone still believes in a conspiracy theory that will eventually deprive you from your dear toolbars, remember that LO is being developed by those developers who prefer toolbars themselves, and there’s no ultimate authority to make decision to drop it regardless of those developers’ PoV.

So, MUFFIN is not a NotebookBar or a Single Toolbar. It is a message to users, and a guideline to developers who make UI-related changes. MUFFIN is not the four presented UI options; rather, they are just demonstration of commitment to make it easy to any group to customize.

The four presented UI options are not meant to be necessarily something new. Of course, you can use standard toolbars, or customize them to let only one custom toolbar visible, or use sidebar with any combination of toolbars: the UI components and their combinations aren’t new. The UI components aren’t perfect, and the concept doesn’t claim they are. The concept is orthogonal to any framework used to implement UI components. So, those who are not pleased because the UI isn’t new or isn’t perfect aren’t right when they blame MUFFIN for that. MUFFIN is a promise – so it’s not something to be kept in a separate development branch; and it’s not something that needs screenshots. I’d say that providing screenshots in the press release actually distracted and misguided many readers from the real message.

TDF didn’t spend money to make this happen. It used a research that is publicly available. It doesn’t spend your money to pay some greedy designers: it uses its community’s great power. So no need to get angry that your money are wasted. (As a side note, I’d suggest people to get familiar with TDF role and understand that it doesn’t develop LO itself; instead, it provides home and shelter to community, and promotes the software. Understanding this can help you avoid disappointment when you find out that your donations go to other needs that you imagined.)

I believe that MUFFIN is a great message if you make an effort to hear it. Please do. And if you believe that there’s something to improve, please contribute and become part of community!

My first hack-(rest-of-the)-week at Collabora

I must say that I’ve been attentive reader of TDF feeds almost from the very beginning of its history, when I couldn’t think to develop for LibreOffice. And reading posts of developers, where they described what they had done at their hack-weeks at SUSE, made me really envy them…

So here is my first hack-(rest-of-the)-week at Collabora Productivity! That’s cool!

And when it happened, it turned out that I need to finish a work that I started this week. What a pity! Well, the task itself was really interesting, and I’m glad that it is in a close-to-be-finished form in gerrit… but still. So, only hack-weekend 🙂

As I didn’t have that much time, I couldn’t afford doing something big and lengthy. So I turned to my favorite topic: SAXParseException.

Here I need to tell you what is it. Until LibreOffice 5.0.4, many errors in XML that LibreOffice reads were simply silently ignored. Note that both LO’s native format, ODF, as well as its most popular competitor, OOXML, are XML formats. It means that the file could be read incorrectly, some data could be dropped, some bugs could go unnoticed, but file opened as if everything was OK. Data loss could go unnoticed.

In January 2015, two days before my daughter’s 3rd birthday , a named hero Michael Stahl took required steps and committed a patch that put that madness to end. Now every error in the XML data that LO reads is reported to user as famous SAXParseException. This annoys users, but it yells loudly “Fix me!”, and it has already enabled us to find and fix quite a number of errors that previously went unnoticed.

So, I must say that I like to hack on these bugs. But since I started to work full-time as developer, I had not that much time for them. And now I took bug 99227 for the rest of my hack-week.

It turned out to be LibreOffice OOXML export filter fault, that caused loss of drawings in footnotes/endnotes. It also created an invalid XML, which was caught on following import, telling that it found “Extra content at the end of the document”.

I’m happy to say that the patch is on its way (it’s on review, and after some rounds of fixing my evident mistakes pointed out by merciless reviewers, it will be pushed).

It may seem odd that I feel so enthusiastic about changing hacking on LO by … hacking on LO. 🙂 But actually, my daily work is so great, I really love it! and being able to choose tasks myself is the only part that I miss sometimes… and that I get now.

Thanks to Collabora Productivity, to all fellow hackers, and thank you for reading this 🙂

Integrating LibreOffice with SharePoint – part 2

In my previous post, I discuss LibreOffice‘s CMIS integration with SharePoint. This post, as promised, is dedicated to WebDAV method of accessing SharePoint servers from LibreOffice.

Basic WebDAV support is already present in LibreOffice, which inherits it from OpenOffice.org. You may put SharePoint URL into OS’s File Open dialog, and browse and open documents as if they were on local file system. But to claim SharePoint WebDAV support, LibreOffice should provide ease of use no less than its competitors.

Existing users of MS Office are used to access the documents on SharePoint server using web browser integration. This allows to navigate to a document and open it in associated office program simply clicking on its link on web page, or using the page’s options to check-out document, open it for reading, etc. A document that is open this way isn’t downloaded by browser to be opened by application locally, but is open by office application via WebDAV, thus presenting easy way to work with document, including saving your work to proper place automatically. This integration is now available for all modern browsers (the level of feature support vary, though).

sp-webdav1

There are two mechanisms used to enable such integration. The first (and older) one is using ActiveX control/browser plugin. This method works for IE (classic mode) using OpenDocuments ActiveX control (and also used to work for Firefox and Chrome using NPAPI plugin, but since NPAPI was deprecated, it doesn’t work there anymore). In essence, a JavaScript click handler on the page instantiates the control, and then uses its methods to open the document with corresponding application in required mode. With all implied security considerations.

The second (newer) method is using custom URI schemes. This requires that corresponding URI handler be registered in OS/browser, and the handler application be able to recognize these URIs and extract actual WebDAV URIs required to access the document on server from them.

Both methods were lately addressed in LibreOffice development by Collabora Productivity. To implement first method, a custom ActiveX control was coded, and is now awaiting review on gerrit.

To utilize SharePoint custom URIs, LibreOffice was taught to recognize MS Office’s existing custom URIs. This is required to allow using LibreOffice to open MS Office’s documents from SharePoint on those computers that don’t have MS Office installed. But that’s not enough; we should be able to treat ODF independently, e.g. to allow for configurations where both LibreOffice and MS Office co-exist on a workstation, and must handle their respective native formats. To achieve that, we introduced our own custom URI scheme, vnd.libreoffice.command. Now LibreOffice can accept any of these URIs, and successfully open the document in proper mode.

But for LibreOffice to open them, these URIs must be passed to it from the SharePoint web page. We added registration of our own custom URI handler into installer for Windows OS, and also we now register LibreOffice as MS Office URIs handler if user chooses to register MS formats to open with LibreOffice.

If you choose to use the second (custom URI) integration method, and you don’t need MS Office as handler for DOCXs, PPTXs etc., then you don’t need to configure your SharePoint server for that to work. All that you need is to install recent enough LibreOffice (v.5.3+), and configure it at installation time to open MS Office files.

But if you choose either first method (ActiveX), or second method with coexisting LibreOffice/MS Office workstations, then you will need to configure SharePoint server for that. In both cases, you will need to edit DOCICON.xml file, that  may be found in server’s %ProgramFiles%\Common Files\Microsoft Shared\Web Server Extensions\XX\TEMPLATE\XML. This is an XML file, and we should edit sub-elements of ByExtension element. Such sub-element has the following syntax:

<Mapping Key="odt" Value="icodt.png" EditText="LibreOffice" OpenControl="LOSPSupport.OpenDocuments" OpenApp="vnd.libreoffice.command"/>
  • Key = file extension
  • Value = icon file
  • EditText = Text on “Edit with application” UI button – required for edit option. Optional.
  • OpenControl = ActiveX OpenDocuments control name (currently IE 32-bit only). Optional.
  • OpenApp = custom URI scheme to handle this. Independent from and appears to have priority over OpenControl. Available in 2013 SP1. Optional.

You should use either OpenControl or OpenApp, depending on which integration method you have chosen. Do it for those extensions that you want to be handled by LibreOffice. Of course, you should backup the file prior to modifying it.

After you have made all necessary modifications, to activate the settings, you need to restart IIS. To do that, you may execute command iisreset from the server’s administrator command line.

That’s it, if you did everything right, LibreOffice should now seamlessly integrate with SharePoint website, and be automatically launched when you choose to open files from there, without downloading the file to local machine first.

This is the second of two posts that discuss the current state of SharePoint integration support in LibreOffice. Thank you for reading, and enjoy the best free office suite ever!

Integrating LibreOffice with SharePoint – part 1

LibreOffice is a productivity suite that is suited well for using in corporate environments with centralized document repositories, such as SharePoint. Of course, to use it there, program should offer required functionality in convenient fashion. So what is the current state of LibreOffice’s support for integration with SharePoint?

LibreOffice has two ways to communicate with SharePoint servers:

Lately, Collabora Productivity has made a number of improvements made to LibreOffice, and fixes for some existing problems, that made configuration and usage of both types of SharePoint integration more robust and easy.

In this and the next post, I am discussing current status of support and the methods to setup (and limited usage examples) for both ways. This post is dedicated to CMIS, and the next one will be devoted to WebDAV.

CMIS integration

To use CMIS integration with SharePoint, you need to configure your SharePoint Server to offer the CMIS service, and add new CMIS service to LibreOffice.

Configuring SharePoint Server for CMIS

The following configuration steps are shown on a SharePoint Server 2013 instance taken as example.

First, basic authentication should be enabled in SharePoint Central Administration.

Go to Security, choose Specify authentication providers in General Security group:

sp-auth1

Choose required Web Application and click Zone that you want to configure, and enable Basic authentication:

sp-auth2

Of course, HTTPS should be enabled for proper security.

Second, configure your site settings.

Login to your site, go to Settings → Site settings and choose Manage site features in Site Actions group:

sp-site1

There you need to activate Content Management Interoperability Services (CMIS) Producer:

sp-site2

That’s it. Now it’s time to setup LibreOffice to access SharePoint via CMIS.

Configuring LibreOffice for SharePoint CMIS access

You will use LibreOffice’s Remote Files feature, that is accessible from Start Center, as well as from main menu:

sp-lo1

Click Add service and choose SharePoint 2013 as Type, replace with the address of your SharePoint site, supply SharePoint username and password, then click Refresh button to the right of Repository chooser, after which LibreOffice will connect SharePoint Server and populate the chooser with existing repositories:

sp-lo2

To make it easier to use the connection, you may put the directory you want to start from into the Root field, e.g. “/Shared Documents”. This will not prevent you to use other directories later.

Now click OK and you will be prompted for password again. The username and password you entered at the previous step were used to get repositories list, so provide your credentials again. If you choose to save password at this stage, in case you didn’t yet saved passwords for web connections previously, LibreOffice will ask you to set Master Password that is used once per LibreOffice session:

sp-lo3

Entering this password first time you access any web connection with stored password, you will not be asked anymore for passwords until program closes. This password is separate from the site’s password, and should differ from it.

Now you are able to navigate your SharePoint server documents, rename or delete them, open, edit and save back (or create new) each time you use the Remote Files feature or accessing last used files from the server. Here is how the Remote Files dialog looks like for a configured connection:

sp-lo4

LibreOffice will show you a warning if you have not checked out the document yet, and allows you to do that immediately:

sp-lo5

The Check-Out, Cancel Check-Out and Check-In functionality is also accessible from LibreOffice’s File menu. Checking in will bring the following window, where you may fill the required information for new version of the file:

sp-lo6

To see the file’s CMIS properties, you may use Properties window, also accessible from File menu. It has CMIS Properties tab, that allows to see and change those properties.

That’s all I wanted to tell about LibreOffice integration with SharePoint using CMIS. The next post will cover WebDAW method to do (almost) the same.