Is there life after open data publication in Russia?
User testing of existing applications and services
Open data have no economic effect without a market of applications based on them. Having tested the existing services based on open data, the Infometer has learned why leading regions succeed in this field and what motivates developers to use open data.
Fast facts
Apps rating
Reset
Regions rating
Analysis
Methodology

Open data usability depends on quality of datasets, of portals, and of organizational management

6 leading regions using 5 or more applications:
  • - publish significant numbers of datasets (totally, 2,840 datasets);
  • - observe requirements to open data publication soundly (at 80% at average);
  • - 4 regions enable data downloading via API;
  • - 5 regions have project centers;
  • - all 6 regions implement the Open Region project;
  • - all 6 regions publish data at regional portals.
In addition to publication of many datasets, practical results are achieved by regions actively promoting their open data and informing developers on them. Leading regions are: Moscow city, St.-Petersburg city, Tula Region, Vologda Region, Perm Territory, Krasnodar Territory.
Formal publication of many datasets does not help to reach positive results, e.g.:
  • 54 datasets are published by Kabardino-Balkarian Republic
  • 50, by Yamal-Nenets Autonomous Area
  • 34, by Amur Region
  • 28, by Republic of Adygeya
  • 17, by Nenets Autonomous Area
However, no one of these regions published any information on working with open data, and quality of their technological sites leaves much to be desired. As a result, there are no applications developed in these regions.
In the beginning of our work with open data, we focused on their number and volumes and published more than a thousand datasets. Now we pay attention to dataset quality that is also crucial from the point of view of developers' demand.
Maria Dobrokhotova
Section Head
Department for Informational Society Development
Perm Territory
There are organizational and also technological complications. and есть и организационные, и технологические. In organizational aspect, data can appear to be irrelevant or can disappear from a portal; licensing issues are usually rather vague, too. In technical aspect, data can be inaccurate or there can be difficulties with automatic acquisition and processing of the data.
Evgeny Nikulin
NextGIS

Regional Open Data Portals provide best results

The Methodical Guidelines allow government bodies to choose a site for open data publication. It may be a government body's own website, a regional portal, or the federal portal data.gov.ru. 75 regions have already made their choice and often use more than one site. We have studied performance indices for each way of open data publication:

Way Number of regions Number of datasets Average openness level Number of applications
Regional open data portal 26 6015 61% 114
Open data section of regional government website 34 817 36% 24
Open data sections on government bodies' websites 23 845 23% 26
Federal portal 2 2 24% 1

The table shows that both quantitative and qualitative indices of open data publication are significantly higher for regional open data portals. There are also rather more applications in regions using open data portals of theirs.

In Tula Region and Ulyanovsk Region, open data portals observe all requirements for open data publication for 100%. They publish correspondingly 769 and 465 datasets and use 11 and 3 applications. However, mere usage of a regional portal is not a success warranty. For instance, open data portals of Republic of Altai and Amur Region do not meet requirements to open data publication quality and portal design. and there are no regional applications in these two regions.

A section for open data publication at a regional government's website also can be efficient. In Tomsk Region, requirements for open data publication are observed for 100%, and there are 167 open datasets and 3 applications available.

In the Methodical Guidelines, requirements to open data portals are wider that to sections at regional governments' websites. To launch a portal is a rather more complicated task than to update an existing website. At the same time, if open data publication infrastructure is soundly built and interaction with developers is active, interactive services at a special portal will increase probability that new applications will be developed.

Independent vs. Governmental Initiative

Within the study, we sought for applications developed by governmental order using the following criteria:
  • - developer status (entities subordinate to government bodies);
  • - direct mentions of governmental order for development.

To get necessary information, we studied open sources and directly contacted government bodies' and developers' representatives.

he study revealed that 63 of 120 applications are created by independent developers, therefore showing demand for open datasets by startups and enthusiasts.

Average user testing scores for applications of both groups (counting also evaluation scores in markets) appear to be practically equal, amounting to 3.7.

Initiative Evaluated higher than 4 points 5 points
Governmental 33 1
Independent 31 4

In our opinion, it is not reasonable to order application development. First, if a dataset has not been used by developers themselves, it may be of no interest for the wide public. Second, application should be as large-scale as possible, touching not only a specific region: it is desirable for an open data application to cover the whole country, and theoretically, the world.
Maria Dobrokhotova
Section Head
Department for Informational Society Development
Perm Territory

Application Scopes

Most applications belong to the "Government" category due to wide coverage of this scope and large number of relevant datasets.

Category Number Average testing
evaluation score
Independent Governmental initiative Total
Government 9 25 34 3,6
Transport 19 3 22 3,9
Recreation and Entertainment 10 3 13 3,8
Health 5 6 11 3,1
Tourism 6 4 10 3,7
Economy 4 6 10 3,6
Culture 0 5 5 4,0
Education 4 1 5 4,1
Construction 2 3 5 3,8
Safety 2 0 2 3,2
Cartography 1 1 2 3,4
Environment («EcoMonitoring») 1 0 1 4,6

These scopes are used due to easy visualization of relevant datasets and possibility for geographic identification of objects listed in the datasets.

By open data application tasks, the following types can be highlighted:
  • Culture/entertainment navigator/guide
  • Appointment of visits to doctors
  • Maps of multifunctional centers, schools, hospitals and other organizations
  • Checking fines imposed
  • Transport route maps
  • "Mobile Citizen" mapping and discussing problems

Of course application types listed above are conventional but they are based on datasets similar by content.

On the one hand, similar applications created by independent developers in the same region (for instance, the Unified Medical Informational and Analytical System (EMIAS) of the Moscow city and EMIAS. INFO, or Mobile Touristic Portal, Touristic Moscow, and Moscow Guide) enable competition and provide possibility of choice for users.

On the other hand, from the point of view of expenditure optimization and interface unification, inter-regional applications created from typical datasets and funded from state budgets seem to be the most feasible solution.

Inter-regional open data applications are really needed. There is a problem of structure difference but we have not met it yet, having not developed applications of that kind. Of course, such difference can make complications when creating an application for several regions.
Evgeny Nikulin
NextGIS
I believe that governmental order for an application can be efficient, for instance, when a typical application is formed at federal level and then adapted at regional level
Alexander Lamonov
Specialist, Open Region department
Tula Region

Most Productive Developers

The study has revealed that of 86 developers, there are 74 each of which has created a single open data application. At the same time, more than a quarter of applications analyzed (47 of 120) are created by the following 12 developers:

Developer Number of Applications
OAO Elektronnaya Moskva 10
m0rg0t (Anton S. Lenev) 7
GAU TO "CIT" (Tula) 6
SPb GUP "SPB IAC" (St.-Petersburg) 6
Krasnodar City Municipal District Administration 3
Yandex 3
NextGIS 2
Squeaky Oak 2
BU VO "CIT" 2
NOU VPO "RSEI" 2
MKU Elektronnyi Krasnodar 2
OOO "Studiya Chetverty Kit" (Fourth Whale Studio) 2
Open data published by executive government bodies allow developers to realize their potential, and the public to get information on government bodies' activities in convenient formats. Useful mobile applications also improve people's life quality by means of fast access to information and services needed.
An official of the St.-Petersburg Committee on IT and Communications

Introduction

While the market of mobile applications based on open governmental data develops actively, it is necessary to evaluate quality of products and services already working.

Analysis of existing mobile applications helps to reveal basic trends of the market development, and to define promising directions and guidelines for regional government bodies on building interaction with target audiences.

When planning activity strategies, government bodies should stimulate factors motivating developers to use governmental datasets as a basis for applications. This will improve performance indices for these governmental bodies' activities.

The Infometer experts have performed the very first analysis of the market of mobile applications based on open governmental data.

Study Objective

  1. To evaluate quality of existing open data software products.
  2. To reveal and to present best examples of open data applications.
  3. To define factors stimulating development of independent projects that is the objective for open data introduction in regions.
  4. To reveal system problems for development of open data projects in regions.

Study Object

  1. Applications linked to by the Russian Federation open data portal and regional open data portals
  2. Applications and services using open data in their work

Study Activities

  1. General information on each application (scope, platform, region, popularity, developer, presence of state order features) was collected and analyzed.
  2. Experts evaluated applications (for various platforms and OS versions) by the following criteria and rates:
    1. Performance – evaluates result of application launch and correctness of its interface display at user's device: 0 if the application can't be installed at user's device, or constantly (not less than 5 times) displays a launch error, or does not allow to use its interface for working with its functionality; 0.5 if the application is installed but some of its functions are not available due to incorrect interface display or interface is a module stub not implementing functionality; 1 if the application is installed at a user's device and allows to use its functionality.
    2. Accuracy – evaluates application working errors leading or not leading to its shutdown: 0.4 if there are errors shutting down the application or leading to full inability to use its functionality; 0.7 if there are errors hindering usage of its functionality or causing incorrect results; 1 if the application works without errors.
    3. Description accuracy – evaluates compliance of application description with its real functionality: 0.3 if description matches real functionality less than for 50%; 0.6 if some of functions described are not implemented or do not work, or part of significant information is absent (e.g., object description cards are empty); 1 if description fully matches functionality of the application.
    1. Speed – evaluates speed of application work (functionality execution). 0.6 if the application works very slowly or there are freezes; 0.85 if there are delays but not freezes; 1 if the application works at comfortable speed.
    2. Usability – evaluates usage user-friendliness, interface response time, and functional design of the application: 0.7 if the application is quite unfriendly for users, it is difficult to understand its functionality or structure; 0.9 if the application functions are clear but some of them are less user-friendly than analogs; 1 if the application is intuitively understandable and its usage does not cause questions on tasks and solution mechanisms.
    3. Installation easiness – evaluates easiness of application search (in Google Play market, Appstore, or Marketplace): 0.1 if the application is absent in the markets and an installer is needed; 0.8 if the application is there in a market but cannot be found in the first ten when searching by its specific name or there are other applications with identical or similar names that can be mixed up; 1 if the application is found in a market top ten (searched by name) and can be clearly identified by name.
  1. Expert evaluation scores were grouped in order to calculate average score for each application.
  2. Final rating was classified by application scopes (categories) to reveal leaders for each scope.
  3. Resulting evaluation score is calculated as simple average between the expert score (modified for five-point scale) and the score from the market. If the application is available for more than one platform resulting score is averaged over all available platforms.

Study Methods

  1. Quantitative content analysis was used to collect information.
  2. To form expert positions and advice, semi-formalized interviews were held with officials of government bodies in regions where successful applications work. Developers of successful applications based on governmental open data were also interviewed.
  3. To evaluate applications and their usability, testing was used.
  4. Correlation analysis was used to reveal factors positively impacting development of applications based on open governmental data.
Problems and solutions
Problem:
Irrelevant Applications at the Federal Portal
There are more than 200 open data projects placed at the data.gov.ru portal. Having studied them, we found many applications not related to open data of the Russian federation either by essential or by formal criteria. Examples: a guide for Minsk city (Belarus), a collection of best cooking recipes of the world, and "Hindu Calendar".
Solution:
We propose to add to the project registration form at the Data.gov.ru portal a field informing what datasets are used by the project/application. To enable compliance of the federal open data portal content to the portal goals, current list of projects should be moderated in order to delete projects not related to federal open data.
Problem:
Lack of transparency in the process of open data project formation
Since there is no open information on initiatives for creation of open data applications (besides scarce news stories), sources that invest resources in open data are often unknown, and one cannot learn whether an application was developed with budget funding or upon someone's independent initiative.
Solution:
We propose to mention developer status (state order, hackathon, or direct dataset downloading to create an independent application) in the application description at an open data portal. This will help to study development directions for the market of open data applications, to reveal specific factors and complications for independent developers' work with specific data scopes, and to make conclusions whether it is feasible to develop user services at public expense.
Problem:
Open data as "additional burden"
Regional officials often consider open data as some burden beyond their job duties – due to lack of sound organizational infrastructure for open data publication. This leads to lack of proactive information placement and to formal attitude to dataset publication, reducing chances for constructive contact with software developers. Moreover, massive "manual" publication of open data leads to risk of developmental dead-end for this direction: it becomes difficult to fulfill requests for open data if there are not enough labor resources.
Solution:
Depending on the model chosen, regional government bodies should provide administrative regulation of their status as responsible publishers of open data. If data are placed by government bodies' personnel directly, the employees should be regularly trained, have enough work time separated from their main duties, and get material award. If data publication is supervised by project center staff, this will need publishers' support, assistance in sound and timely dataset publication, and consideration of possibilities for automated data upload via API tools. If there are expert coordinating bodies, regular work for revealing most significant and demanded datasets as well as structure unification will be needed.
About research
Regions rating Apps rating