Click here to subscribe to the Helix Nebula & PICSE Newsletter

Case study - Approaching cloud through a pilot test

Who we are

We are an independent intergovernmental organization providing a weather forecasting service and weather data catalogue to 34 countries (member states) and commercial clients. We also provide computing resources to the meteorological user community. We have a supercomputer facility (and associated data archive) which is one of the largest of its type in Europe. Our procurement process is designed to achieve both quality and value for money in the procurement of goods and services. We use an open tender process and as an international organisation, we are governed by our convention and its associated financial regulations.

Where the contract involves an amount exceeding €210,000, invitations to tender have to be made public in the countries that the centre serves, unless the nature of the contract or special circumstances justify an exception. In the case of goods or services which, because of their nature or amount, cannot be the subject of a public invitation to tender, the Director General of our organization may ask a restricted number of specialised contractors from as wide a selection of countries as possible to submit offers. Contracts shall be concluded following adjudication or requests for tenders.

Contracts may also be concluded directly:

» where the contract involves an amount not exceeding €210,000 for supplies, services and for scientific and technical equipment and construction work;

» where goods and services are so urgently required that it is not possible for either of the tendering procedures;

» where tendering or invitations to tender have produced no results or where the prices quoted are not acceptable and where a repeat invitation to tender is not expected to produce a better result;

» where for technical, practical or legal reasons the services can only be rendered by a specific contractor or supplier;

» where supplies, services and scientific and technical equipment and construction work are purchased through frame work agreements that have been established by public procurement agencies which select suppliers via a tendering process which complies with EU procurement rules.

» Where the expected value of services for a single indivisible unit does not exceed €4,200 a contract may be placed against bill or invoice only, under the conditions laid down by the Finance Committee.

 

Why the cloud?

We have a large supercomputing facility hosting at present two Cray XC-30 supercomputers used mainly for running 24x7 operations to generate medium-range weather forecasts for our member states, for research, and to provide computing resources to the meteorological user community. Our global weather model and related software produces millions of products on a daily basis. The products generated are highly customized to the individual requirements of our users and are transferred to their home sites either via private networks or via the Internet. The number of products generated is not only high in number but also some of the products have very large volumes in the order of terabytes of data. The transfer of these very large data-sets to our customer’s home sites is not always feasible especially when having strict time delivery constraints for 24x7 operations. A solution is to run the user processes on computing resources close to where the data-sets are stored. To implement this solution we have two options.

Either we build a private cloud in our data centre and offer the resources to external users from other organizations, or we establish a partnership with a public cloud provider connected to our organisation with a high bandwidth network link such that the data is stored directly on the cloud. External users from other organizations could then use cloud computing resources from the same provider to post-process the data and generate their own output and products. As a second use case, we are also a research centre that maintains one of the world’s largest archives of meteorological data.

This multi-petabyte archive is available for research purposes and it is envisaged that a subset of the most important data-sets with a total aggregated volume between two and three petabytes could be made accessible to a much larger scientific community using cloud technologies. These data-sets are useful not only for meteorological and climate research but also in many other scientific fields, as an example, models for the spread of malaria in Africa benefit greatly from having access to this type of meteorological historical data. As yet we have not procured any cloud computing services.

Our main perceived barriers are a lack of information on security assurance and an increased dependence on cloud service providers and vendor lock in.

» In 2014 we joined an international initiative setting up a federated cloud marketplace for science to extend our cloud computing competences and to speed up the process of procuring cloud services. We currently plan to do a pilot test for establishing whether the performance offered by different cloud providers is sufficient for our use cases.