We have prepared a video to introduce to the capabilities of XtreemOS:
Take a look at it on the homepage of our project website
We are pleased to announce that we have another new demo video on-line:
Source code and packages are available for download on http://www.xtreemfs.org.
There is no change in the database or OSD storage, so an update from 1.2 should work out of the box. The usual advice: backup important data before upgrading.
XtreemFS 1.2 ("Luscious Lebkuchen") is now available at http://www.xtreemfs.org.
- asynchronous MRC database backups (database dumps are created in the background)
- new GRID SSL mode: SSL is used for authentication only, data is transferred via plain TCP for a better performance
- Vivaldi-based replica selection and placement
- MRC and OSD wait and retry for a configurable time span if they cannot reach the DIR on startup
- faster file copying of files with 'cp' (by providing correct values for 'st_blksize' with 'getattr')
- additional binaries with standard Unix names (mount.xtreemfs, mkfs.xtreemfs, lsfs.xtreemfs, rmfs.xtreemfs)
- experimental Hadoop file system driver for XtreemFS
- web-based admin console
Early in 2004 the Ministry of Education and Science (Spanish Government), Generalitat de Catalunya (local Catalan Government) and Technical University of Catalonia (UPC) took the initiative of creating a National Supercomputing Center in Barcelona. BSC-CNS (Barcelona Supercomputing Center – Centro Nacional de Supercomputación) is the National Supercomputing Facility in Spain and was officially constituted in April 2005. BSC-CNS manages MareNostrum, one of the most powerful supercomputers in Europe, located at the Torre Girona chapel. The mission of BSC-CNS is to investigate, develop and manage information technology in order to facilitate scientific progress. With this aim, special dedication has been taken to areas such as Computational Sciences, Life Sciences and Earth Sciences.
All these activities are complementary to each other and very tightly related. In this way, a multidisciplinary loop is set up: our exposure to industrial and non-computer science academic practices improves our understanding of the needs and helps us focusing our basic research towards improving those practices. The result is very positive both for our research work as well as for improving the way we service our society.
BSC contribution to XtreemOS
- Application execution management
The main task of BSC in XtreemOS is to lead the research and development of the Application Execution Management (AEM) module. This module is in charge of setting the whole environment to run jobs in the grid (build reservations, allocate resources, schedule jobs, launch processes, etc.) and to control and monitor them (maintaining the state, allow thread-level monitoring, etc.).
Specifically BSC is highly involved in the scalable job management avoiding any kind of bottleneck as well as on the monitoring that allows users to monitor a job just as if it were a Linux process (you can even monitor the change of thread states).
In the data management module, the storage system research group is in charge of the algorithm to locate the best replica of a file as well as to decide when and where replicas should be created and/or removed. This mechanism is base on an improved version of the Vivaldi algorithm that places all XtreemOS nodes into a 2D space based on communication latencies.
Regarding mobile devices, BSC is cooperating with TiD to build the embedded version of XtreemOS to be run on mobile devices. Mainly, BSC is working on the AEM and XtreemFS modules of XtreemOS-G for these devices.
Finally, BSC is porting two applications trying to take full advantage of the benefits of this new system. These applications are specweb 2005 and hmmpfam which runs on top of COMPSs a workflow manager that is also ported to its full efficiency to XtreemOS by BSC.
- heise (english) http://www.h-online.com/open/Distributed-file-system-for-the-internet--/news/114009
- heise (german) http://www.heise.de/open/Verteiltes-Dateisystem-fuers-Internet--/news/meldung/143550
- golem (german) http://www.golem.de/0908/69078.html
- Linux Magazin (german) http://www.linux-magazin.de/NEWS/XtreemFS-1.0-ist-fertig
- 01net.com (french) http://pro.01net.com/editorial/505038/avec-xtreemfs-leurope-a-son-systeme-de-fichiers-en-ligne/
More about XtreemFS: here
XtreemFS 1.0 is out. XtreemFS now supports read-only replication with failover, data center replica maps, parallel reads and writes, and a native Windows client, among other new features.
Download it at xtreemfs.org.
We demonstrate the XtreemFS replication and failover with an unmodified MPlayer. We play a regular movie file stored on two XtreemFS OSDs.
The Linux Symposium was held at Montreal, Canada, from July 13th to July 17th.
John Mehnert-Spahn (UDUS) gave a technical presentation of 45 minutes on his work "Incremental Checkpointing for Grids" on July 15th. John explained the motivation behind incremental checkpoint and its implementation in Linux SSI as well as its integration in XtreemOS. There was much interest in the implementation of incremental checkpoint.
On 17th July Surbhi Chitre (INRIA) held a BOFs session for XtreemOS. She explained why XtreemOS is attractive and how it is a complete solution for grids. She also explained her work of integrating OpenVZ in XtreemOS, why it was chosen and the challenges in doing so. In her talk, she also gave a brief introduction to XtreemFS features. This was followed by a presentation by Louis Rilling of Kerlabs of Kerrighed Clusters which is supported by XtreemOS.
June 24-27, 2009
LinuxTag is Europe's leading event for Linux and free software. The four-day event has been taking place annually on the Berlin Exhibition Grounds since 2007. A total of 10,780 visitors from around the world attended LinuxTag 2009 in Berlin.
We presented XtreemOS and XtreemFS with a community booth at LinuxTag 2009 in Berlin.
During the four days of the exhibition we had a chat with about 60 visitors. For us it was very interesting to see what kind of problem the people had in mind to solve with XtreemFS. This ranged from using XtreemFS to replace VPN+Samba to video recording, editing and replication for streaming. Of course, there was quite some interest in using XtreemFS as a kind of cloud storage.
More than 10,000 visitors make LinuxTag the largest industry event in Europe - Wider-ranging conference programme and greater international participation (135 exhibiting companies and projects and around 300 speakers).
The next LinuxTag will take place from 9 to 12 June 2010 in Berlin!
The Italian National Research Council (CNR) is a public organization whose purpose is to carry out, promote, and improve research activities in the main sectors of knowledge, spread their results and help their transfer to the industry and their application, to further the scientific, technological, economic and social development of Italy. To this purpose, a network of CNR institutes is distributed all over Italy. To promote international cooperation and contribute to the development of key technologies, CNR participates in agreements, contracts and research projects with the European Union and with the other international organizations; among them the CNR membership of ERCIM and of W3C. Two of the CNR institutes, ISTI and ICAR, are involved in the development of XtreemOS.
CNR's contributions to XtreemOS
Scalable, Highly available services
Within XtreemOS, the CNR team is involved in the development of the highly available services that tie up into a system the XtreemOS individual machines on a grid. CNR has developed the Service/Resource Discovery System (SRDS), that leverages on overlay networks and exploits the functionalities of the XtreemFS-F basic layer of XtreemOS,
The SRDS daemons run distributed over the nodes of the XtreemOS platform, and offer resource and service discovery services to other XtreemOS modules and applications. SRDS resolves key-based and range-based queries, over both dynamic and static information. The main clients of the SRDS are currently the Application Execution Management (AEM) and the Job Directory Service (JDS) modules. They access information about computing resources and running jobs within a Virtual Organization.
The SRDS layered architecture employs a combination of overlay networks and query resolution algorithms to improve on previous protocols (like LDAP) in terms of scalability, fault tolerance and flexibility. We exploit overlays based on the Distributed Hash Tree (DHT) paradigm, as well as custom networks like the RSS (also developed within XtreemOS, by VUA). The SRDS provides scalability, reliability and high availability on very large Grids, while also being able to satisfy the needs of diverse clients in terms of information management primitives and of access protocols (DIXI, RMI, Https).
The CNR team also participates in the development of the Object Storage Device component (OSD) of XtreemFS, the XtreemOS File System. According to the object-based storage design paradigm for parallel and distributed file systems, within XtreemFS the content of a single file is represented by one or more objects. The OSD is responsible for storing file content as a set of objects, providing access to file data interacting with the other XtreemFS components.
CNR contributes to the second development phase of XtreemFS focusing on testing it for POSIX-compliance. In this phase, the NTFS-3G test suite has been selected and is being used to assess the XtreemFS behaviour.
The Institute of Information Science and Technologies (ISTI) is located in the CNR Research Area of Pisa. ISTI was constituted in September 2000, becoming fully operational in 2002, as a result of a merger between the former CNUCE and the IEI institutes of CNR. As a result, ISTI is the largest public IT research institute in Italy, enrolling about 90 permanent researchers, for a total of about 170 personell units with research fellows and PhD students.
The domain of competence of ISTI covers Information Science, related technologies and a wide range of their applications. ISTI groups into 5 main thematic areas its 16 research laboratories, covering IT application fields as different as Visual Computing from Knowledge discovery, and of course including High Performance Computing (http://www.isti.cnr.it/ResearchUnits/Labs/hpc-lab/).
Basic and applied research in the IT domain requires close integration of all components of the research community and academia with wide sectors of the industry, of business and of society at large. ISTI hosts the Italian Regional W3C Office, and is actively involved in collaborations with the academic world and in cooperative research, as well as in training and development programs, both national and international.
The Institute of High Performance Computing and Networking (ICAR) (http://www.icar.cnr.it) has been established on May 2001 and is operative since June 2002. The Institute is located at Rende, Cosenza (Italy) and has two branches in Naples and in Palermo. ICAR currently enrolls about 60 researchers and technicians, distributed on the three sites. ICAR performs basic and applied research in the areas of High Performance Computing and Networking and Intelligent Systems, by cooperating with the academic world and other private and public research organizations active in these areas. In addition to its research activities, the Institute works on pre-competitive development and technological transfer of research results, and carries out educational and training activities, through scholarships and research fellowships, advanced post-graduate specialization courses, and non-university higher education activities.