2025-01-15 - Happy new year, storage cost and the closing of Mjolnir1

Created by Bent Petersen, Modified on Wed, 15 Jan at 2:43 PM by Bent Petersen

Dear Mjolnir Users and PI's,

As we bid farewell to 2024 and welcome 2025, I want to take a moment to reflect on the incredible progress we’ve achieved together and share some very important updates for the year ahead.

Reflecting on 2024

Milestone Achieved: In January, Mjolnir surpassed 10 million jobs processed, and by the end of the year, we had processed over 24 million jobs in 2024 alone. This tremendous growth necessitated moving the accounting database from a virtual server to the old mjolnirhead01 node to provide a dedicated server for job tracking.
Enhanced Stability: In April, we implemented a new gateserver with increased memory. Since then, system crashes have been reduced to almost zero, thanks to additional CPU and memory monitoring. This ensures a stable environment for users and eliminates crashes caused by mistakenly launched compute jobs.
Teaching Node: In December, the teaching node was used for the first time during the Ancient DNA course, allowing over 30 students to experience our HPC.
Organized Databases: Databases were cleaned and moved to an organized, read-only repository accessible across all Globe HPC systems.
User Tools:
- Cluster Utilization Overview: https://mjolnir-ucph.dk/current_cluster_utilization.html
- Job Usage Statistics: https://mjolnir-ucph.dk/job_statistics.html
- Ticket System: Introduced in 2024, the ticket system (available from https://mjolnir-ucph.dk/index.html) streamlines support requests. Until now there has been 221 tickets and counting. Please use it for issues instead of email to keep everything organized and to make my life easier.
Recognition Beyond UCPH: Other faculties and even another university have contacted me for guidance on building a replicate of Mjolnir, showcasing its success as a model for research infrastructure.

Challenges and Storage Cost Changes

The most significant challenge for 2025 is the introduction of storage costs for data hosted on KU-IT filesystems. This policy applies to all users across the university and the decision stems from the highest management.

Expected Costs: The current estimate is 1,100 DKK per TB per year, higher than Computerome (708 DKK/TB/year). However, KU-IT plans to offer a “free allowance,” such as 10 TB per expense owner.
Expense Ownership: Every project folder will have an expense owner responsible for its associated storage costs. PIs will oversee their group’s data, and users will be reorganized into PI-specific projects to simplify management and billing.
Mjolnir1 Closure: The /projects/mjolnir1 directory will close within a few months. All data from /projects/mjolnir1/data must be moved within the next month. Users who fail to respond by the deadline I give in separate emails will have their data deleted after multiple contact attempts.

I expect that the first bills will be sent out in Q1, which necessities that we work fast with reducing the storage and I need you all to cooperate with me.

How Much Data Do I Have and Where?

On December 19, I gained access to a new dashboard that allows me to monitor storage usage for projects hosted on Mjolnir. I spent the next 7 days creating a system which have the ability to inform with monthly reports:

For Users: Each user will receive an email summarizing their storage usage in TB, along with the estimated monthly and annual costs across all projects. This report only includes data stored in user-specific folders (“people” folders).
For PIs: PIs will receive an overview of their group’s total usage, including data left by deleted users or by users with a different PI. This will help PIs make informed decisions about retaining or deleting legacy data. Usage in shared folders such as “apps,” “data,” and “scratch” will also be summarized and attributed to the PI, as individual contributions cannot be differentiated automatically.

The first email was sent out 28 December and since then 470TB data has been cleared, corresponding to a yearly cost saving of 517,000 DKK. Keep up the good work!

These reports will be sent monthly, but I can generate them on request from PI's, for example, before group meetings or budget discussions.

Actions Required

Clean Your Data: Delete old files, gzip raw sequencing data, and organize your storage. Avoid offloading everything to ERDA, as it may soon be replaced by DEIC storage, which will also incur costs. Only upload raw data and results, no temporary data.
Read the Mjolnir related emails when I send them to avoid disruption in your work. This also applies to all PIs.
PI Group Projects:
1. PIs must provide their preferred project group name(s) and a list of associated users (I will reach out in a separate email). If there is no response their users access will be temporarily revoked.
2. Users will be moved into these new projects, and /projects/mjolnir1 will be phased out. During the process there will be disruption to the individual users work, it is unavoidable.

New Software Repository

In December, I introduced /datasets/globe_software as a new shared repository for software modules across all Globe HPC systems.

Moving forward, all new software will be installed here, and existing software will be migrated from /projects/mjolnir1. This process will take time, given the over 1,150 software modules I have currently installed.

Looking Ahead

Despite these challenges, 2025 promises to be another exciting year. Here’s what you can expect:

Monthly storage and compute reports to improve transparency.
Ongoing enhancements to system stability and documentation.
Upgraded OS and new Slurm
Better tools for managing jobs and disruptions, including a disruption notification list and forms for job time extensions.
Possibility to have external users for short term courses, with no guest contract

Zoom Meetings next week if you have any questions:

I will host a few Zoom meetings next week, which you can join if you have any questions. I do not plan to make a presentation, it will be an open interactive sessions where you can join if you have any questions in relation to the upcoming changes. I encourage PI’s to join as well.

Monday 20th 11.00 - 12.00: https://ucph-ku.zoom.us/j/66723663247?pwd=nRx1RTJ66qGvWBfJcA3BJaxa6xYbHh.1

Monday 20th 14.00 - 15.00: https://ucph-ku.zoom.us/j/68795255251?pwd=ImyhZjzpF4X9qclMSamM3N8LRJ6Y5H.1

Thursday 23rd 11.00 - 12.00: https://ucph-ku.zoom.us/j/65642684906?pwd=xbVNqD9WNfrp20gvJfglaUaczbnZ86.1

Thursday 23rd 14.00 - 15.00: https://ucph-ku.zoom.us/j/68032305681?pwd=l9lVNbbiaGgBbYrIT4UeLAV76QKt4G.1

I want to acknowledge that the coming changes, particularly those related to storage, may cause disruptions for users and their work.

Please, I therefore kindly ask for your patience with me and for your understanding, as the steps I will take in the coming months may feel overwhelming but are essential for managing costs effectively and ensuring Mjolnir’s long-term sustainability.

Thank you for your collaboration and contributions to making Mjolnir such a successful system. I look forward to continuing working with Mjolnir and all of you in 2025 and beyond.

If you have questions, concerns, or feedback, please don’t hesitate to reach out via the ticket system or email. Wishing you all a productive, innovative, and happy new year!

Best regards,

Bent