Dear all,
Following up on the data migration from Mjolnir1 to your new project home, we need to speed up the process. I’ve heard rumor that billing may start from Q2, meaning all data must be moved by March 14 to be cleared from the cache before April 1.
Since we have scheduled downtime this weekend when no jobs can run, I will take advantage of this time to clean up data.
Process for the coming weeks:
1. User Removal from Accounting Database
- During the maintenance window (Saturday), I will remove all users from the accounting database, with no exceptions.
- This means you will not be able to submit jobs to the SLURM queue until I reinstate your account. Your account will not be reinstated unless this form has been filled out: https://airtable.com/app0YI6HJDQGmtIru/pagOoCi5v6Z8siKml/form
- This process is unfortunately necessary as some users are still generating data in Mjolnir1, despite agreements allowing only three people to do so due to pre-scheduled jobs.
2. Data Migration Confirmation & Access Restoration
- Once you have fully migrated your data, please fill out this form: https://airtable.com/app0YI6HJDQGmtIru/pagOoCi5v6Z8siKml/form.
- Users who have already moved their data should also fill in this form to ensure their user-id will be reinstated in the accounting database after I have checked for leftover data. You can do this already now.
- After I verify that no data remains under your username in /projects/mjolnir1, I will remove your access to Mjolnir1 and reinstate you in the accounting database. This may take a few days depending on the workload for me and Mjolnir.
- Once reinstated, you can use the cluster as normal.
3. Access for Users Still Migrating Data
- If you still have data left to move, the form will include an option to apply for job submission access to copy/rsync your data.
- Your user will be added to a QoS limited rsync queue, instructions will follow
- I will be actively monitoring these jobs—if they are not transferring data and are instead running compute, they will be killed with no warning and your access will be removed while I move your data.
- I will not be kept responsible for data loss in this process.
4. If you don’t know how to migrate the data?
- Please fill out the same form as above, https://airtable.com/app0YI6HJDQGmtIru/pagOoCi5v6Z8siKml/form, and I will do it for you starting immediately. During that process I might remove your access to ensure a smooth process and minimize risk of corrupted data.
5. Software Access Changes
- After being removed from Mjolnir1, you will lose access to software in /projects/mjolnir1/apps/modules and /projects/mjolnir1/apps/conda.
- You can check where your software is located by running, e.g., module apropos bwa.
- I am working hard to reinstall software, but some packages take hours to process, and there are only so many hours in a day. If you need specific software prioritized, please submit a ticket.
I understand this is an inconvenience, but we still have 1.4 PB of data to move (DKK 129,000/month – DKK 1,500,000/year).
Since we are one of the largest projects, we may be among the first to be billed, making it even more urgent to complete the migration.
Also, please bear with me during this period—I know I might not be the most cheerful person right now. Between handling emails, tickets, managing this transition and still doing my other work, I’m doing my best to make it as smooth as possible for everyone while keeping costs down for the future.
Best regards,
Bent
Was this article helpful?
That’s Great!
Thank you for your feedback
Sorry! We couldn't be helpful
Thank you for your feedback
Feedback sent
We appreciate your effort and will try to fix the article