cancel
Showing results for 
Search instead for 
Did you mean: 
Highlighted
rmokkink Mission Specialist
Mission Specialist
  • 850 Views

ceph lab environment

I opened a case regarding the performance of the labs. First feedback, it was my firewall or vpn.
I explained it was the lab, because we you have a console to compute resources in the lab, and there it is slow.


Installing the ceph cluster takes hours. I deleted and reprovisioned the lab again, but still it is slow.
When the cluster is up, everything is healthy, but creating a pool takes minutes.

More people experiencing these issues.

 

 

Labels (1)
24 Replies
Highlighted
Flight Engineer littlebigfab Flight Engineer
Flight Engineer
  • 841 Views

Re: ceph lab environment unworkable slow

Hello @rmokkink,

I have the exact same issue here (France).

It took me more than an hour to run each installation playbook (the training course says it's supposed to take 5 minutes).

Likewise, creating a pool takes about 10 minutes instead of a few seconds.

It's extremely annoying, first because it wastes a lot of my lab time, and second because I have scheduled the corresponding certification exam pretty soon and I can't afford to waste time Smiley Sad

@cschunke, could you please check ?

 

0 Kudos
Reply
Loading...
Highlighted
rmokkink Mission Specialist
Mission Specialist
  • 825 Views

Re: ceph lab environment unworkable slow

Hi,

Thanks for the response. I was so frustrated that i installed my own ceph lab on my laptop.
It took 5 minutes for the playbook to install the same setup as the lab environment.
To mimic the lab i used the following:

centos 7.7
ceph-ansible branch table-3.2.30
ansible 2.6.9 => https://docs.ceph.com/ceph-ansible/master/#releases

I still haven't heard anything from my supportcase, since i escalated it.
I will put this thread in the case and maybe escalate again.

0 Kudos
Reply
Loading...
Highlighted
Flight Engineer littlebigfab Flight Engineer
Flight Engineer
  • 818 Views

Re: ceph lab environment unworkable slow

Hi,

After some performance tests, it seems that the issue comes from the network only.

Maximum attainable bandwidtch between nodes is under 5MB/s, which is way below the 1Gb/s minimum requirement for Ceph.

You can easily verify that by copying a large file from one server to an other.

As a result, I'm stuck at the first step of lab 3.2, being totally unable to create a replicated pool on server[c-e]. It takes hours and never returns.

On the all-in-one cluster serverf, the same command works in about 2min45, which is fine. This confirms that the issue comes from poor inter-nodes network performance.

 

Reply
Loading...
rmokkink Mission Specialist
Mission Specialist
  • 812 Views

Re: ceph lab environment unworkable slow

I do the labs on my homelab, i have attached this thread to the supportcase.
If he issues are not fixed on monday, i will escalate the case.

0 Kudos
Reply
Loading...
Highlighted
Moderator
Moderator
  • 621 Views

Re: ceph lab environment unworkable slow

First, sorry to hear about the performance issue that you are all experiencing. This must be frustrating. I will forward that to the team responsible for this course to see if there's a workable solution.

Reply
Loading...
Highlighted
Flight Engineer littlebigfab Flight Engineer
Flight Engineer
  • 598 Views

Re: ceph lab environment unworkable slow

Hi @Razique and thank you very mutch for your concern.

I had created a support case too. They have just changed my lab to a different datacenter. I had to re-provision the lab from scratch though, which I have just done. I can now transfer at 9MB/s between my lab's VMs, which is almost twice faster than before, but still pretty slow compared to Ceph network requirements (1Gb/s).

I will re-try the CEPH125 first labs later today and post an update here.

0 Kudos
Reply
Loading...
Highlighted
Moderator
Moderator
  • 541 Views

Re: ceph lab environment unworkable slow

Hey @littlebigfab how long did it take you to provision in this new DC?

0 Kudos
Reply
Loading...
Highlighted
Flight Engineer littlebigfab Flight Engineer
Flight Engineer
  • 533 Views

Re: ceph lab environment unworkable slow

Hi @Razique Smiley Happy

Thank you for asking.

I did the test only last night. Unfortunately, it still took 2 hours and 40 minutes to install Ceph on 3 nodes + 1 client with the ansible-playbook command.

 

Capture d’écran 2019-12-12 à 03.00.29.png

 

 

 

I have updated my support case with that information.

 

It's troubling how even the skipped tasks seem to be pretty slow. Based on my personal experience with ansible in general, they usually go so fast you can't read them.

I had activated SSH pipelining for that run, but it didn't help at all.

Now I feel miserable because I have my EX125 exam scheduled early January with travel arrangements, and I feel like I won't be able to practice at all until then Smiley Sad

Another hurdle is that even preparing a clean lab environment for such an Ansible installation itself is pretty tedious with the ansible inventory file and all the group_vars files to populate manually before being able to run the ansible-playbook command.

Thank you very much in advance if you can help here. Since the issue has been encountered by @rmokkink too, and since it has occured in different datacenters too, I guess it should be pretty easy for Red Hat to reproduce it.

Best !

0 Kudos
Reply
Loading...
Highlighted
rmokkink Mission Specialist
Mission Specialist
  • 530 Views

Re: ceph lab environment unworkable slow

Hi,

I have heard from support that the issue is escalated to the ROL backend team, this was on the 10th of december. If i hear anything i will report back.

At the moment i am still doing labs on my laptop, most labs i can do pretty easy.

0 Kudos
Reply
Loading...
Join the discussion
You must log in to join this conversation.