tambakhk
Cadet
Cadet
  • 147 Views

Ansible: Reduce copy time when copying within localhost

Jump to solution

copy module performing recursive copy from localhost to localhost is taking lot of time(more than 3 minutes). How can I reduce this time ?

 

  - name: copy folders
    copy:
      dest: "{{ tmp_repo_directory }}"
      src: "{{ item }}"
      force: true
    loop:
      - "{{ playbook_dir }}/outputs"
      - "{{ playbook_dir }}/inventory"

 

 

Labels (2)
0 Kudos
1 Solution

Accepted Solutions
bonnevil
Starfighter Starfighter
Starfighter
  • 129 Views

So, I assume that this task is running in a play that only has localhost in its hosts specification.    I'm also assuming that tmp_repo_directory and playbook_directory each hold the name of one directory.

The documentation for the copy module mentions that it scales poorly once you're dealing with a large number of files.  I don't recall why that is, but I know I had a discussion with upstream developers about that once.  It might be that copy is shoving all the file content through the Ansible connection, which if true would add overhead, but I don't know whether or not that is true.  There might be ways to play with that (especially if the copy is going over an SSH connection instead of using local).

You could try to use the synchronize (ansible.posix.synchronize) module, which is a wrapper for rsync.  Read the Notes in the documentation for that module to understand how it works in an Ansible context, the details on how to specify src and dest, and do some careful testing first.  If I remember correctly (I might not), by default the source is on the control node and the dest is on the current host in the play, but you can adjust the source with delegate_to.  The remote_src option for the module also looks interesting.

View solution in original post

2 Replies
bonnevil
Starfighter Starfighter
Starfighter
  • 130 Views

So, I assume that this task is running in a play that only has localhost in its hosts specification.    I'm also assuming that tmp_repo_directory and playbook_directory each hold the name of one directory.

The documentation for the copy module mentions that it scales poorly once you're dealing with a large number of files.  I don't recall why that is, but I know I had a discussion with upstream developers about that once.  It might be that copy is shoving all the file content through the Ansible connection, which if true would add overhead, but I don't know whether or not that is true.  There might be ways to play with that (especially if the copy is going over an SSH connection instead of using local).

You could try to use the synchronize (ansible.posix.synchronize) module, which is a wrapper for rsync.  Read the Notes in the documentation for that module to understand how it works in an Ansible context, the details on how to specify src and dest, and do some careful testing first.  If I remember correctly (I might not), by default the source is on the control node and the dest is on the current host in the play, but you can adjust the source with delegate_to.  The remote_src option for the module also looks interesting.

tambakhk
Cadet
Cadet
  • 117 Views

Yes, synchorise option seems to reduce time by a good margin. Since, I have used hosts as the localhost, I have not used delegate option. 

- name: copy folders
  synchronize:
    dest: "{{ tmp_repo_directory }}"
    src: "{{ item }}"
  loop:
  - "{{ playbook_dir }}/outputs"
  - "{{ playbook_dir }}/inventory"
0 Kudos
Join the discussion
You must log in to join this conversation.