Introduction

This wiki page presents some results on bulk lfc methods.

Bulk methods which have been tested

There are four methods.
  • Test 3: int lfc_delfilesbyguid (int nbguids, const char * guids, int force, int * nbstatuses, int * statuses) : allows deleting the replicas and LFNs associated with the given list of guids if the flag force = 1. if force = 0 and there are still replicas for a given guid, nothing is deleted.
  • Test 4: int lfc_delfilesbyname (int nbfiles, const char * paths, int force, int *nbstatuses, int *statuses) : allows deleting the replicas and LFNs associated with the given list of LFNs if the flag force = 1. if force = 0 and there are still replicas for a given LFN, nothing is deleted.
  • Test 5: int lfc_delfilesbypattern (const char *path, const char *pattern, int force, int *nbstatuses, Cns_filestatus **statuses) : allows deleting the replicas and LFNs if the LFN is like 'path/pattern' and the flag force = 1. if force = 0 and there are still replicas for a given LFN which matches the pattern, nothing is deleted.
  • Test 6: int lfc_delreplicas (int nbguids, const char * guids, char * se, int * nbstatuses, int * statuses) allows deleting the replicas only.

Local LFC server and client

Hw and nw properties

  • LFC server :lxb0981.cern.ch with MySQL (SL4)
  • LFC client : lxb0728.cern.ch (SL3)
  • ping time 0.34ms
  • nb of entries in the DB : 1349134

Test conditions

Tests on different ways to delete files in the LFC. Each test has been repeated three times (same number of entries in the DB).
  • Test 1 : lfc_delreplica and lfc_unlink to delete both replicas and LFNs.
  • Test 2 : lfc_delreplica and lfc_unlink to delete both replicas and LFNs inside a session.
  • Test 3 : lfc_delfilesbyguid to delete both replicas and LFNs (force = 1).
  • Test 4 : lfc_delfilesbyname to delete both replicas and LFNs (force = 1).
  • Test 5 : lfc_delfilesbypattern to delete both replicas and LFNs (force = 1).
  • Test 6 : lfc_delreplicas to delete replicas only.

Test results

Execution time is given in second.
Type of test \ #files to delete Test 1 Test 2 Test 3 Test 4 Test 5 Test 6
0 0.32 0.32 0.33 0.37 12.92 0.32
1 0.68 0.35 0.35 0.38 13.41 0.33
10 6.56 0.55 0.44 0.52 12.59 0.41
100 66.94 2.18 1.07 1.19 13.07 0.83
500 335.92 8.39 4.16 4.64 15.89 3.85
1000 677.16 24.65 9.04 9.30 24.78 7.68
5000 3344.46 96.79 56.27 62.05 71.56 34.75
10000 6730.47 194.48 121.54 120.45 128.22 68.76

Execution time of the different type of test in a graph.



This table gives the execution time to delete one file (one LFN + one replica, except for Test 6 where there is only one replica deleted, no LFN). Execution time is given in second.

Type of test \ #files to delete Test 1 Test 2 Test 3 Test 4 Test 5 Test 6
0 0.320 0.320 0.330 0.370 12.920 0.320
1 0.680 0.350 0.350 0.380 13.410 0.330
10 0.656 0.055 0.044 0.052 0.126 0.041
100 0.669 0.022 0.011 0.012 0.131 0.008
500 0.672 0.017 0.008 0.009 0.032 0.008
1000 0.677 0.025 0.009 0.009 0.025 0.008
5000 0.669 0.019 0.011 0.012 0.014 0.007
10000 0.673 0.019 0.012 0.012 0.013 0.007



Local LFC server and remote LFC client

Hw and nw properties

  • LFC server :lxb0981.cern.ch with MySQL (SL4)
  • LFC client (Tawain) : lcg00122.grid.sinica.edu (SL3)
  • ping time : 312.64 ms
  • nb of entries in the DB : 1349134

Test conditions

Tests on different ways to delete files in the LFC. Each test has been repeated three times (same number of entries in the DB).
  • Test 1 : lfc_delreplica and lfc_unlink to delete both replicas and LFNs.
  • Test 2 : lfc_delreplica and lfc_unlink to delete both replicas and LFNs inside a session.
  • Test 3 : lfc_delfilesbyguid to delete both replicas and LFNs (force = 1).
  • Test 4 : lfc_delfilesbyname to delete both replicas and LFNs (force = 1).
  • Test 5 : lfc_delfilesbypattern to delete both replicas and LFNs (force = 1).
  • Test 6 : lfc_delreplicas to delete replicas only.

Test results

Execution time is given in second.
Type of test \ #files to delete Test 1 Test 2 Test 3 Test 4 Test 5 Test 6
0 3.06 3.68 3.06 3.21 15.37 3.07
1 6.19 4.03 3.08 3.23 15.88 3.07
10 63.30 9.84 3.19 3.38 15.58 3.41
100 626.30 71.22 4.09 4.52 16.31 3.89
500 3182.93 331.01 8.09 8.31 19.26 7.90
1000 6233.26 654.35 12.25 12.94 23.08 11.28
5000   3294.76 61.30 68.45 72.01 42.23
10000   6534.10 125.96 131.92 133.49 78.75

Execution time of the different type of test in a graph.

Execution time is given in second.

Type of test \ #files to delete Test 1 Test 2 Test 3 Test 4 Test 5 Test 6
0 3.060 3.680 3.060 3.210 15.370 3.070
1 6.190 4.030 3.080 3.230 15.880 3.070
10 6.330 0.984 0.319 0.338 0.156 0.341
100 6.263 0.712 0.041 0.042 0.163 0.039
500 6.365 0.662 0.016 0.017 0.039 0.016
1000 6.233 0.654 0.012 0.013 0.023 0.011
5000   0.651 0.012 0.014 0.014 0.008
10000   0.653 0.0126 0.013 0.0133 0.008


-- LanaAbadie - 27 Nov 2007

Topic attachments
I Attachment History Action Size DateSorted ascending Who Comment
JPEGjpg lfc_local_5.JPG r1 manage 23.4 K 2007-12-10 - 17:06 LanaAbadie LFC bulk test : local client
JPEGjpg lfc_local_6.JPG r1 manage 22.6 K 2007-12-10 - 17:07 LanaAbadie LFC bulk test : local client without test 1
JPEGjpg lfc_local_10.JPG r1 manage 20.6 K 2007-12-13 - 10:00 LanaAbadie LFC local client : time to delete one file (Test 2, 3 and 4 only)
JPEGjpg lfc_local_11.JPG r1 manage 28.8 K 2007-12-13 - 10:02 LanaAbadie LFC local client : time to delete one file
JPEGjpg lfc_local_7.JPG r1 manage 25.5 K 2007-12-13 - 09:54 LanaAbadie LFC remote client : all tests
JPEGjpg lfc_local_8.JPG r1 manage 24.1 K 2007-12-13 - 09:55 LanaAbadie LFC remote client : without Test 1 and Test 2
JPEGjpg lfc_local_9.JPG r1 manage 26.0 K 2007-12-13 - 09:57 LanaAbadie LFC local client : time to delete one file
C source code filec lfc_delete_test.c r1 manage 12.6 K 2007-12-14 - 15:08 LanaAbadie source code used to make the test for the bulk delete methods
Edit | Attach | Watch | Print version | History: r11 < r10 < r9 < r8 < r7 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r11 - 2008-02-13 - LanaAbadie
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback