gLite WMS Work Log

rb101 (Experimental gLite WMS 3.1 for Atlas)

  • 2007-04-10: Sandbox cleaning cronjob takes too much time and there already many of them started. Temporary solution: killall glite-wms-purgeStorage.
  • 2007-04-04: installed hourly cronjob glite-restart for automatic restart of interlogd service.
  • 2007-04-03: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-30: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-27: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-26: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-26: alarm NOSPMA has been disabled.
  • 2007-03-26: Sandbox cleaning cronjob takes too much time and there already many of them started. Temporary solution: killall glite-wms-purgeStorage.
  • 2007-03-25: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-23: interlogd service restarted.
  • 2007-03-21: interlogd service restarted.
  • 2007-03-20: New packages glite-wms-ns-common-3.1.3-1 and glite-jdl-api-cpp-3.1.5-1 installed. Services glite-wms-ns and glite-wms-wm restarted.
  • 2007-03-20: The following new packages have been installed:
                glite-wms-wmproxy-interface-3.1.6-1
                glite-wms-wmproxy-3.1.12-1
                glite-wms-matchmaking-3.1.4-1
                glite-wms-manager-3.1.13-1
                glite-wms-jobsubmission-3.1.6-1
                glite-wms-helper-3.1.9-1
                glite-wms-configuration-3.1.4-1
                glite-wms-common-3.1.8-1
                glite-wms-checkpointing-3.1.2-1
                glite-wms-brokerinfo-access-3.1.2-1
  • 2007-03-14: Set the following variables in the /opt/glite/etc/glite_wms.conf file:
           JobController = [
                ...
                DagmanMaxPre = 5;
                dagmanloglevel = 5;
           ]
           WorkloadManager = [
                ....
                MatchRetryPeriod = 14400;
           ]
Services glite-wms-wm restarted and glite-wms-jc reloaded.

  • 2007-03-13: upgrade to gLite 3.1.
  • 2006-12-22: root access given to two developers.

rb102 (experimental gLite WMS for CMS and Atlas)

  • 2006-07-19: machine under full GD responsability.
  • 2006-07-21: Upgraded to the latest release candidate (3.0.2 RC6):
    • glite-wms-workload_manager daemon could not start because of bug 18343.
    • Downgrading glite-wms-manager to the version in previous RC doesn't work too.
    • In order to have a working WMS, glite-wms-helper, glite-wms-ism and glite-wms-manager have been downgraded to glite-wms-ism-1.5.8-1.i386.rpm, glite-wms-helper-1.5.17-1.i386.rpm and glite-wms-manager-1.5.14-1.i386.rpm
    • Applying patches: 1) Replace /opt/condor-6.7.10/bin/condor_dagman with one from version 6.7.19, 2) Add "GlobusDownTimeout = 7200; " in the LogMonitor section of /opt/glite/etc/glite_wms.conf, 3) Add "PipeDepth = 200;" and "WorkerThreads = 20;" and delete "NumberOfWorkerThreads = 1;" in the WorkloadManager section of /opt/glite/etc/glite_wms.conf
  • 2006-07-24: Upgraded to the latest WMS patch which fixes the bug 18343.
  • 2006-07-26: Upgraded to Condor 6.7.19 and the latest release candidate compiled against the newer Condor.
    • Remove gLite packages by hand and then reinstall the ones compiled against the newer Condor since the glite packages version are the same.
    • Change the condor version to 6.7.19 in /opt/glite/etc/config/glite-wms.cfg.xml.
    • Add CONDOR_CONFIG=/opt/condor-6.7.19/etc/condor_config into gLite.def.
    • Add Condor library path /opt/condor-6.7.19/lib into glite environment setting script /etc/glite/profile.d/glite_setenv.*sh.
  • 2006-07-27: VO Atlas now supported on this machine.
  • 2006-07-27: User giaco authorized to access this machine via his AFS account.
  • 2006-07-27: 6 glite-lb-bkserverd processes in infinite loop, this is a known bug (18123), kill them by hand and mail sent to developer to ask tag version for the fix.
  • 2006-07-28: 10 glite-lb-bkserverd and 8 glite-lb-logd in infinite loop, kill them by hand.
  • 2006-07-28: Two parameters, "GRIDMANAGER_MAX_JOBMANAGERS_PER_RESOURCE = 5" and "HOLD_JOB_IF_CREDENTIAL_EXPIRES = TRUE", added into condor configuration.
  • 2006-07-30: Some services not automatically restarted after the power cut at Cern (glite-proxy-renewd, glite-lb-proxy, httpd). All the services have been stopped and restarted successfuly.
  • 2006-07-31: User emolina authorized to access this machine via her AFS account.
  • 2006-08-04: Added GRIDMANAGER_MAX_PENDING_REQUESTS = 1000 into condor configuration file, and run /opt/condor-c/sbin/condor_reconfig, GRIDMANAGER_MINIMUM_PROXY_TIME = 600, GRIDMANAGER_MAX_SUBMITTED_JOBS_PER_RESOURCE = 32000, GRIDMANAGER_GAHP_CALL_TIMEOUT = 900, GRIDMANAGER_GLOBUS_COMMIT_TIMEOUT = 3600, GRID_MONITOR_HEARTBEAT_TIMEOUT = 300, GRID_MONITOR_RETRY_DURATION = 31536000, GLOBUS_GATEKEEPER_TIMEOUT = 1800 added later.
  • 2006-08-06: Directory /tmp full of files generated by condor (for example condorc-advertiser.21086.19110.err, condorc-advertiser.21086.19110.out, condorc-launcher.21086.19110.err, etc.). There was about 131541 files, and it was no more possible to create new files in this directory. All these files have been saved in /root/tmp-bak.tar.bz2) and I deleted all the empty files. At the end of the afternoon, I deleted once again all the empty files (about 3600 files...).
  • 2006-08-08: CAs upgraded to version 1.8-1.
  • 2006-08-08: Two new patches(772 for loop on broken connection and credential loading and 794 for WMS must distinguish VOMS groups and roles) applied, condor configuration file changed to use LCG modified GAHP server and grid monitor script, GRIDMANAGER_MAX_JOBMANAGERS_PER_RESOURCE increased to 30, and also all services restarted.
  • 2006-08-09: Cron job /etc/cron.d/delete-empty-condor-file.cron written in order to solve the problem described on 2006-08-06 (directory /tmp full of condor files).
  • 2006-08-09: Jobs monitoring tool installed in order to monitor the number of running and idle jobs in the condor queue. Results available at the following link (only from inside the Cern): http://lxb1524.cern.ch/plots.html.
  • 2006-08-11: Added option "-c 60" into daemon glite-lb-proxy startup line in /opt/glite/etc/init.d/glite-lb-proxy and restarted all services.
  • 2006-08-11: created, as root, the directory /opt/glite/var/log to get rid of the recurring error in /var/log/messages: rb102 gridftpd[3718]: cannot open logfile /opt/glite/var/log/xferlog: No such file or directory (see bug http://savannah.cern.ch/bugs/?func=detailitem&item_id=18306)
  • 2006-08-15: Applied two patches (patch 807: For jobs submitted to WMProxy, the job wrapper tries to upload to the WMS also non-existent files, and patch 808: LB proxy retry on EAGAIN), and restarted all services since library changed.
  • 2006-08-17: Increased BacklogSize from 16 to 64 and added "ConnectionTimeout = 300;" in NetworkServer section of /opt/glite/etc/glite_wms.conf, and restarted glite-wms-ns.
  • 2006-08-09: Jobs monitoring tool installed on a new machine accessible from outside the CERN. The results are available at the following link: http://lxb2007.cern.ch/plots.html.
  • 2006-08-24: Kernel upgrade. Machine rebooted and services restarted.
  • 2006-08-25: Modified files under /var/glite/logmonito/CondorG.log to temporarily solve the LM daemon crash problem, for example, when it complained "DAG subjob: https://rb102.cern.ch:9000/utebHt3WzZppApLW2Qk27g appeared in a log file not associated with a DAG" and crashed:
    • With grep command found it is the log info in dag.https_3a_2f_2frb102.cern.ch_3a9000_2fpkxB0v670tLO8_5fNbHCkvRw.log under /var/glite/logmonito/CondorG.log.
    • Stop all services by /etc/init.d/gLite stop
    • Delete the blocks related to that subjob (for example, its condor job ID is 243988, then delete all blocks for 243988) in dag.https_3a_2f_2frb102.cern.ch_3a9000_2fpkxB0v670tLO8_5fNbHCkvRw.log
    • Restart all services by etc/init.d/gLite start
  • 2006-8-29: Added "dagmanloglevel = 5;" into LM and JC section of /opt/glite/etc/glite_wms.conf to increase dagman log level, and restarted LM, WM and JC daemon.
  • 2006-08-29: Add AFS user gianelle for interactive access.
  • 2006-08-30: Stopped gLite services to do IO benchmark with bonnie, it got 37.7MB/s for character writing and 41.5MB/s for block writing.
  • 2006-08-30: Added GLITE_WMS_QUERY_TIMEOUT=480 into /etc/glite/profile.d/glite_setenv.(c)sh and also /opt/condor-c/local.rb102/condor_config.local, restarted services
  • 2006-08-31: Logmonitor crashed again, deleted the blocks whoch caused it, and restarted LM.
  • 2006-09-05: Applied patches, 833, 834, 835 and 836, and restarted services
  • 2006-09-08: There was a problem /tmp partition, reformated it
  • 2006-09-11: Tim split the rb-state RAID5 partition into three RAID1 partition, followings were done:
    • /etc/init.d/mysql stop ; mkdir -p /data03/var/lib/ ; mv /var/lib/mysql/ /data03/var/lib ; ln -s /data03/var/lib/mysql/ /var/lib/; mkdir -p /data01/var/glite; chown glite:glite /data01/var/glite; ln -s /data01/var/glite /var/; mkdir -p /data01/var/tmp/; chmod 1777 /data01/var/tmp/; ln -s /data01/var/tmp /var/; mkdir -p /data03/var/log/glite; chmod 1775 /data03/var/log/glite; ln -s /data03/var/log/glite/ /var/log/; mkdir -p /data03/var/glite/networkserver; mkdir -p /data03/var/glite/proxycache; mkdir -p /data03/var/glite/spool; mkdir -p /data03/var/glite/workload_manager; chown glite:glite /data03/var/glite/networkserver; chown glite:glite /data03/var/glite/proxycache; chown glite:glite /data03/var/glite/workload_manager; chmod 771 /data03/var/glite/proxycache; mkdir -p /data02/var/glite/jobcontrol; mkdir -p /data02/var/glite/logmonitor; chown glite:glite /data02/var/glite/jobcontrol; chown glite:glite /data02/var/glite/logmonitor/; chmod g+w /data02/var/glite/jobcontrol; chmod g+w /data02/var/glite/logmonitor/; ln -s /data02/var/glite/* /data01/var/glite/;
  • 2006-09-14: For compatibility, change GRIDMANAGER_MAX_JOBMANAGERS_PER_RESOURCE to 20 in condor config, and in glite_setenv.* add GLITE_PR_TIMEOUT = 300 and change GLITE_WMS_QUERY_TIMEOUT to 300
  • 2006-10-01: Increased GLITE_WMS_QUERY_TIMEOUT to 480 again, and also added "PassEnv GLITE_PR_TIMEOUT" in /opt/glite/etc/glite_wms_wmproxy_httpd.conf
  • 2006-10-12: Added HOSTNAME variable into /etc/glite/profile.d/glite_setenv.*sh to avoid the missing hostname in __maradonaprotocol
  • 2006-10-12: Changed the last line of /opt/glite/etc/glite_wms_wmproxy_httpd.conf to "FastCgiConfig -restart -restart-delay 5 -idle-timeout 3600 -KillInterval 150 -maxProcesses 25 -maxClassProcesses 10 -minProcesses 5 -listen-queue-depth 200 -initial-env LD_LIBRARY_PATH -initial-env GLITE_LOCATION_VAR -initial-env GLITE_WMS_LOCATION_VAR -initial-env GLITE_WMS_TMP -initial-env GLITE_LOCATION_LOG -initial-env GLITE_LOCATION_TMP" to reduce the glite_wms_wmproxy_server process number to reduce memory usage
  • 2006-10-12: Changed DagmanMaxPre in JobController section of /opt/glite/etc/glite_wms.conf to 2 to reduce the maximum number of planners per dag to reduce the total number of planners to reduce the memory usage
  • 2006-10-12: Changed /opt/condor-c/libexec/glite/condorc-launcher and /opt/condor-c/libexec/glite/condorc-initialize to use /var/tmp instead of /tmp for their logs.
  • 2007-03-26: alarm NOSPMA has been disabled.

rb109 (experimental gLite WMS for CMS and Atlas)

  • 2006-11-29: machine back in production. According to sysadmins, the air flow is apparently working fine now.
  • 2006-11-27: It seems that there is an airflow problem on this node. This machine will be shutdown tomorrow Tuesday 28 november and a vendor call will be opened.
  • 2006-10-18: Back in production this morning.
  • 2006-10-12: cooling problem. Need to shutdown this machine. ITCM ticket generated.
  • 2006-09-14: installation of the middleware with the same configuration than for rb102.
  • 2007-03-26: alarm NOSPMA has been disabled.

rb110 (experimental gLite WMS 3.1 for CMS)

  • 2007-04-04: installed hourly cronjob glite-restart for automatic restart of interlogd service.
  • 2007-04-03: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-30: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-27: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-26: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-26: alarm NOSPMA has been disabled.
  • 2007-03-25: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-20: New packages glite-wms-ns-common-3.1.3-1 and glite-jdl-api-cpp-3.1.5-1 installed. Services glite-wms-ns and glite-wms-wm restarted.
  • 2007-03-20: The following new packages have been installed:
                glite-wms-wmproxy-interface-3.1.6-1
                glite-wms-wmproxy-3.1.12-1
                glite-wms-matchmaking-3.1.4-1
                glite-wms-manager-3.1.13-1
                glite-wms-jobsubmission-3.1.6-1
                glite-wms-helper-3.1.9-1
                glite-wms-configuration-3.1.4-1
                glite-wms-common-3.1.8-1
                glite-wms-checkpointing-3.1.2-1
                glite-wms-brokerinfo-access-3.1.2-1
  • 2007-03-12: Remove string $maildir in /opt/glite/etc/init.d/glite-lb-bkserverd startup script and service restarted.
  • 2007-03-01: Added roles into /opt/glite/etc/glite_wms_wmproxy.gacl as attachment, for 3.1 it must be there.
  • 2006-10-23: machine put in production.
  • 2006-10-23: installation and configuration of the middleware (same configuration as for rb102).
  • 2006-11-24: Change the value of the parameter GlobusDownTimeout in /opt/glite/etc/glite_wms.conf from the default value 7200 to 43200 (12 hours). Service glite-wms-lm restarted (after sourcing file /etc/glite/profile.d/glite_setenv.sh).

rb112 (experimental gLite WMS 3.1 for LHCb)

  • 2007-04-04: installed hourly cronjob glite-restart for automatic restart of interlogd service.
  • 2007-04-04: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-29: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-26: alarm NOSPMA has been disabled.
  • 2007-03-20: New packages glite-wms-ns-common-3.1.3-1 and glite-jdl-api-cpp-3.1.5-1 installed. Services glite-wms-ns and glite-wms-wm restarted.
  • 2007-03-12: Reinstallation from scratch with gLite 3.1. I followed the same steps that the ones used for the installation of rb126.

rb117 (experimental gLite WMS 3.1 for LHCb)

  • 2007-04-04: installed hourly cronjob glite-restart for automatic restart of interlogd service.
  • 2007-04-04: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-29: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-26: alarm NOSPMA has been disabled.
  • 2007-03-20: New packages glite-wms-ns-common-3.1.3-1 and glite-jdl-api-cpp-3.1.5-1 installed. Services glite-wms-ns and glite-wms-wm restarted.
  • 2007-03-12: Reinstallation from scratch with gLite 3.1. I followed the same steps that the ones used for the installation of rb126.

rb125 (Experimental gLite WMS 3.1 for CMS)

  • 2007-04-04: installed hourly cronjob glite-restart for automatic restart of interlogd service.
  • 2007-04-03: interlogd (glite-lb-locallogger) service restarted.
  • 2007-04-02: interlogd (glite-lb-locallogger) service restarted.
  • 2007-04-02: Sandbox cleaning cronjob takes too much time and there already many of them started. Temporary solution: killall glite-wms-purgeStorage.
  • 2007-03-29: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-26: interlogd (glite-lb-locallogger) service restarted.
  • 2007-03-26: Sandbox cleaning cronjob takes too much time and there already many of them started. Temporary solution: killall glite-wms-purgeStorage.
  • 2007-03-26: alarm NOSPMA has been disabled.
  • 2007-03-20: The following new packages have been installed:
                glite-wms-wmproxy-interface-3.1.6-1
                glite-wms-wmproxy-3.1.12-1
                glite-wms-matchmaking-3.1.4-1
                glite-wms-manager-3.1.13-1
                glite-wms-jobsubmission-3.1.6-1
                glite-wms-helper-3.1.9-1
                glite-wms-configuration-3.1.4-1
                glite-wms-common-3.1.8-1
                glite-wms-checkpointing-3.1.2-1
                glite-wms-brokerinfo-access-3.1.2-1
                glite-wms-ns-common-3.1.3-1
                glite-jdl-api-cpp-3.1.5-1
Services glite-wms-ns, glite-wms-wm and glite-wms-wmproxy restarted. The II_Contact variable was set with a bad value, put lcg-bdii.cern.ch back instead.
  • 2007-03-12: Remove string $maildir in /opt/glite/etc/init.d/glite-lb-bkserverd startup script and service restarted.
  • 2007-02-14: Upgraded to 3.1 WMS
    • Create file /etc/nospma to avoid SPMA downgrade.
    • Install apt: rpm -ivh /afs/cern.ch/project/linux/cern/slc30X/i386/SL/RPMS/apt-0.5.15cnc6-8.SL.cern.i386.rpm
    • Add 3.1 WMS APT repository: rpm http://lxb2042.cern.ch/gLite/APT/R3.1-RB-pretest rhel30 externals Release3.1 updates
    • Install new repackaged c-ares and gridsite by: rpm -e --nodeps c-ares gridsite; rpm -ivh gridsite-shared-1.1.18.1-1.i386.rpm gridsite-apache-1.1.18.1-1.i386.rpm; apt-get install c-ares; rpm -e glite-LB glite-WMSLB; apt-get install glite-WMSLB; apt-get update; apt-get dist-upgrade
    • rpm -Uvh condor-6.8.4-linux-x86-rhel3-dynamic-1.i386.rpm
    • Change line 215 of /opt/condor-6.8.4/etc/condor_config to: HOSTALLOW_WRITE = *.cern.ch (Need to verify)
    • Change condor version to 6.8.4 in /opt/glite/etc/config/templates/glite-wms.cfg.xml
    • /opt/glite/yaim/scripts/configure_node /root/lcg-quattor-site-info.def glite-WMSLB
    • Add /opt/c-ares/lib/ into /etc/glite/profile.d/glite_setenv.(c)sh, source /etc/glite/profile.d/glite_setenv.sh
    • /opt/glite/yaim/scripts/configure_node /root/lcg-quattor-site-info.def glite-WMSLB
    • Modify /opt/glite/etc/init.d/glite-lb-bkserverd to add "--rgmaexport" in the bkserverd startup line
    • Add "!" after "if" in the line 99 in /opt/glite/etc/init.d/glite-lb-bkserverd
    • Add following lines into /opt/condor-c/local.rb125/condor_config.local: GRIDMANAGER_TIMEOUT_MULTIPLIER = 3, SCHEDD_TIMEOUT_MULTIPLIER = 3, COLLECTOR_TIMEOUT_MULTIPLIER = 3, C_GAHP_TIMEOUT_MULTIPLIER = 3, C_GAHP_WORKER_THREAD_TIMEOUT_MULTIPLIER = 3, TOOL_TIMEOUT_MULTIPLIER = 3, GLITE_CONDORC_DEBUG_LEVEL = 2, GLITE_CONDORC_LOG_DIR = /var/tmp, NEGOTIATOR_MATCHLIST_CACHING = False
    • Increase GLITE_WMS_QUERY_TIMEOUT from 300 to 480 and add GLITE_PR_TIMEOUT=300 in glite_setenv.(c)sh
    • rpm -Uvh glite-wms-manager-3.1.10-1.i386.rpm google-perftools-* (to reduce WM daemon memory consumption)
    • rpm -Uvh glite-wms-wmproxy-3.1.8-1_U200702091704.i386.rpm (upgrade to suicidal wmproxy server)
    • Add "PassEnv GLITE_WMS_WMPROXY_MAX_SERVED_REQUEST" and "PassEnv GLITE_PR_TIMEOUT" into glite_wms_wmproxy_httpd.conf
    • In glite_wms_wmproxy_httpd.conf replace FastCgiConfig line with: "FastCgiConfig -restart -restart-delay 5 -idle-timeout 3600 -maxProcesses 25 -maxClassProcesses 20 -minProcesses 2 -listen-queue-depth 200 -gainValue 0.75 -killInterval 240 -updateInterval 240 -singleThreshold 15 -initial-env GLITE_WMS_WMPROXY_MAX_SERVED_REQUEST -initial-env LD_LIBRARY_PATH -initial-env GLITE_LOCATION_VAR -initial-env GLITE_LOCATION_LOG -initial-env GLITE_LOCATION_TMP -initial-env RGMA_HOME initial-env GLITE_SD_VO -initial-env GLITE_SD_PLUGIN -initial-env LCG_GFAL_INFOSYS -initial-env HOSTNAME -initial-env LITE_WMS_WMPROXY_WEIGHTS_UPPER_LIMIT"
    • In the WM section of glite_wms.conf, add CeForwardParameters = { "GlueHostMainMemoryVirtualSize", "GlueHostMainMemoryRAMSize" };
    • copy /opt/glite/etc/lcmaps/lcmaps.db.template to /opt/glite/etc/lcmaps/lcmaps.db
    • /root/WMS-post-install
    • mkdir /data02/var/glite/ice; chown glite:glite /data02/var/glite/ice; ln -s /data02/var/glite/ice/ /var/glite/
    • in /etc/cron.d/glite-wms-wmproxy-purge-proxycache.cron, change glite_wms_wmproxy_purge_proxycache to glite-wms-wmproxy-purge-proxycache
    • mkdir /data02/var/glite/logging; chown glite:glite /data02/var/glite/logging; ln -s /data02/var/glite/logging /var/glite/
    • Add GLITE_WMS_LCGMON_FILE=/var/glite/logging/status.log into /etc/glite/profile.d/glite_setenv.(c)sh

rb126 (experimental gLite WMS 3.1 for Atlas)

  • 2007-04-04: installed hourly cronjob glite-restart for automatic restart of interlogd service.
  • 2007-03-26: alarm NOSPMA has been disabled.
  • 2006-03-06: Add "export GLITE_WMS_ENABLE_BULKMM=true" (resp. "setenv GLITE_WMS_ENABLE_BULKMM true") in /etc/glite/profile.d/glite_setenv.sh (resp. /etc/glite/profile.d/glite_setenv.csh) to replace bulksubmisison with dagless, upgraded glite-wms-manager to 3.1.13 and glite-jdl-api-cpp to 3.1.7.
  • 2007-03-01: Added roles into /opt/glite/etc/glite_wms_wmproxy.gacl as attachment, for 3.1 it must be there
  • 2007-02-21: gLite WMS 3.1 installation and configuration
    • Create file /etc/nospma to avoid SPMA downgrade.
    • Install apt: rpm -ivh /afs/cern.ch/project/linux/cern/slc30X/i386/SL/RPMS/apt-0.5.15cnc6-8.SL.cern.i386.rpm
    • Add 3.1 WMS APT repository in file /etc/apt-sources.list.d/glite31.list: rpm http://lxb2042.cern.ch/gLite/APT/R3.1-RB-pretest rhel30 externals Release3.1 updates
    • Remove the following packages: rpm -e --nodeps c-ares gridsite glite-LB glite-WMSLB
    • Install the following packages: apt-get install c-ares gridsite-shared gridsite-apache
    • Install the middleware metapackage: apt-get install glite-WMSLB
    • Install all the packages related to the middleware: apt-get dist-upgrade
    • Install the new condor RPM: rpm -Uvh extrarpms/condor-6.8.4-linux-x86-rhel3-dynamic-1.i386.rpm
    • Change line 215 of /opt/condor-6.8.4/etc/condor_config to: HOSTALLOW_WRITE = *.cern.ch (Need to verify)
    • Change condor version to 6.8.4 in /opt/glite/etc/config/templates/glite-wms.cfg.xml (line 178).
    • Update the CDB template profile_rb126 by adding the following lines:
                "/software/components/yaim/VOs" = list("dteam","ops","atlas");
                "/software/components/yaim/conf/WMS_HOST" = "rb126.cern.ch";
                "/system/function" = "Experimental gLite WMS 3.1 for Atlas";
                "/system/accounting/name" = "atlas";
    • Update the site-info.def file on rb126: ccm-fetch; ncm-ncd --configure yaim
    • Add the following line in /etc/glite/profile.d/glite_setenv.sh:
if [ -z "$LD_LIBRARY_PATH" ]; then export LD_LIBRARY_PATH=/opt/c-ares/lib; else export LD_LIBRARY_PATH=/opt/c-ares/lib:$LD_LIBRARY_PATH; fi
    • Source the file /etc/glite/profile.d/glite_setenv.sh.
    • Run the configure_node script coming from yaim: /opt/glite/yaim/scripts/configure_node /etc/lcg-quattor-site-info.def glite-WMSLB
    • Modify /opt/glite/etc/init.d/glite-lb-bkserverd to add "--rgmaexport" in the bkserverd startup line (line 57).
    • Add "!" after "if" in the line 98 in /opt/glite/etc/init.d/glite-lb-bkserverd (if ! netstat -an --unix...).
    • Add following lines into /opt/condor-c/local.rb126/condor_config.local:
               NEGOTIATOR_MATCHLIST_CACHING = False

               GRIDMANAGER_TIMEOUT_MULTIPLIER = 3
               SCHEDD_TIMEOUT_MULTIPLIER = 3
               COLLECTOR_TIMEOUT_MULTIPLIER = 3
               C_GAHP_TIMEOUT_MULTIPLIER = 3
               C_GAHP_WORKER_THREAD_TIMEOUT_MULTIPLIER = 3
               TOOL_TIMEOUT_MULTIPLIER = 3

               GLITE_CONDORC_DEBUG_LEVEL = 2
               GLITE_CONDORC_LOG_DIR = /var/tmp
    • Increase GLITE_WMS_QUERY_TIMEOUT from 300 to 480 in /etc/glite/profile.d/glite_setenv.(c)sh.
    • Add the line export GLITE_PR_TIMEOUT=300 (resp. setenv GLITE_PR_TIMEOUT 300) in /etc/glite/profile.d/glite_setenv.sh (resp. /etc/glite/profile.d/glite_setenv.csh).
    • rpm -Uvh /root/extrarpms/glite-wms-manager-3.1.10-1.i386.rpm extrarpms/google-perftools-0.8-1.i386.rpm extrarpms/google-perftools-devel-0.8-1.i386.rpm (to reduce WM daemon memory consumption).
    • rpm -Uvh /root/extrarpms/glite-wms-wmproxy-3.1.8-1_U200702091704.i386.rpm (upgrade to suicidal wmproxy server)
    • Add at the end of the PassEnv section of file /opt/glite/etc/glite_wms_wmproxy_httpd.conf the following two lines:
                PassEnv GLITE_WMS_WMPROXY_MAX_SERVED_REQUEST
                PassEnv GLITE_PR_TIMEOUT
    • In /opt/glite/etc/glite_wms_wmproxy_httpd.conf, replace FastCgiConfig line with: "FastCgiConfig -restart -restart-delay 5 -idle-timeout 3600 -maxProcesses 25 -maxClassProcesses 20 -minProcesses 2 -listen-queue-depth 200 -gainValue 0.75 -killInterval 240 -updateInterval 240 -singleThreshold 15 -initial-env GLITE_WMS_WMPROXY_MAX_SERVED_REQUEST -initial-env LD_LIBRARY_PATH -initial-env GLITE_LOCATION_VAR -initial-env GLITE_LOCATION_LOG -initial-env GLITE_LOCATION_TMP -initial-env RGMA_HOME -initial-env GLITE_SD_VO -initial-env GLITE_SD_PLUGIN -initial-env LCG_GFAL_INFOSYS -initial-env HOSTNAME -initial-env GLITE_WMS_WMPROXY_WEIGHTS_UPPER_LIMIT"
    • In the WorloadManager section of file /opt/glite/etc/glite_wms.conf, add the following lines:
               CeForwardParameters = {
                       "GlueHostMainMemoryVirtualSize",
                       "GlueHostMainMemoryRAMSize"
               };
    • cp /opt/glite/etc/lcmaps/lcmaps.db.template /opt/glite/etc/lcmaps/lcmaps.db
    • /root/WMS-post-install
    • mkdir /data02/var/glite/ice; chown glite:glite /data02/var/glite/ice; ln -s /data02/var/glite/ice/ /var/glite/
    • in /etc/cron.d/glite-wms-wmproxy-purge-proxycache.cron, change glite_wms_wmproxy_purge_proxycache to glite-wms-wmproxy-purge-proxycache.
    • mkdir /data02/var/glite/logging; chown glite:glite /data02/var/glite/logging; ln -s /data02/var/glite/logging /var/glite/
    • Add "export GLITE_WMS_LCGMON_FILE=/var/glite/logging/status.log" (resp. "setenv GLITE_WMS_LCGMON_FILE /var/glite/logging/status.log") into /etc/glite/profile.d/glite_setenv.sh (resp. /etc/glite/profile.d/glite_setenv.csh).
    • Add "export GLITE_WMS_WMPROXY_MAX_SERVED_REQUEST=50" (resp. "setenv GLITE_WMS_WMPROXY_MAX_SERVED_REQUEST 50") into /etc/glite/profile.d/glite_setenv.sh (resp. /etc/glite/profile.d/glite_setenv.csh).
    • Modify the /opt/edg/etc/edg-mkgridmap.conf:
      • remove the ldap lines for VO ops, dteam and atlas.
      • change the .cg and .zp strings by dteamsgm, dteamprd, opssgm, opsprd, atlassgm and atlasprd
      • Regenerate the grid-mapfile file: /opt/edg/sbin/edg-mkgridmap --output=/etc/grid-security/grid-mapfile --safe >> /var/log/edg-mkgridmap.log 2>&1

lxb7026 (experimental gLite LB for CMS + Atlas)

  • 2006-07-26: machine given to GD.
  • 2006-08-08: CAs upgraded to version 1.8-1.
  • 2006-08-22: kernel upgrade.
  • 2006-08-31: Installed and configured with same WMS+LB middleware as rb102 on it, also the configuration is the same as rb102.
  • 2006-09-05: Applied patches, 833, 834, 835 and 836, and restarted services.
  • 2006-11-14: Problem with the configuration of the monitoring (false exceptions triggered like cputemperature_wrong, fanspeed_wrong and voltage3_3_wrong). The problem has been fixed by running spma, but there is stil a problem (exception iss_nologin) due to the buggy version of the script /usr/libexec/SetToDesiredState.gridbdii.
  • 2006-11-27: added option "--super-users-file /opt/glite/etc/LB_query_host" in the startup line of bkserverd, and LB_query_host contains the DN of WMSs which use lxb7026 as the separate LB, restarted bkserverd.
  • 2007-02-01: Updated to 3.1 WMS and LB
    • Unstalled c-ares-1.3.0-2.slc3 and gridsite
    • Added /opt/c-ares/lib/ into LD_LIBRARY_PATH in /etc/glite/profile.d/glite_setenv.*sh since c-ares-1.3.0-1.slc3 is installed as an external package and its location is changed to /opt/c-ares
    • added option "--super-users-file /opt/glite/etc/LB_query_host" in the startup line of bkserverd, and LB_query_host contains the DN of WMSs which use lxb7026 as the separate LB, restarted bkserverd.
    • add GLITE_WMS_LCGMON_FILE=/var/glite/logging/status.log in environment setting scripts and add "--rgmaexport" in bkserverd startup line.
    • /etc/init.d/gLite status reports glite-lb-notif-interlogd not running is not running, however it is running. GLITE_LB_NOTIF_SOCK is not defined in status function, changed it to its real value.
    • Changed user from daemon to glite in /etc/init.d/lcg-mon-job-status since access for daemon denied and mkdir -p /var/glite/logging, chown glite.glite /var/glite/logging
  • 20070216: in /etc/cron.d/glite-wms-wmproxy-purge-proxycache.cron, change glite_wms_wmproxy_purge_proxycache to glite-wms-wmproxy-purge-proxycache
  • 20070122: Increase database size by: alter table short_fields max_rows=1000000000; alter table long_fields max_rows=55000000; alter table states max_rows=9500000; alter table events max_rows=175000000;
  • 20070619: Installed from scratch with 3.0 LB and then upgraded 3.1 check point patch 1167. And configured it with new yaim configuration scripts which are rewritten in bash shell.
  • 20070625: Modifed /opt/glite/etc/init.d/glite-lb-bkserverd to add option "-b 0" to bkserverd startup option, this is to disable transactional database support since a memory leak bug found.

lxb6115 (experimental CE for CMS + Atlas)

  • 2006-08-10: machine under full GD responsability.
  • 2006-08-18: operating system installed (not with quattor).
  • 2006-08-24: Middleware installed and configured, condor version is 6.7.19
    • Change the condor version to 6.7.19 in /opt/glite/etc/config/glite-wms.cfg.xml.
    • Add CONDOR_CONFIG=/opt/condor-6.7.19/etc/condor_config into gLite.def.
    • Add GLITE_LOCATION=/opt/glite/ into condor_config.local
    • CERN modified lcg-info-dynamic-lsf(2.0.9-1) used
    • Modified lsf_submit.sh to fit CERN LSF environment(e.g. add one line: echo "cd .." >> $tmp_file )
  • 2007-01-17: Upgrade condor to 6.8.3
    • Change the condor version to 6.8.3 in /opt/glite/etc/config/glite-wms.cfg.xml
    • Modify line 215 in /opt/condor-c/etc/condor_config by hand
    • Add /opt/c-ares/lib/ into LD_LIBRARY_PATH of environment
    • Modify blah.config and lsf_submit.sh

lxb7283 (Experimental WMS for CMS)

  • 2007-03-26: added root acces to jfrey from Wisconsin to debug Condor problems.
  • 2006-09-05: machine under full GD responsability.
  • 2006-10-16: Upgraded to 3.1 branch with the repositoy, "rpm http://lxb2042.cern.ch/gLite/APT/R3.1-RB-pretest rhel30 externals Release3.1"
    • Installed j2re although j2sdk was installed since glite-WMSLB depends on j2re
    • Added OS apt repository since some packages which needed by WMS were not installed, like boost and httpd
    • Added /opt/c-ares/lib/ into LD_LIBRARY_PATH in /etc/glite/profile.d/glite_setenv.*sh since c-ares-1.3.0-1.slc3 is installed as an external package and its location is changed to /opt/c-ares
    • Added ". /etc/glite/profile.d/glite_setenv.sh" into glite-wms-ns, glite-wms-lm, glite-wms-jc under /opt/glite/etc/init.d/ to get correct environment, specially the LD_LIBRARY_PATH for c-ares-1.3.0-1.slc3
    • Added ". /etc/glite/profile.d/glite_setenv.sh" into /opt/glite/etc/config/scripts/glite-lb-config.py before starting glite-lb-bkindex to get LD_LIBRARY_PATH for c-ares-1.3.0-1.slc3.
    • Installed jre-1_5_0_09-linux-i586.rpm to test java 1.5 version, and keep j2re 1.4 version since name is different and WMSLB depends on j2re, besides changed JAVA_LOCATION in site-info.def to /usr/java/jre1.5.0_09.
  • 2006-10-20: alarm spma_error on lxb7283 (problem with gridsite-shared and gridsite-apache packages). Fixed by updating CDB.
  • 2006-11-17: partition /tmp full. In fact there was a problem with some services. Di restarted all the services and the problem has been solved by itself (see remedy ticket CT382223).
  • 2006-12-01: Upgraded a lot of new tags.
  • 2006-12-06: Applied patch 918 and 919.
  • 2007-01-12: directory tmp almost full on lxb7283. Condor files deleted to fix the problem.
  • 2007-01-15: add GLITE_WMS_LCGMON_FILE=/var/glite/logging/status.log in environment setting scripts and add "--rgmaexport" in bkserverd startup line.
  • 2007-01-26: Reduced WorkerThreads from 20 to 5 in WM section and DagmanMaxPre from 20 to 2 in JC section of /opt/glite/etc/glite_wms.conf
  • 2007-01-31: Upgraded glite-wms-manager from 3.1.8 to 3.1.10 which should reduce the memory usage of WM daemon, and also added "EnableIsmDump = false;" into the WM section of /opt/glite/etc/glite_wms.conf
  • 2007-01-31: added /opt/ and /var/glite in "exclude" option of /etc/cron.daily/slocate.cron
  • 2007-02-01: Increased GLITE_WMS_QUERY_TIMEOUT from 300 to 480 and added GLITE_PR_TIMEOUT=300 in glite_setenv.(c)sh
  • 2007-02-04: Alarm tmp_full triggered due to some huge Condor files. After stopping the job controller, emptying the content of these files and restarting the job controller, two of these files came back again with the same size. I then checked the processes and there was two processes related to the job controller still running:
                [root@lxb7283 tmp]# ps auxwwww | grep condor
                glite     8083  0.0  0.0  5784 1944 ?        S    Feb01   0:30 perl /var/local/condor/spool/cluster363149.ickpt.subproc0
                glite     8068  3.3  0.0  5912 1956 ?        S    Feb01 121:12 perl /var/local/condor/spool/cluster363148.ickpt.subproc0
I killed them with the -TERM signal before restarting the job controller once again. This problem has then been fixed.
  • 20070205: Increased condor timeout parameters like: GRIDMANAGER_TIMEOUT_MULTIPLIER = 3, SCHEDD_TIMEOUT_MULTIPLIER = 3, COLLECTOR_TIMEOUT_MULTIPLIER = 3, C_GAHP_TIMEOUT_MULTIPLIER = 3, C_GAHP_WORKER_THREAD_TIMEOUT_MULTIPLIER = 3, TOOL_TIMEOUT_MULTIPLIER = 3 in /opt/condor-c/local.lxb7283/condor_config.local
  • 20070205: Changed the debug level and log location of condor launcher jobs by GLITE_CONDORC_DEBUG_LEVEL = 2, GLITE_CONDORC_LOG_DIR = /var/tmp in /opt/condor-c/local.lxb7283/condor_config.local
  • 20070209: Upgraded glite-wms-wmproxy package with which wmproxy instances will terminate when the number of served requests reach certain value (default 500). Added GLITE_WMS_WMPROXY_MAX_SERVED_REQUEST into environment setting scripts and PassEnv GLITE_WMS_WMPROXY_MAX_SERVED_REQUEST and FastCgiConfig -restart -restart-delay 5 -idle-timeout 3600 -maxProcesses 25 -maxClassProcesses 20 -minProcesses 2 -listen-queue-depth 200 -gainValue 0.75 -killInterval 240 -updateInterval 240 -singleThreshold 15 -initial-env GLITE_WMS_WMPROXY_MAX_SERVED_REQUEST ..... (all the rest unchanged) into /opt/glite/etc/glite_wms_wmproxy_httpd.conf:
  • 20070215: Reduced GLITE_WMS_WMPROXY_MAX_SERVED_REQUEST from 500 to 250 in environment, and restarted services
  • 20070215: mkdir /var/glite/ice, chown glite:glite /var/glite/ice
  • 20070216: in /etc/cron.d/glite-wms-wmproxy-purge-proxycache.cron, change glite_wms_wmproxy_purge_proxycache to glite-wms-wmproxy-purge-proxycache
  • 20070219: chmod a+x /opt/glite/sbin/glite_wms_wmproxy_load_monitor
  • 2007-03-01: Added roles into /opt/glite/etc/glite_wms_wmproxy.gacl as attachment, for 3.1 it must be there
  • 20070619: Installed from scratch with 3.0 WMS and then upgraded 3.1 check point patch 1167. And configured it with new yaim configuration scripts which are rewritten in bash shell.
  • 20070623: Move /var/glite to /opt/var/glite, and create link /opt/var/glite to /var/glite since the free space under partition /opt is much larger than the one on /var

rb201 (experimental gLite LB 3.0 for CMS and Atlas)

  • 2007-04-19: Middleware upgrade (update 21 for gLite 3.0).
  • 2007-03-14: Middleware upgrade (update 17 for gLite 3.0).
  • 2007-03-01: Upgraded glite-lb-server and glite-lb-logger to version 1.3.12 and 1.2.4 as patch 1052, and renamed LB_query_host to LB-super-users since --super-users-file $GLITE_LOCATION/etc/LB-super-users is enabled in defualt.
  • 2006-11-20: machine under full GD responsibility.

lxb2173 (experimental gLite LB 3.1 for CMS and Atlas)

  • Installation from scratch.

ce104 (CMS + Atlas) -- not used anymore --

  • 2006-08-03: machine under full GD responsability.
  • 2006-08-04: Upgraded to condor 6.7.19 and gLite 3.0.2 compiled against condor 6.7.19
    • Change the condor version to 6.7.19 in /opt/glite/etc/config/glite-wms.cfg.xml.
    • Add CONDOR_CONFIG=/opt/condor-6.7.19/etc/condor_config into gLite.def.
    • Add GLITE_LOCATION=/opt/glite/ into condor_config.local
  • 2006-08-08: CAs upgraded to version 1.8-1. A downgrade of gLite packages has been made by mistake by Yvan during the CAs upgrade. Di solved the problem (resinstallation of apt-get, apt-get upgrade and services restarted).
  • 2006-08-10: go back to FIO responsability.

devel09 (WMS 3.1 (patch 1251) at CNAF on SLC3.0.8 - use devel11 as LB)

devel10 (WMS 3.1 (patch 1251) at CNAF on SLC3.0.8 - use devel12 as LB)

devel11 (LB 3.1 at CNAF - used by devel09)

devel12 (LB 3.1 at CNAF - used by devel10)

Tests

  • CMS tests on gLite WMS
  • Interactive job submission tests done by Nikolai Klopov at CERN: report
  • 2006-10-23: 12000 simple sleep jobs to lxb7283
    • Description: three parallel streams through wmproxy interface plus 3000 jobs in one stream though network server. There are 30 dag jobs per stream to wmproxy interface and 100 node jobs per dag. Each job will simply sleep 10 seconds.
    • Results: 1. One dag failed to submit, the error message is as: Unable to register the job to the service: https://lxb7283.cern.ch:7443/glite_wms_wmproxy_server, probably it is because the machine is too busy to response. 2. Since all jobs in one dag were submitted to the same site, when this site is not in the information system, there are only 20 jobs in "Waiting" status because of "no matching resources found" and other 80 jobs in "Submitted" status. I guess that WMS only try to match the first 20 jobs, if it failed, others of the same dag will not be started, 20 may be because there are 10 planners for each dag job. 3. For some dag jobs, all 100 jobs stay in "Submitted" status. 4. After submitting the stress test, when I try a very simple job list match, it hangs for ever(or it may takes too long), thus the machine must be in very strange status, item 2 and 3 may be caused by this. After another few days, job list match worked fine.
  • 2006-10-30: 12000 simple sleep jobs to lxb7283
    • Description: three parallel streams through wmproxy interface plus 3000 jobs in one stream though network server. There are 30 dag jobs per stream to wmproxy interface and 100 node jobs per dag. Each job will simply sleep 10 seconds.
    • Lemon monitoring pages: 1) CPU load avarage reached 20 when submitting 2) all 4GB memory used, but less than 2% of swap space used.
    • Results: 1) it spent about 4.5 hours for the submission of the 90 dags in 3 three streams. 2) 2 jobs failed to submit because "Unable to register the job to the service: https://lxb7283.cern.ch:7443/glite_wms_wmproxy_server". 29 jobs failed to submit because "LB[Proxy] Error: Resource temporarily unavailable"; 3)Some dag jobs are in "Done (Success)" status, but parts of its nodes are in "Submitted" status. It's true that these jobs registered "Done" event to LB, but the last event is a "RegJob" event and the time order seems not correct. Besides, for dag jobs, certainly they were not submitted via Network Server interface, but in log info, it is shown that the RegJob events are from Network Server as the attachment, this should be a bug. 4) The all the nodes of the dag jobs to one of CE aborted with "hit job shallow retry count (10)", but the dag jobs is still in "Running" status, the job id is https://lxb7283.cern.ch:9000/h4pIbD6Nigsg27UI6kwwpA, I think the status should be turn to something others.

  • 2006-12-14: 12000 simple sleep jobs to lxb7283
    • Description: three parallel streams through wmproxy interface.There are 30 dag jobs per stream to wmproxy interface and 100 node jobs per dag. Each job will simply sleep 10 seconds.
    • Lemon monitoring pages: 1) CPU load avarage occasionally reached more than 15 when submitting 2) all 4GB memory used, but less than 2% of swap space used.
    • Results: 1) it spend about 7 hours for the submission. 2) 1 dag failed to submit due to "SOAP version mismatch or invalid SOAP message", 60 dag jobs failed to submit because LB[Proxy] Error: Resource temporarily unavailable. 3) Some dag jobs in Running status for ever, but all of their sub nodes are either in "Aborted" status or "Done (Success)" or "Done(Failed)" 4) Some dag jobs are in "Done (Success)", but one of their sub nodes is still in "Done (Failed)" or "Running" status 5) the problem, which dag jobs are in "Done (Success)" status, but parts of its nodes are in "Submitted" status, disappear since the the patch we applied is to fix it.

Documentations

  • Customizations done to a gLite WMS at CNAF dedicated to CMS tests:
http://grid-it.cnaf.infn.it/certification/?INFN_Grid_Certification_Testbed:Patching_the_WMS%2BLB

  • Andrea's presentation (19 July 2006):
http://indico.cern.ch/getFile.py/access?contribId=2&resId=2&materialId=slides&confId=4669

  • Di's and Yvan's presentation (16 November 2006): Slides.

Some bugs

  • [[https://savannah.cern.ch/bugs/?21909][#21909]: glite-wms-check-daemons.cron needs to redirect stderr to /dev/null
Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf SectionMeeting-20061116.pdf r1 manage 439.0 K 2006-11-16 - 10:38 YvanCalas slides yvan
Unknown file formatlocal condor_config.local r2 r1 manage 5.9 K 2006-09-01 - 10:07 DiQing rb102-condor_config.local
Unknown file formatlocal-31 condor_config.local-31 r1 manage 6.6 K 2007-02-22 - 18:01 DiQing 3.1 condor_config.local
XMLxml glite-lb.cfg.xml r2 r1 manage 2.1 K 2006-09-01 - 10:08 DiQing rb102-glite-lb.cfg.xml
XMLxml glite-wms.cfg.xml r2 r1 manage 8.1 K 2006-09-01 - 10:08 DiQing rb102-glite-wms.cfg.xml
Unknown file formatconf glite_wms.conf r2 r1 manage 3.7 K 2006-09-01 - 10:07 DiQing rb102-glite_wms.conf
Unknown file formatconf-31 glite_wms.conf-31 r1 manage 6.1 K 2007-02-22 - 18:02 DiQing 3.1 glite_wms.conf
Unknown file formatgacl glite_wms_wmproxy.gacl r1 manage 3.0 K 2007-03-01 - 20:32 DiQing gacl plus roles
Unknown file formatconf-31 glite_wms_wmproxy_httpd.conf-31 r1 manage 9.9 K 2007-02-22 - 18:03 DiQing 3.1 glite_wms_wmproxy_httpd.conf
Texttxt wms-interactive-test-1.txt r1 manage 18.5 K 2007-02-16 - 16:04 AndreasUnterkircher interactive job submission tests
Edit | Attach | Watch | Print version | History: r186 < r185 < r184 < r183 < r182 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r186 - 2007-11-13 - AlessioGianelle
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    LCG All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback