r8 - 08 Jan 2014 - 15:17:15 - IlijaVukoticYou are here: TWiki >  Admins Web > MinutesJan082014

MinutesJan082014

Introduction

Minutes of the Facilities Integration Program meeting, January 8, 2013
  • Previous meetings and background : IntegrationProgram
  • Coordinates: Wednesdays, 1:00pm Eastern
  • Dial Toll-Free Number: 866-740-1260 (U.S. & Canada)
  • Your access code: 2913843

Attending

  • Meeting attendees: Michael, Rob, Saul, Myuko UTA, Fred, Wei, Torre, Patrick, Mark, Armen, Kaushik, Bob, John Brunelle, Hiro, Horst, Ilija, Alden, Dave Lesny
  • Apologies:
  • Guests: Will Stecker-Kellog, Joe Metzger

Integration program update (Rob, Michael)

  • Special meetings
    • Tuesday (12 noon Central, bi-weekly - convened by Armen) : Data management
    • Tuesday (2pm Central, bi-weekly - convened by Shawn): North American Throughput meetings
    • Monday (10 am Central, bi-weekly - convened by Wei or Rob): ATLAS Federated Xrootd
  • Upcoming related meetings:
  • For reference:
  • Program notes:
    • last week(s)
      • New Google docs for reported capacities, see CapacitySummary.
        • Review for: accuracy, retirements, and consistency with OIM
        • Update with upgrades as deployed this quarter.
      • Arizona meeting. 13 registrants.
      • New information in SiteCertificationP27 - Rucio renaming column information.
    • this week
      • Multicore resources. Two Tier2 sites have not yet provided MCORE resources. Will provide this.
      • LHCONE - proposal was made to increase transatlantic capacity using the new 100g link, integrate with existing VRFs. Will allow to run pilot applications which require the bandwidth. Use FAX as demonstrator. Count on a four week period.
      • Queue consolidation - how to handle Panda queues in the future. Discussed at SW at CERN - want to follow-up. Artificial separation between PROD and ANALY queues. Requires lots of coding changes in Panda. Wei: How would requirements be communicated to the job scheduler? Kaushik: will require changes in autopyfactory, but its possible. Michael: goals are to reduce fragmentation at the site level, while allowing priority adjustments at the Panda level. Kaushik: not yet on the Panda to-do list. (Most work is now on DEFT and JEDI). Kaushik: will get to this this year, will provide a proposal. BNL can be used as an integration test site.
      • Management of LOCALGROUPDISK; Kaushik and Armen are discussion.
      • Observed issue with single streams > 10Gbps, as reported by Brian Tierney at the Tucson meeting. Would like to work with the ESnet outreach team to address the issue. Ho

Dynamic Scheduling for Multi-Core using Condor (Will, BNL)

  • Multicore jobs are supported using Partitionable / Dynamic slot configuration.
    • Every farm node has 1 partitionable slot containing all of that machine's resources
    • Jobs match that partitionable slot and "slice-off" their resource requirements (request_cpus, etc...) to form "dynamic" sub-slots.
    • Supports running multicore and single-core and high-memory jobs all with the same configuration
  • Condor's Group-Quotas allow dynamic partitioning of our "flat" farm into a hierarchy of queues with different quotas and idle-resource sharing between them
    • Works transparently with multicore jobs, where a job's usage of it's group's "quota" is defined by the number of CPUs it asks for
    • Could have multi- and single-core jobs integrated in the same group, but for now we have a separate multicore group
  • Potential fragmentation is handled by filling our farm depth-first rather than (the previous) breadth-first; done with a NEGOTIATOR_POST_JOB_RANK expression
    • Works sufficiently for now, but we may end up running the condor_defrag daemon if the need arises
  • Configuration is very easy, just a few lines in our standard config-file:
    SLOT_TYPE_1 = 100%
    
    NUM_SLOTS = 1
    NUM_SLOTS_TYPE_1 = 1
    SLOT_TYPE_1_PARTITIONABLE = True
    
    SlotWeight = Cpus 

Managing LOCALGROUPDISK at all sites (Kaushik)

previously
  • LOCALGROUPDISK - first draft from Armen, Kaushik reviewing.
  • Beyond pledge production storage
  • Tools will be needed for policy enforcement.
  • Rucio features for quota management not available yet.
  • Hard limits versus soft limits. Enforcement.
  • Will present the plan in Arizona.

this meeting

  • Rescheduled for next data management meeting, next week or next.

Reports on program-funded network upgrade activities

AGLT2

last meeting
  • Ordered Juniper EX9208 (100 Gbps on a channel) for both UM and MSU. Getting them installed now.
  • Will be retargeting some of the tier2 funds to complete the circuits between sites.
  • LR optics being purchased ($1200 per transceiver at the Junipers).
  • Need to get a 40g line card for the MX router on campus.
  • Probably a month away before 40g or 80g connectivity to CC NIE.
  • UM-MSU routing will be only 40g.
  • Likely end of November.
previous meeting
  • LR optics from ColorChip have been shipped. (for UM)
  • Still waiting on info to connect to the CC NIE router
  • Also, final budget info
  • Hope to get this by Friday.
previous meeting, 11/13
  • Juniper connect at 2x40g to cluster in place; 100g in place to Chicago
  • New wavelength for Mylar
  • MSU router to be procured.
this meeting, (1/8/14)
  • MSU funding issues getting resolved; will soon order parts for the Juniper.
  • At UM, all parts have been ordered 40g line card for CC-NIE router.

MWT2

last meeting(s)
  • Timeframe - end of November
  • Juniper in place in at UC, connected to SciDMZ
  • IU - still at 2x10g
  • UIUC- network configuration change next wed, move campus cluster consolidation switch to 100g.
last meeting (11/27/13)
  • IU: Network: All our 10Gb hosts, including storage servers, are attached to one of two 4810 switches, each with a 4x10Gb uplink. The 1Gb hosts are on the 6248 switch stack, which is connected to our 100Gb switch via a 2x10Gb uplink. The two pieces we are missing are the VLT connections between the 4810 switches, and moving the 6248 switch stack to uplink to the 4810s. We attempted to move the 6248 to the 4810 when we moved the 10Gb hosts, but found the combination of the trunk to the 6248 and the VLT caused routing issues. We also found that the VLT was causing routing asymmetries for the directly-connected 10Gb hosts. We have the VLT disabled while we investigate that issue. We plan to role out a new test config on Mon Dec 2, and to iterate on that through the week until we are in the final configuration.
  • Illinois: testing of 40 Gbps next week. There have been some checksum errors that are being investigated. 100Gb wave to 710S LSD: Fiber cleaned but not enough testing at load to know if it fixed the low level checksum issue. Working with I2 to try and bring up as a 40Gb link for testing. Currently we have a 10Gb link. Plans for a go-nogo on 100Gb are a week from this friday Second wave via west route (Peoria and up I55) to 600W did not get funding via cc-nie grant. Other funded sources being looked into. On campus: Campus Cluster consolidation switch is now directly connected to CARNE router (100Gb Juniper) Current connection is a 2x10Gb LAG. The equipment for an 8x10Gb LAG is in place, however there are not enough fibers between ACB and node-1 (where CARNE lives) for 8 connections. Spare fibers not passing tests. Could pull more fibers but the conduits are full. Options being looking into. We can use working fibers and add to LAG without any downtime So I believe right now we are limited to 10Gb to 710S LSD (uplink to Chicago), but the limit soon will be the 2x10Gb LAG (CCS to CARNE - 40Gb to Chicago) which will be raised as the LAG is increased. In two weeks we might have 100Gb.
  • UC: 40 Gbps to server room. Will start transitioning hosts next week to new VLANs.

this meeting (1/8/14)

  • MWT2_UC: First phase (40g) of network upgrade complete: Juniper EX9208 installed, trunked to campus Science DMZ at 4x10g. New VLAN with most nodes on the public network. Each MWT2 internal switch (Cisco 6509, two Dell PC6248 stacks, a Dell PC8034F stack) are trunked to the Juniper router with 4x10g LAGs. Found it necessary to purchase an additional PC6248 with optics to connect to Juniper at 40g for the second stack for new worker nodes (this will be trunked at 20g until the parts arrive). Brocade line card, optics and fiber for remaining 6x10g to complete MWT2_UC to SciDMZ at 100g expected by end of January. The graph below is the main SciDMZ Router at 1155 (the campus edge). It illustrates the total traffic flowing through the router. The different colors represent the physical interfaces the traffic is entering and leaving on. In this specific graph. Positive values represent traffic entering the router and negative is traffic leaving the router. Since most traffic merely transits this router, both inbound/outbound should be roughly the same amount. The traffic is dominated by ATLAS at present.
  • SciDMZ-MWT2-Graph7Jan2014.png: SciDMZ-MWT2-Graph7Jan2014.png
  • MWT2_IU network upgrades are complete, MWT2_IU connected at 8x10g to wide area. Summary from Fred Luehring: Using DOE money provided by BNL, the outbound connectivity of the IU component of the MWT2 was further upgraded to better advantage of the existing 100 Gbps connection between the Indianapolis machine room and OmniPoP peering site at 600 W Chicago Ave in downtown Chicago. Over the course of the quarter all 6 storage farms were connected to S4810s. These 10 Gbps connections used Twinax cables connected to SFP+ ports. Also the gatekeeper nodes (iut2-grid10 and iut2-grid11) were moved to being connected directly to the S4810s again using 10 Gbps Twinax to SFP+ connections. Each S4810 had two PerfSONAR nodes (one latency and one for throughput) connected to it for monitoring purposes. The PerfSONAR nodes also used 10 Gbps Twinax SFP+ connections. Over the course of the quarter the VLT was disabled for several weeks to facilitate moving 6248 switch stack to being connected to the S4810s using eight 10 Gbps Twinax cables connected to SFP+ ports at both ends. The VLT was re-enabled after the first 6 connections were in place. On December 23 a final SFP+ card for on of the 6248 switches allowed the final 2 connections to be put it place between the 6248s and the S4810s. The eight uplink fibers serving as the uplink from the S4810 to the Brocade chassis were split between four Brocade line cards to avoid a CAM memory overflow problem. The final set of connections between the Dell 6248 stack, Dell S4810 VLT, and Brocade NI-MLX-10GX8 cards was designed for good redundancy. Both the sets of 8 links are in LAGs to bind them into in a single virtual 8 x 10 Gbps connection. This was accomplished with $65k RBT funding:
    Dell SFP+ Modules for 6248s (12/3):                     321.82
    Dell PerfSONAR nodes and cables (9/30):               7,774.02
    Dell second S4810 and fibers (9/16):                 12,558.60
    Dell first S4810, optics, and cables (8/13):         14,562.49
    Matrix Integration Brocade line card & optics (8/8): 21,093.74
    FiberDyne optical fibers (8/8)                          385.12
                                                        =========
    Total:                                               56,695.79
    
  • MWT2_IU network diagram:
    screenshot_1050.png
  • MWT2_UIUC upgrade status:
    • OmniPOP ports fee (split with Campus IT Services)
      • Hardware to enable 8 x 10G links from ICC to WAN
      • MPC3e Carrier Card
      • Juniper MIC 10x10GE CFP Module
      • 16 SPF+ 10GE pluggable transceivers
      • Slim Tpas, LC, fiber
      • CITES Stock for Misc Fiber connectors
    • Currently ICC connected at 2x10g. Will add 4x10g (total to 6x10g) and then will add another 2x10g for total 8x10g.
  • WAN peering in Chicago
    • The current UC to IU path is as follows: MWT2_UC<--40G-->!UCSciDMZ<--100G-->CIC OmniPop<--100G-->MREN<--10G-->Indiana GigaPop<--100G-->IU<--80G-->MWT2_IU
      • This path is simply a migration of the old 10G path to the new 100G path and is not ideal. We intend to move this peering to the following path to avoid the 10G bottleneck at MREN. The final path will be: MWT2_UC<--100G-->!UCSciDMZ<--100G-->CIC OmniPop(710)<--200G-->CIC OmniPop(600W)<--100G-->Indiana Gigapop<--100G-->IU<--80G-->MWT2_IU
      • Status of 200G crosstown OmniPoP trunk:
        The CIC(710)--200G--CIC(600W) is in place.
        (On one of the CIC switches)
        Physical interface: ae0, Enabled, Physical link is Up
         Interface index: 128, SNMP ifIndex: 592
         Description: L2 BB sw2.600WChicag to sw2.star 200G LAG | CIC-600WCHICAG-STAR-LAG-01580
         Link-level type: Flexible-Ethernet, MTU: 9192, Speed: 200Gbps,
    • Today, the UC-UIUC path is via LHCONE and looks like the following: UCSciDMZ<--100G-->CIC OmniPoP<--100G-->MREN<--10G-->LHCONE. UIUC will move to 710 OmniPoP.
    • ESnet is connected to the new CIC OmniPoP switch at 100G. Ryan Hardin (UC) working on getting a peering to them via UC's 100G. It will probably be a couple weeks. Some of the delay here is CIC is working to retire the old Force10 switch and they would like to keep the add/moves/changes to a minimum until that work is completed.
    • BNL: Ryan will work with BNL and ESnet to schedule the 100G-100G UC-BNL peering. Presume we'd use ESnet at OmniPoP? as before but we need to work out the details. Presently its via I2:
      [rwg@uct2-int ~]$ traceroute nettest10g.usatlas.bnl.gov
      traceroute to nettest10g.usatlas.bnl.gov (192.12.15.25), 30 hops max, 40 byte packets
       1  scidmz-mwt2.mwt2.uchicago.edu (192.170.226.1)  0.445 ms  0.591 ms  0.749 ms
       2  192.170.224.77 (192.170.224.77)  0.152 ms  0.182 ms  0.200 ms
       3  xe-0-3-3.2010.rtr.chic.net.internet2.edu (64.57.30.132)  8.346 ms  8.471 ms  8.478 ms
       4  et-10-0-0.307.rtr.clev.net.internet2.edu (198.71.45.37)  18.005 ms  17.998 ms  17.987 ms
       5  xe-0-3-0.2001.rtr.newy.net.internet2.edu (64.57.30.225)  35.327 ms  35.319 ms  34.990 ms
       6  64.57.30.229 (64.57.30.229)  34.041 ms  34.129 ms bnl-lhc1-esnet.es.net (198.124.80.6)  35.475 ms
       7  bnl-lhc1-esnet.es.net (198.124.80.6)  35.522 ms nettest10g.usatlas.bnl.gov (192.12.15.25)  63.969 ms bnl-lhc1-esnet.es.net (198.124.80.6)  34.548 ms

SWT2-UTA

last meeting(s)
  • Replacing 6248 backbone to Z9000 as central switch, plus additional satellite switches connected to the central switch, likely dell 8132s.
  • Might even put compute nodes into 8132Fs (5 - 6) at 10g. Has a QSFP module for uplinks.
  • Waiting for quotes from Dell
  • Michael: should look at per-port cost when considering compute nodes
  • Early December timeframe
  • 100g from campus - still no definite plans

last meeting (10/30/13)

  • Waiting for another set of quotes from Dell.
  • No news on 100g from campus; likely will be 10g to and from campus, though LEARN route will change.
  • Not sure what the prognosis is going to be for 100g. Kaushik has had discussions with OIT and networking management. There are 2x10g links at the moment.

last meeting (11/13/13)

  • Will get Dell quotes into purchasing this week; this is for the internal networking, close to storage.
  • Kaushik: we still have to meet with the new network manager at UTA.

previous meeting (11/27/13)

  • Had a long series of meetings last week with new director of networking. Much better understanding of the UTA networking roadmap. LEARN and UT system research networks. Problem is now coordinating among the different groups.
  • Right now there are multiple 10g links; two 100g links are coming soon. CIO is about to sign for this.
  • Provisioning has started for the campus. Will need to make sure we're plugged into it. Need to make sure SWT2 is as close to edge router as possible. #1 priority. Will create DMZ. Now problem current exceeding 8Gbps.
  • Logical diagram of WAN and LAN networks?
  • Michael: interested in the 2x100g beyond campus (e.g. to Internet2). How is LEARN connected?
  • OU: 20-40g coming. Will produce a diagram.

this meeting (1/8/14)

  • Orders have been placed. Replacing stacks of 6248s. Waiting on delivery date.
  • Waiting on second round of funds.

FY13 Procurements - Compute Server Subcommittee (Bob)

last meeting
  • AdHocComputeServerWG
  • SLAC: PO was sent to Dell, but now pulled back.
  • AGLT2:
  • NET2: have a request for quote to Dell for 38 nodes. Option for C6200s.
  • SWT2: no updates
  • MWT2: 48 R620 with Ivybridge - POs have gone out to Dell. 17 compute nodes.

Previous meeting: (11/13/13)

  • AGLT2: have two quotes for R620s with differing memory. Some equipment money will go into networking; probably purchase 11-14 nodes.
  • NET2: quotes just arrived from Dell. Will likely go for the C6000s. Will submit immediately.
  • SWT2: putting together a package with Dell. Timing: have funds at OU; but not at UTA.
  • MWT2: 48 nodes

previous meeting: (11/27/13)

  • AGLT2:
  • MWT2:16 new servers for the UIUC ICC. Four servers are already running. 12 more coming early December.
  • NET2: Placed an order for 42 nodes. Not sure about delivery. Expect after Jan 1. Have not decided whether these will be BU or HU.
  • SWT2: Still waiting for next round of funding. Expect January or Feb.

this meeting: (1/8/14)

  • AGLT2: three R620s arrived, another 16 ordered. Will be replacing PE1950s at UM.
  • MWT2: See FabricUpgradeP27#MWT2. 48 nodes at UC are racked, half are powered. Need to benchmark nodes, network cable to top-of-rack switches, complete power cabling. Expect to have online next week. UIUC: 12 of 16 nodes arrived and are online.
  • NET2: 42 C6220s nodes have arrived. Will install within the next few days. Install all on BU side.
  • SWT2: Still waiting for next round of funding. Expect January or Feb.

Integration program issues

Reviewing LHCONE connectivity for the US ATLAS Facility (Shawn)

last meeting(s)
  • June 1 is the milestone date to get all sites on.
  • BNL DONE, AGLT2 DONE, 2 sites from MWT2 DONE
  • SLAC DONE

notes:

  • Updates?
  • Shawn - Mike O'Conner has been putting together a document with best practices. Will have examples on how to route specific subnets that are announced on LHCONE.
  • Three configurations: 1. PBR (policy based routing). 2. Providing a dedicated routing instance. Virtual router for LHCONE subnets. 3) Physical routers for gateway for LHCONE subnets.
  • NET2: have not been pushing it, but will get ball rolling again - will contact Mike O'Conner and provide feedback.
  • OU: there was a problem at MANLAN which has been fixed. Direct replacement from BNL to OU. Will start on LHCONE next.

previous meeting

  • NET2: status unsure: either waiting on instructions from Mike O'Conner (unless there have been direct communications with Chuck). Will ramp things up.
  • OU: status: waiting for a large latency issue to be resolved from Internet2, then reestablish the BNL link. Believes throughput input matrix has improved (a packet loss problem seems to be resolved). Timeline unknown. Will ping existing tickets.
  • UTA: will need to talk with network staff this week. Attempting to advertise only a portion of the campus. Could PBR be implemented properly. After visit can provide update.

previous meeting (8/14/13)

  • Updates?
  • Saul sent a note to Mike O'Conner - no answer. There are management changes at Holyoke. Would like a set of instructions to drive progress.
  • OU: will check the link.
  • UTA - still need to get a hold of network staff. A new manager coming online. Will see about implementing PBR. Update next.

previous meeting (8/21/13)

  • Updates
  • OU - network problems were fixed. Then turned direct link back on. Then perfsonar issues, then resolved. Expect to have a either a Tier 2 or the OSCER site done within a few.
  • BU and Holyoke. Put the network engineers in touch. Still unknown when it will happen. Have not been able to extract a date to do it.
  • UTA - no progress.

previous meeting (9/4/13)

  • Updates?
  • UTA: meeting with new network director schedule this Friday or next week. Back on the page.

this meeting (9/18/13)

  • Updates?
  • UTA - no update; getting on the new director's manager. Before the next meeting.
  • BU & HU - made some headway with Chuck and Mike O'Conner. NOX at Holyoke to be at 100g in 6 months. (Michael: from LHCONE operations call, NOX will extend to MANLAN, initially 10g link on short notice; sounded promising.)
  • OU - OU network folks think we can be on LHCONE by Oct 1

previous meeting (10/16/13)

  • Updates?
  • UTA - had meeting with new director of campus network computing, and LEARN representative. Possible separate routing instance. Will meet with them tomorrow morning.
  • OU - new switch being purchased, that also sets a separate routing instance, so as to separate traffic.
  • BU - no news. HU will not join LHCONE? Michael: raises question of NET2 architecture. Saul: HU is connected by 2x10g links; discussing it with James.

previous meeting (10/30/13)

  • Updates?
  • UTA (Mark): There is second 2x 10g link into campus, a UT research network. Has the link on campus. Trying to decide where the traffic should route.
  • OU (Horst):
  • BU (Saul): News from Chuck was it would be very expensive (but hearing things second hand.

previous meeting (11/13/13)

  • Updates?
  • UTA (Patrick). Kaushik, previous attempt to peer to LHCONE failed, had to back out of it. Have had conversations with UTA and LEARN - now have options, there are additional paths. Estimate - next couple of weeks.
  • OU (Horst):
    From Matt Runion: The fiber terminations are done. We are still awaiting approval for a couple of connections within the 4PP datacenter. I've also begun coordination with the OSCER folks as to a date for installation and cutover for the new switch. Unfortunately, with SC2013, cutover is unlikely until after Thanksgiving. We're tentatively shooting for Wed the 4th or Wed the 11th for installation and cutover. (Wednesdays there is a built-in maintenance window for OSCER). Following that, some configuration/coordination with !OneNet, and finally vlan provisioning and and router configuration. Realistically, factoring in holiday leave, end of semester, etc, etc, I'm guessing it will be sometime in January before we have packets flowing in and out of !LHCONE.
  • LU (Horst): Have to talk to OneNet and LU Networking folks.
  • BU (Saul): Nothing definitive, but met with people at Holyoke who manage it. Spoke with Leo Donnelly. Not yet ready to work technically. Michael - is the BU and BNL dedicated circuit still used? Perhaps use it to connect NET2 to MANLAN, and hook into the VRF.
  • HU (John): Same data center as BU. Just getting started with it.

SHOULD we ask for a dedicated meeting with experts?

  • Yes, Shawn will convene a meeting between phone/video meeting for network experts.

this meeting (11/27/13)

  • UTA: campus has ordered Cisco switches, two weeks ago. 4500x switches. Expect to complete LHCONE peering before the holidays. Will this include the two Z9000's? No. Dell 4810.
  • OU: nothing new. Got info from Mat Ryun, for Shawn's document. Don't expect until after the new year. Expect, right after the beginning of the year, definitely. LU: will discuss following the new year.
  • BU: nothing new. Will have meeting on December 5 - will meet with Holyoke networking people. Next step for LHCONE? Expect nothing will happen until January.
  • Shall we convene a general Tier2-LHCONE meeting? Yes.

this meeting (1/8/14)

  • UTA: need to meet with campus infrastructure people. Will schedule a meeting.
  • OU: no updates. Expect to hear something soon.
  • BU: no update. December 5 meeting: all agreed to use dedicated circuit to BNL, but no schedule. Then discussion about how 100g will get to the room (via Albany? Move NOX there?). Some time in 2014 will have 100g. Michael: can we make it work now?
  • Michael: At LHCONE-LHCOPN Pasadena meeting it was agree it was timely to discuss networking requirements of the LHC at 10-11 Feb at CERN. Idea was to bring network providers together to see how the infrastructure develops in the future. Perhaps merge the infrastructure? A refined usage of the LHCOPN allowing Tier2s? Comprehensive discussion planned.

Operations overview: Production and Analysis (Kaushik)

  • Production reference:
  • last meeting(s):
    • Production sporadic. How long will it continue? The next big sample is mc13, "relatively early in 2014". A good time for downtimes.
    • Multi-core queues: how long will this happen?
      • SWT2 and NET2 will add MCORE queues next week. Good check of the system.
      • BNL - will be completely dynamic in the future using Condor.
      • Still have a large number of jobs waiting in analysis queues.
  • this meeting:
    • Multi-core decision not yet made, but may ask sites to split 50/50.
    • mc14 will be multi-core only. 1-2 weeks.

Shift Operations (Mark)

  • last week: Operations summary:
    Summary from the weekly ADCoS meeting:
    CERN closed for the holidays
    
    1)  12/26: SLACXRD - production jobs failing with stage-in errors ("lsm-get failed (201): 201 Copy command failed:Last server error 10000 ('')"). As of 12/30 no additional 
    errors over the previous two days, so the issue was apparently resolved. https://ggus.eu/ws/ticket_info.php?ticket=99953 was closed, eLog 47521.
    2)  12/27: SWT2_CPB - site was draining, very few production jobs were running.  Batch system/maui were in a bad state due to a couple of WN's that had their memory 
    exhausted, and then became "black holes" for pilots.  Batch system was cleared up, and production resumed.
    3)  12/29: BNL - Michael reported on a dCache SE issue at the site. Experts were notified, and the problem was fixed after a few hours. 
    https://atlas-logbook.cern.ch/elog/ATLAS+Computer+Operations+Logbook/47504.
    4)  12/31: ADC Weekly meeting:
    CERN closed for the holidays
    
    Follow-ups from earlier reports:
    
    (i)  12/12: WISC DDM deletion errors ("atlas07.cs.wisc.edu    [SE][srmRm][] httpg://atlas07.cs.wisc.edu:8443/srm/v2/server: CGSI-gSOAP running on voatlas311.cern.ch 
    reports Error reading token data header: Connection reset by peer"). https://ggus.eu/ws/ticket_info.php?ticket=99731, eLog 47329.
    Update 12/30: deletion errors continue - no response to the ticket from the site. eLog 47514.
    

  • this week: Operations summary:
    Summary from the weekly ADCoS meeting:
    https://indico.cern.ch/getFile.py/access?contribId=0&resId=0&materialId=2&confId=286609 (Dec.25-Jan.1)
    http://www-hep.uta.edu/~sosebee/ADCoS/ADCoS-WeeklyReport-January1st-7th2014.html (Jan.1-Jan.7)
    
    1)  1/7: MWT2 - frontier squid at the site was shown as down in the monitor (had been in this state for several weeks) - update from Dave: Several weeks ago, the assigned 
    IP space for MWT2 nodes at UChicago was changed. It appears that the DNS registration of the newly assigned IP for uct2-grid1.uchicago.edu was overlooked. The node 
    has been up and operational for internal use, but DNS to sites outside UChicago does not reflect its new address. https://ggus.eu/ws/ticket_info.php?ticket=100091 
    in-progress, eLog 47574.
    2)  1/7: ADC Weekly meeting:
    https://indico.cern.ch/conferenceDisplay.py?confId=289674
    
    Follow-ups from earlier reports:
    
    (i)  12/12: WISC DDM deletion errors ("atlas07.cs.wisc.edu    [SE][srmRm][] httpg://atlas07.cs.wisc.edu:8443/srm/v2/server: CGSI-gSOAP running on voatlas311.cern.ch 
    reports Error reading token data header: Connection reset by peer"). https://ggus.eu/ws/ticket_info.php?ticket=99731, eLog 47329.
    Update 12/30: deletion errors continue - no response to the ticket from the site. eLog 47514.
    

Data Management and Storage Validation (Armen)

  • Reference
  • last meetings(s):
    • Not much to report. There was a deletion problem with Lucille - understood now.
    • Request to DDM operations about reducing primary data at Tier 2s. There was some cleanup, but then filled again.
    • 500 TB at BNL that was related to a RAC request, "Extra" category. Armen will make a proposal.
    • Another 600 TB at BNL in "default" - status unknown, a difficult category to figure out.
    • USERDISK cleanup is scheduled for the end of next week.
    • Zero secondary datasets at BNL - meaning PD2P? is shutdown at BNL.
    • Is there any hope of using DATADISK more effectively, such that we could reduce usable capacity but replicate data by a factor of two. Kaushik and Michael will get in touch with Borut.
    • USERDISK cleanup is in progress.
    • There is a well-known issue associated with two users that submitted the same jobs twice. DESY sites affected, as well as US sites. There is some data which should be declared obsolete. Sarah provided a list of data files to be declared lost. DQ2-ops owns the issue; meanwhile Hiro's error checker continues to send notifications every hour.
  • this meeting:
    • Armen: DATADISK issue at BNL. Default type of data must be defined by ADC and wasn't. Then "Extra" needs to be defined. (Archive tags: primary, secondary, default, extra). BNL is 98% non-deleted. All new tasks stopped being assigned to the US cloud. Kaushik: raised it with ADC, which evidently has no plan.
    • Hiro: can move some files to tape.
    • Kaushik: ADC is responsible for managing DATADISK.
    • Michael: We need to act now, bring this up with computing management and ADC. There was no reaction from ADC at yesterday's meeting.
    • Armen: the main problem is primary dataset.
    • Kaushik will send a draft email to send to Borut and Richard.
    • Hiro: what is the future of Pandamover? Kaushik: plan was to abandon it with Rucio.
    • Armen: cleaned up 250 TB of LOCALGROUPDISK at BNL. 30 TB at MWT2.
    • USERDISK cleanup ...

DDM Operations (Hiro)

Throughput and Networking (Shawn)

Federated Xrootd deployment in the US (Wei, Ilija)

FAX status & reference

last meeting (11/27/13)

  • Wei: still working with German sites, deploying Rucio N2N, a few minor issues to resolve.
  • Deployment document updated.
  • Ilija: stability issue - dcache-xrootd door stopped responding. Still trying to understand the cause. Working with Asoka to get a user script for optimal redirector location. Working with Valeri to get fax failover monitoring improved, requiring. Few week at earliest.
  • UTA stability issues. Wei gave Patrick some suggestions. A week of stability. Memory allocation on the environment variable, since RHEL6. Configuration change in xrootd configuration. Stress test?
  • Wei: prefers a small stress test on Rucio-converted sites.
  • Ilija - will be stress-testing MWT2. Also, there will be a change in notification for fax endpoint problems. A new test dataset has been distributed to all sites.

this meeting (1/8/14)

  • BNL has enabled Rucio for its N2N, restarted the service.
  • All other US sites have updated and are stably running.
  • Real time mailing proved to be rather infrequent.
  • Still a lot of sites mainly FR and DE cloud do not deploy Rucio enabled N2N? .
  • localSetupFAX deployed - significantly simplifies end-user's life.
  • Still debugging cmsd instabilities at few sites.

Site news and issues (all sites)

  • T1:
    • last meeting(s): 6400 max analysis slots at BNL. Working on next gen storage deployment -- to replace 5 year old equipment. 2.5PB to replace. Needs to be done quickly (3-6 months). Tier 1 will go down Dec 16,17. All services will be affected. New Arista switch will be fully integrated. New version of spanning-tree routing in the LAN, a new algorithm. Day long intervention. Will update dCache to version 2.6, primarily for SHA-2. And new version of Condor.
    • this meeting: New Arista switch is providing many internal 100g links. New dcache version running.

  • AGLT2:
    • last meeting(s):
    • this meeting: Upgraded dcache to 2.6.19. Working on MCORE configuration.

  • NET2:
    • last meeting(s): Retired two racks of 1TB drives, went down 100 TB. SGE configuration. Downtime on Dec 8 for power upgrade. Next week will do all SHA-2 upgrades.
    • this week: installing new nodes, bringing up new MCORE queues. Introduce new SGE queues.

  • MWT2:
    • last meeting(s): Major task was the dCache 2.6.15 upgrade. Upgraded CVMFS to 2.1.15. Went to Condor 8.0.4. Completely SHA-2 compliant. (Hiro: can you test.)
    • this meeting: static MCORE queues now; will try out partitionable slots. dCache server testing on GPFS as a locality cache.

  • SWT2 (UTA):
    • last meeting(s): Machine room issue at CPB - AC issue today. Had a machine that was problematic - resolved now. SHA-2 compliance done at both sites. Would like to see John's tests.
    • this meeting: MCORE, LHCONE. Resurrecting some old UTA_SWT2 servers. CPB issues. Rucio.

  • SWT2 (OU, OSCER):
    • last meeting(s): SHA-2 compliant except for Xrootd.
    • this meeting: OSCER MCORE

  • SWT2 (LU):
    • last meeting(s): Fully functional and operational and active.
    • this meeting:

  • WT2:
    • last meeting(s): SHA-2 compliance. Some issue with the new gridftp server - seeing some performance issues, but without error messages? Will out a full day outage on Dec 2 to remove Thumpers.
    • this meeting: several Thors are having issues, no redundancy. Setting them in read-only mode. CPU purchases: 60 blades in receiving.

AOB

last meeting this meeting
  • Need to check v29 spreadsheet for SL6.


-- RobertGardner - 07 Jan 2014

About This Site

Please note that this site is a content mirror of the BNL US ATLAS TWiki. To edit the content of this page, click the Edit this page button at the top of the page and log in with your US ATLAS computing account name and password.


Attachments


png SciDMZ-MWT2-Graph7Jan2014.png (161.1K) | RobertGardner, 08 Jan 2014 - 10:53 |
pdf MWT2-IU-Network-v11.pdf (91.1K) | RobertGardner, 08 Jan 2014 - 10:59 |
png screenshot_1050.png (250.8K) | RobertGardner, 08 Jan 2014 - 11:00 |
 
Powered by TWiki
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback