LTA Development Workshop

Europe/Amsterdam
ASTRON

ASTRON

Oude Hoogeveensedijk 4 7991 PD Dwingeloo The Netherlands
John Swinbank (ASTRON)
Description

Motivation

LOFAR is now well into its second decade, and has an enviable scientific and technical track record: the LOFAR team and partners have published ground-breaking papers; developed new and remarkable data reduction and processing techniques; and securely archived and distributed tens of petabytes of data.

As we prepare for the LOFAR2.0 system to enter operations within a couple of years, it is appropriate for us to take this opportunity to reflect on lessons learned from the past and from other facilities, and to plan for the future. How will we scale up to meet the increased data volumes from LOFAR2.0, and potential future extensions to LOFAR beyond that? What can we learn from the technologies under development for the SKA Regional Centres Network, or in use at other facilities both in radio astronomy (ASKAP, MeerKAT, ...) and beyond (Rubin Observatory, CERN, ...)?

We anticipate a future with more — and more complex — next-to-data pipeline processing, of interactive data analysis through tools like Jupyter notebooks, of FAIR data and interoperable archives, of an increasing number of partners stepping up to join the LTA system, and, of course, of ever increasing data volumes.

At this meeting, we aim to look multiple years into the future. Rather than focusing how to address short-term bugs or operational issues, we will focus on the long-term roadmap for the LTA, looking to define common research questions and implementation decisions to ensure LOFAR's archive and data processing system remains cutting-edge for many years to come.

This meeting will focus on data archiving, curation, and processing as well as service monitoring and reliability; we'll set issues around networking and connectivity aside for now.

Confirmed Speakers

  • Maciej Brzezniak (PSNC)
  • Michiel van Haarlem (LOFAR ERIC)
  • Yan Grange (ASTRON)
  • Fabio Hernandez (Vera C. Rubin Observatory / CC-IN2P3)
  • Hanno Holties (ASTRON)
  • Marcel Loose (ASTRON)
  • Cristina Manzano (FZJ)
  • Lodewijk Nauta (SURF)
  • André Offringa (ASTRON)
  • Raymond Oonk (SURF)
  • Roberto Pizzo (ASTRON)
  • Oleg Smirnov (Rhodes University)
  • Holger Stiele (FZJ)
  • Onno Zweers (SURF)

Format

The meeting will consist of a number of invited and contributed talks, interspersed with facilitated discussion sessions. The total number of participants will be restricted to around 20 in order to encourage plenty of interaction.

Logistics

Travel

The meeting will take place at ASTRON headquarters in Dwingeloo, the Netherlands; refer to our website for travel information.

We can provide help with arranging transportation between ASTRON and Hoogeveen Railway Station. Please reach out using the contact details provided below.

Accommodation

A block booking has been made at Landhotel de Borken, which is within walking distance of the meeting venue and has bikes available to hire. Note number of rooms available is limited; they will be allocated on a first-come, first-served basis. The hotel will hold the block for us until 4 September.

Details on how to book at the special rate will be provided upon successful registration for the meeting.

Remote Participation

If you can't join us at ASTRON, you can participate in the meeting using Microsoft Teams. You'll need separate connection details for each day:

Wednesday

 

Thursday

 

Friday

Registration
Registration Form
Participants
  • Arpad Miskolczi
  • Cristina Knapic
  • Cristina Manzano
  • Emanuela Orru
  • Fabio Hernandez
  • Giacomo Cannizzaro
  • Haili Hu
  • Holger Stiele
  • Jean-Pierre Vilotte
  • Karin Wessel
  • MACIEJ BRZEZNIAK
  • Marcin Pospieszny
  • Natalie Danezi
  • Roberto Pizzo
  • Yan Grange
  • +6
    • 12:00
      Lunch (provided)
    • 1
      Welcome & Logistics
      Speaker: John Swinbank (ASTRON)
    • 2
      LOFAR ERIC: Now & Next

      An overview of the current status of LOFAR ERIC. What is the plan for LOFAR2.0, and what happens after that?

      Speaker: Michiel van Haarlem (ASTRON)
    • 3
      LOFAR Data Services: now and next
      • Current status & successes
      • Future ambition
      • Data policies
      • Resource challenges
      Speaker: Roberto Pizzo
    • 4
      A Vision for Future LOFAR Data Services
      Speaker: John Swinbank (ASTRON)
    • 5
      Technical Overview of the LTA
      Speaker: Hanno Holties
    • 15:00
      Break
    • 6
      Lessons Learned
      Speaker: Emanuela Orru (Astron)
    • 7
      Brainstorm: What are the biggest challenges?

      Given everything that we've heard today, what are the major challenges that face the LTA in the future? We'll use these to help seed the agenda over the next couple of days.

      Miro Board: https://edu.nl/h6g3f

      Speaker: John Swinbank (ASTRON)
    • 8
      Site Update: PSNC
      Speaker: Maciej Brzezniak (PSNC)
    • 9
      Site Update: FZJ
      Speaker: Cristina Manzano (FZJ)
    • 10
      Site Update: SURF
      Speaker: Lodewijk Nauta (SURF)
    • 10:30
      Break
    • 11
      The SKA Regional Centre Network
      Speaker: Raymond Oonk (SURF)
    • 12
      The Vera C. Rubin Observatory
      Speaker: Fabio Hernandez (CNRS-IN2P3 Computing Centre / Vera C. Rubin Observatory)
    • 13
      Discussion: Learning together; Federation, EOSC, and Open Science
      1. Are there things we heard that we should be collaborating / learning together on?

      2. All of the sites have been showing their own fantastic infrastructure. How do we give LOFAR users a uniform experience while also taking advantage of those capabilities?

      3. Should we see LTA sites as “storage” or “data management”?

      4. We've already heard that the LTA is moving towards a federated AAI system, and the LOFAR ERIC Data Policy (https://www.lofar.eu/lofar2-0-documentation/) affirms a commitment “to adhere to the principles of Open Science, Open Access, and FAIR data”. What else is needed? Can we leverage the EOSC EU Node? Other EOSC services?

    • 12:30
      Lunch (provided)
    • 14
      Token authentication in dCache

      dCache supports authentication with Macaroon tokens and with OIDC tokens. I'd like to explain what they are, how they differ, how to use them, how to configure them in dCache, and how to troubleshoot them.

      Speaker: Onno Zweers
    • 15
      Towards more structured long-term data archiving

      At the moment, the long-term archive sites do not have knowledge about the (scientific) content of the observations that are stored. If we know, e.g. which observations belong from an observational and astrophysical point of view together, it may be possible to archive these data on the same tape, minimising the costs in time and energy to retrieve them. As with the upgrade to LOFAR 2.0 it is also foreseen to save advanced data products it will be necessary to store the data in a comprehensible way so that it is clear which data sets of different levels belong together. Based on this information, we would like to build a knowledge tree that in the end would e.g. allow to suggest to observers other archived observation that cover similar objects. Such a structure will also help to provide data according to the FAIR principles and facilitate reuse in the age of multi-wavelength astronomy.

      Speaker: Holger Stiele
    • 16
      LOFAR science pipelines: technologies and future plans
      Speaker: Marcel Loose
    • 17
      ASTRON's ATDB-based pipeline execution system
      Speaker: Hanno Holties
    • 14:50
      Break
    • 18
      The Stimela Workflow Management Framework
      Speaker: Oleg Smirnov
    • 19
      Discussion: distributed processing within the LTA
      • What technical options are there — frameworks or other services that the data centres support or suggest and that we can build on?
      • What should the interaction between ASTRON and the LTA sites look like as we roll out process in Jülich & Poznan? Are there skills, people, expertise at the sites we can look to for help?
      • What sort of coupling is needed across LTA sites — do we need to e.g. transfer data stored at one site to process at another?
    • 20
      Discussion: Communication Needs

      We heard yesterday that LTA sites would like:

      • More / earlier involvement in design decisions and future plans
      • More information on staging requests
      • More insight into how data is going to be used

      What communications and data management plans do we need to put in place to ensure these needs are appropriately met?

    • 21
      Group Dinner

      Onder de Eiken, Dwingeloo
      https://onderdeeiken.nl

    • 22
      Future Science with the LOFAR Observatory
      Speaker: Andre Offringa (ASTRON)
    • 23
      Data discovery and (interactive) analysis work in SRCNet and at ASTRON
      • Science Platform Vision Document
      • Science Gateway
      • Canfar, Azimuth, etc.
      • ADEX, ESAP, etc
      Speaker: Yan Grange (ASTRON)
    • 24
      Discussion: A science platform for the LTA?
    • 10:30
      Break
    • 25
      Discussion: Conclusions and Next Steps

      To consider:

      • Brainstorm: what technologies do the LTA sites have access to that we should be exploiting? What are the topics of common interest that we can work together on in the future?
      • A roadmap for LTA evolution
      • Was this meeting useful? Should we have more like it? A different format?
      • Topics arising from discussions throughout the meeting
    • 12:30
      Lunch (provided)