ANSYS Fluent - MetaCentrum

ANSYS HPC computing effective source allocation

Petr Koňas

1/163

© 2012 SVS FEM spol. s r.o.

November 25, 2013

Content • Who is SVS FEM? • What is ANSYS? • ANSYS in Academic World • Comparison ANSYS CFX and OpenFOAM • HPC Configuration – How to select HW resources for your job? • ANSYS Cloud Tools • Solution of Large models (Superelements)

• Benchmarks of large clusters (Metacentrum, IT4I, UV2000)

2/163


November 25, 2013

Consulting

Education Training

Engineering Projects

ANSYS Provider

Customization Tools Provider

SVS FEM is the original ANSYS Channel partner for Czech Rep. and Slovak Rep. since 1992 and partner of CADFEM GmbH.

HW Solutions Provider

HPC Cluster designer

ISO 9001:2009

3/163


November 25, 2013

Ing. Ludvík Láníček, Ph.D. doc. Ing. Petr Koňas, Ph.D. Ing. Zuzana Kodajková CFX, FLUENT, Tgrid, ICEM CFD, IcePack, Polyflow

Ing. László Iván, Ph.D. Ing. Zdeněk Čada Ing. Karel Kubáček Ing. Zuzana Kodajková Ing. Oldřich Veverka Ing. Jan Schwangmaier doc. Ing. Petr Koňas, Ph.D.

Ing. Tibor Bachorec, Ph.D.

HFSS, Maxwell, Optimetrics Mechanical, MAPDL

Ing. Ludvík Láníček, Ph.D. Ing. Tibor Bchorec, Ph.D. Ing. László Iván, Ph.D. doc. Ing. Petr Koňas, Ph.D. Ing. Karel Kubáček Ing. Zdeněk Čada Ing. Zuzana KodajkováCFX, FLUENT, Tgrid,

Ing. Radek Řídký Ing. Miroslav Popovič Prof. Ing. Jaroslav Buchar, DrSc.

Multiphysics

ICEM CFD, IcePack, Polyflow, Mechanical, MAPDL

Workbench

Ing. László Iván, Ph.D. Ing. Miroslav Popovič doc. Ing. Petr Koňas, Ph.D.

LS=DYNA, AUTODYN

DXplorer, OptiSlang ACT, SDK 4/163


November 25, 2013

doc. Ing. Petr Koňas, Ph.D. Ing. László Iván, Ph.D. Ing. Zdeněk Čada Ing. Miroslav Popovič

Školení SVS FEM ANSYS Mechanical APDL 1 ANSYS Mechanical APDL 2 ANSYS Mechanical APDL - Dynamics ANSYS Mechanical APDL - Nonlinear 1 ANSYS Mechanical APDL - Nonlinear 2 ANSYS Mechanical APDL - Nonlinear 3 ANSYS Mechanical APDL - Nonlinear 4 ANSYS Mechanical APDL - Optimalization ANSYS Mechanical APDL - Programming ANSYS Mechanical APDL - Emag NF ANSYS Mechanical APDL - Emag HF ANSYS CFX ANSYS FLUENT ANSYS Icepak ANSYS from FLUENT to CFX ANSYS TurboGrid ANSYS BladeModeler ANSYS CFD-Post ANSYS ICEM CFD Gambit to DesignModeler & ANSYS Meshing transition ANSYS CFX - Spalování a Radiace ANSYS FLUENT - Spalování a Radiace ANSYS CFX FSI ANSYS FLUENT FSI ANSYS Workbench Mechanical 1 / ANSYS DesignSpace ANSYS Workbench Mechanical 2 ANSYS Workbench Mechanical - Dynamics ANSYS Workbench Mechanical - Nonlinear 1 ANSYS Workbench Mechanical - Nonlinear 2 ANSYS Workbench Mechanical - Nonlinear 3 ANSYS Workbench Mechanical - Thermal ANSYS Workbench Mechanical - Programming 5/163


November 25, 2013

ANSYS Workbench Mechancial - Fatigue Modul ANSYS Workbench Mechanical - nCode DesignLife ANSYS Workbench Mechanical - Emag NF ANSYS Workbench Geometry (ANSYS DesignModeler) ANSYS Workbench Geometry (ANSYS SpaceClaim Direct Modeler) ANSYS Workbench DesignXplorer ANSYS Workbench Explicit Dynamics Elektromagnetické simulace - Maxwell Elektromechanické simulace - Maxwell, Simplorer, RMxprt ANSYS AUTODYN LS-DYNA Školení OptiSlang

http://www.svsfem.cz/content/plán-školení-pro-rok-2013

We can solve Extreme Models

140M cells We are able to formulate, compute and evaluate the most complex and the largest models…

280M cells

6/163


November 25, 2013

We can build infrastructure for HPC • Our Workstation won in official benchmarks of Fluent as workstation with the best power per core and the best scaling.

• Our workstations are able to solve models with 111M cells on 2 - 8 cores.

7/163


November 25, 2013

Pro podporu našich uživatelů i po stránce HW stala se SVS FEM s.r.o. oficiálním partnerem firem . Pro každý ze SW produktů u nás zakoupených dokážeme navrhnout optimální hardware a nabízíme dodání celé instalace na klíč.

Hardware od SVS FEM SVS FEM je váš specialista na CAE Hardware. Od přípravy přes konfiguraci, instalaci až po vlastní užívání.

• • • • •

Pracovní stanice Notebooky Servery Clustery Komponenty a příslušenství

SVS FEM je Váš partner ve výpočtech

Proč zvolit právě naši konfiguraci HW a instalaci ANSYSu? Sestavujeme pouze takový hardware, který je kompatibilní s produkty ANSYS a který dosahuje optimálního výkonu dle Vašich potřeb Provádíme kontrolu kvality dodávaných PC vůči stabilitě ANSYSu Instalujeme požadovaný operační systém i balíky ANSYSu včetně nejnovějších aktualizací. Provedeme tunning profilu testovacího uživatele. Testujeme/aktualizujeme ovladače BIOSu a provádíme jejich tunning pro maximální výkon ANSYSu Konfigurujeme Hardware dle vašich požadavků: RAID disků, firmware disků, obslužné rutiny, konfigurace/tunning síťových adapterů, automatický backup Konfigurujeme plně MPI (Platform, Open, Intel) pro maximální výkon při distribuovaném výpočtu u jednotlivých aplikací ANSYS Naši konfiguraci dodáváme včetně podrobného auditu. Výpis profilu uživatele Test kompatibility ANSYS kritérií MPI test Vytvoření geometrie, vysíťování a provedení řešení na jednoduché úloze Středně těžké až extrémní CFX/Fluent/Mechanical benchmarky Intenzivní HW benchmarky Konfigurace RSM, test úloh přes RSM Verifikace RAID polí, simulace selhání a následná obnova Výsledky všech testů Vám zůstanou k dispozici na předávaném HW. Support (hotlin/webex/zásah na místě) v případě HW či SW problémů. Webex pro první krůčky v naší instalaci. 8/163


November 25, 2013

What is ANSYS? Sofisticated engineering tool for solution of complex physical tasks and...

9/163


November 25, 2013

What is ANSYS? ...complex geometry of the real problems...

10/163


November 25, 2013

11/163 © 2012 SVS © FEM 2012 spol. SVS s r.o. FEM spol. November s r.o. 25, November 2013 25, 2013












In Addition To Multiphysics, Data Mapping Can Apply Measured/Experimental Results To Complex Surfaces

23/163


November 25, 2013

Connects to Design Environment in Multiple Ways File Reader • PTC: Creo Parametric • Siemens: NX, Solid Edge • Dassault: CATIA, Solidworks • Autodesk: Inventor, AutoCAD • Cadence: Allegro, APD, Virtuoso • Synopsys: Encore • Zuken: CR5000 • Mentor: Boardstation, Expedition, PADS • Neutral Files: Parasolid, SAT, STEP, IGES, ODB++ Bidirectional Associative Link • PTC: Creo Parametric, Creo Direct Modeling • Siemens: NX, Solid Edge • Dassault: CATIA, Solidworks • Autodesk: Inventor 24/163


November 25, 2013

Parametric Simulation • Today’s Engineering Applications Create Parameters in Many forms: • • •

Dimensions and Instances Attributes and Names Properties

• Parametric capabilities enable rapid updates of models for new designs and requirements scenarios. 25/163


November 25, 2013

ANSYS Workbench Makes Parametric Simulation Part of a Standard Workflow

User Select Parameters

26/163


November 25, 2013

Workbench Collects Parameters from All Project Applications

Results that Evaluate Trade-off Require Just a Few More Mouse Clicks

Highly Adaptable and Customizable • ANSYS Customization Suite • Expose custom calculations and results in a native environment • Attractive solution for automation and democratization of simulation • Available at 14.5 Example: Exposing MAPDL acoustics functionality in Mechanical

27/163


November 25, 2013

28/163


November 25, 2013

Architecture Provides a Unique Solution to Connect Solver Data and More… R 14.0

PRODUCTS

R15.0

“ASIM” – Common 3D Environment Fluids

Strut.

Emag

… PRESENTATION

TECHNOLOGY

CUSTOMIZATION APPLICATION SCRIPTING OPTIMIZATION

OPTIMIZATION

REMOTE SOLUTIONS

REMOTE SOLUTIONS

PROJECT MGMT

PROJECT MGMT

PARAMETERIZATION

PARAMETERIZATION

29/163


November 25, 2013

SYSTEMS & HIERARCHY PROJECT DATA APPLICATION DATA COLLABORATION

DATA

PROJECT DATA APPLICATION DATA COLLABORATION

APPLICATION

EXPRESSIONS

“Expanded” Workbench

REPORTING

ANSYS Academic - examples Problem Due to the complexities of bone cells, the variables affecting bone growth are not easily identified and understood.

Solution The Bone Mechanobiology Group at the National University of Ireland Galway uses ANSYS Multiphysics to provide a deeper understanding of the mechanics of bone cells. The new data provided from simulations allowed prediction of the stress and strain signals that simulate bone growth, as well as developing the next generation of cures for bone diseases.

“Our ANSYS simulations are now being used to inform the next generation of cures for bone diseases, such as osteoporosis.”

Biomechanics Research Centre (BMEC), National University of Ireland Galway 30/163


November 25, 2013

ANSYS Academic - examples Problem Patients who receive a cardiovascular stent may sometimes have post-treatment complications. Developments in healthcare engineering have the potential to decrease risks of post-operative disease and improve overall success rate.

Solution University of Pittsburgh students used ANSYS CFD tools to simulate and analyze the complex interactions between the deployment of a cardiovascular stent, the perturbations of blood flow, altered mass transfer and the risk of post-operative diseases.

“Because of the flexibility of ANSYS tools, developing and using computational models covering a wide range of problems no longer requires considerable efforts or highly trained personnel.”

University of Pittsburgh 31/163


November 25, 2013

ANSYS Academic - examples Problem Discover a method to treat and eliminate cancer.

Solution A group at Duke University is working towards a noninvasive approach to treat bladder cancer with the help of ANSYS HFSS and Fluent.

Result They have developed a heated antenna that was able to deliver chemotherapeutics directly to the bladder without burning the skin. Currently, optimization of non-invasive heating tests on human bladders is ongoing.

By integrating ANSYS simulations at an early stage, it was possible to optimize the heat applicator without repetitive experimental testing. In the end, just a single prototype was needed and it performed as expected, resulting in a design efficiency of 100%.

32/163


November 25, 2013

Duke University, Hyperthermia Group

ANSYS Academic - examples Problem Understand blood flow patterns before and after surgery to show whether a surgery has been successful.

Solution A research group at Linkoping University used ANSYS CFX to accurately capture the turbulent scales of blood flow through an aortic coarctation before and after surgery. They determined that a decrease in turbulent levels after the surgery which meant that the surgery was successful and that the workload of the heart decreased. Both quantitative and qualitative results agreed very well with MRI measurements.

The simulations provided an unprecedented detail of the flow, which is useful on both a clinical level when planning interventions and treatments, and on an engineering level in terms of understanding the flow features and transition to turbulence.

Linkoping University 33/163


November 25, 2013

ANSYS Academic - examples Problem As a newborn’s brain grows, the skull must expand rapidly to accommodate it. In some cases, the bone plates of the infant’s skull close before this growth is complete which can lead to developmental difficulties. Surgeries (osteotomies ) are required but surgeons need to trade off between more cuts guaranteeing proper brain expansion and fewer cuts which mean less risk to the patient.

Solution Use ANSYS structural mechanics software to help determine the placement and number of cuts required during surgery,

Result Without simulation, doctors probably would have performed unnecessary additional osteotomies to ensure there was enough room for brain expansion

The simulation results helped the surgeon to prepare better for the operation; the procedure itself was much faster, and it was easier on the infant because of the reduced number of osteotomies. Operations were successful on each of 20 patients, and the children are all doing well.

34/163


November 25, 2013

Medical University of Silesia

ANSYS Academic - examples Problem In heavily populated cities in Greece, using certain building materials and techniques in wall construction can help improve human thermal comfort.

Solution Using ANSYS CFX to test various building materials, Democritus University of Thrace proposed that materials such as green roofs and water surfaces should be considered.

“ANSYS CFX solves coupled environmental and thermal flows even for large scale geometries where conventional models fail to succeed.”

35/163


November 25, 2013

Democritus University of Thrace

ANSYS Academic - examples Problem Racing yachts can move faster when sail design contributes to reduced drag in upwind conditions.

Solution The University of Ljubljana studied pressure distribution on upwind sails using ANSYS CFX, applying turbulence models to compare computed and experimental results.

“Simulation enabled the team to collect data for reducing mainsail drag. By shortening analysis time, we could study more design variables, which led to an optimized design with maximum performance.”

Laboratory for Aeronautics University of Ljubljana 36/163


November 25, 2013

ANSYS Academic - examples Problem To successfully identify the stresses for a carbon-fiber–reinforced mountain bike frame for GHOST Bikes GmbH.

Solution Used ANSYS Composite PrepPost software to simplify analysis of the carbon-fiber reinforced polymer (CFRP) bicycle components.

Result • Stiffness and resistance characteristics of the bicycle components met design requirements including European standards. • ANSYS Composite PrePost within the ANSYS Workbench environment significantly improved design efficiency. • Compared to typical trial-and-error development methods used in the bicycle industry, the number of cost- and time-intensive physical prototypes was greatly reduced.

ANSYS Composite PrepPost software, integrated within the ANSYS Workbench environment, takes advantage of outstanding features and solver technologies from ANSYS. This technology substantially simplifies analysis of CFRP structures using innovative modeling and analysis capabilities.

Chemnitz University 37/163


November 25, 2013

ANSYS Academic - examples Problem The tennis equipment market is highly competitive, so manufacturers must continually innovate, producing advanced equipment within very short time frames.

Solution Sheffield Hallam University used ANSYS LS-DYNA to simulate a tennis ball impacting a racket. The model can be used as an engineering tool to accurately predict the effect of changing specific parameters, including structural rigidity or shot style.

“The simulation model created with ANSYS software will reduce the time and cost of the R&D process. It can also help to develop tennis equipment that is specifically designed for the end user.”

Sheffield Hallam University 38/163


November 25, 2013

ANSYS Academic - examples Problem Plymouth University students produced a concept design for a heat exchanger capable of heating domestic water to 40˚C from boat engine exhaust gases with minimal pressure losses and power requirements.

Solution Students applied ANSYS DesignModeler, ANSYS Workbench and ANSYS CFD, which predicted water outlet temperature within 8 percent of theoretical values, providing time- and cost-effective analysis for the client.

“Through simulation, potential design improvements highlighted at an early state to improve heat exchanger performance.”

University of Plymouth 39/163


November 25, 2013

ANSYS Academic - examples Problem For hurricane prediction models, unresolved physics, especially near the air-sea interface, are the weakest components.

Solution Nova Southeastern University used ANSYS Fluent to simulate the process of disruption of the air–sea interface under hurricane conditions. The results of the study justify the extension of existing theory of wind–wave generation to hurricane conditions by including the effect of two-phase environment on ultragravity waves. This project has been supported by the National Ocean Partnership Program (via NSU subcontract to the University of Rhode Island) and by the Gulf of Mexico Research Initiative (via NSU subcontract to the University of Miami/CARTHE).

“Simulation has played a significant role in the dramatic improvements that HEAD has achieved in the performance and durability of its tennis rackets over the past decade.”

40/163


November 25, 2013

Oceanographic Center Nova Southeastern University

ANSYS Academic - examples Problem Gas turbine combustors produce turbulent combustion creating a strong reliance on experimental measurements and expensive setup costs.

Solution Cranfield University used ANSYS Fluent to simulate turbulent combustion in gas turbine combustors to provide a better understanding of the complex flow phenomena involved and predict quantities such as velocity, temperature, combustion products and pollutants with high fidelity and at a relatively low cost.

“The results obtained from ANSYS Fluent were used to calibrate simpler in-house combustion performance and emission models.”

Cranfield University 41/163


November 25, 2013

ANSYS Academic - examples Problem As electrical energy requirements increase, companies look at new options to provide energy to customers through improved infrastructure.

Solution Researchers at Lucerne University used ANSYS Mechanical to test, improve and develop more reliable and practical common overhead lines.

By using ANSYS simulations, researchers were able to better understand the complexity of common overhead lines to improve the technology..

42/163


November 25, 2013

Lucerne University of Applied Sciences and Arts

ANSYS Academic - examples Problem Interconnect reliability is crucial to the reliability of integrated circuit that consists of billions of transistors interconnected to form a circuit. Stress in the interconnect lines during fabrication can form voids and create an open circuit, decreasing the reliability of the integrated circuit.

Solution Nanyang Technological University used ANSYS multiphysics modeling to understand why the voids are caused and can fine tune the fabrication process to prolong the lifetime of an integrated circuit.

Multiphysics modeling clearly explains the mechanism of the voiding.

43/163


November 25, 2013

Nanyang Technological University

ANSYS Academic - examples Problem Study the Heat Island Index of a piece of land

Solution The simulation, conducted in Fluent, took a 2-D mesh of a landscape design of a plot of residential buildings. The mesh was imported into Gambit and the whole plot was generated.

Result The simulation showed the Heat Island Index was reduced by the greenery on the plot of land.

ANSYS Fluent and Gambit was used to study the Heat Island Index for a plot of land. The study concluded that greenery reduced the Index number.

Carnegie Mellon University 44/163


November 25, 2013

ANSYS Academic - examples Problem Hydro-electric plants must operate efficiently and economically. Unstable water flow produces a vortex rope that increases pressure fluctuations and reduces turbine efficiency.

Solution Using ANSYS Fluent, Pennsylvania State University analyzed the complex phenomena occurring in the draft tube. This is enabling the team to investigate control techniques to prevent vortex rope formation and to improve draft tube performance.

“By using simulation, detailed features of the flow can be studied that were impossible to capture using experiments.”

45/163


November 25, 2013

Penn State Hydropower Research Program

ANSYS Academic - examples Problem To calibrate the “weather station” for the Phoenix Mars Lander so that lander itself does not affect the readings. During the mission to quickly adjust the data to aid in the next day’s mission.

Solution Calibrate instruments pre-flight using fluid dynamics then during the mission use CFD and HPC to provide rapid turnaround to guide data gathering for the next day’s mission.

Result Data gathering proved successful in both phases and this gave time for additional simulations to be performed to help explain certain phenomena found in the raw data.

Parallel processing and the multi-domain scheme in ANSYS CFX combined with AMD’s multi-core architecture enabled simulations to be completed within the timeframe for decision making.

University of Alberta 46/163


November 25, 2013

ANSYS Academic - examples Problem Researchers at the University of Palermo needed to design a new cross flow turbine.

Solution ANSYS CFX was used in a two phase fluid (water and air) and a rotor-stator domain simulations. The efficiencies of the new turbine design was determined with the simulations.

ANSYS simulations provided researchers the ability to design, test and improve the efficiency of the cross flow turbine prior to the prototype stage.

47/163


November 25, 2013

Università degli Studi di Palermo

ANSYS Academic - examples Problem Wing design is normally tested one iteration at a time and can take many months to optimize a design.

Solution Piaggio Aero Industries teamed with the University of Rome Tor Vergatta to design a new optimization method that generates a single mesh and morphs it to any new geometry using ANSYS DesignXplorer, ANSYS ICEM CFD and RBF Morph.

Result Researchers were able to evaluate the robustness of the various wing designs to determine the ones that delivered consistent performance as design parameters were varied.

With ANSYS software, the design optimization took a couple of weeks, less than a tenth of the time required to optimize the design using conventional methods.

University of Rome Tor Vergata and Piaggio Aero Industries 48/163


November 25, 2013

ANSYS Academic - examples Problem When designing a stadium, builders must consider many factors – two of the overall considerations are spectator comfort/safety and how the stadium could affect play.

Solution Conduct 3-D studies of stadium designs using ANSYS CFD software to determine the ideal architectural designs to limit wind flow and wind-driven rain.

Result Planning for these venues can now incorporate wind-flow patterns and wind-driven rain to optimize the stadium design, and include cost saving techniques.

Simulation results can be used to improve the design of stadiums as well as to diagnose and correct problems with existing stadiums – such as using special paint to protect seats that frequently get wet, to reduce maintenance costs.

49/163


Courtesy van Hooff & Blocken. November 25, 2013

Eindhoven University of Technology

ANSYS Academic - examples Problem Students at the University of Nottingham discovered damage from an unknown source to the engine piston.

Solution With the help of ANSYS Fluent, students discovered that soot deposition on the cylinder liner and entrainment into the engine’s oil correlate to oil starvation and damage to the engine piston.

“With ANSYS Fluent, detailed visualization of complex diesel combustion shows the interdependence between the soot entrainment process and the in-cylinder gas motion, the location of combustion and evolution of the soot cloud.”

University of Nottingham 50/163


November 25, 2013

ANSYS Academic - examples Problem As oil prices soar, consumers and governments are demanding improved vehicle gas efficiency. Even minor improvements can help manufacturers to improve mpg while meeting industry standards and requirements.

Solution The University of Magdeburg used ANSYS CFD to determine unsteady turbulent flows around a car’s sideview mirrors. The software’s built-in models and userdefined functions — including consideration of a film flow on the surface of investigated geometry — enabled the detailed analysis.

“ANSYS high-quality mesh provides high accuracy and fast convergence compared to the fully unstructured mesh.”

University of Magdeburg 51/163


November 25, 2013

ANSYS Academic - examples Problem Manufacture less expensive biofuels by introducing microbubbles of CO2-rich gasses into the bioreactor.

Solution Use ANSYS Fluent to help develop an oscillator that delivers microbubbles of appropriate diameter to enhance algae growth.

Result With the 50-fold increase in mass transfer rate afforded by this oscillator design, CO2 dispersal is accelerated in the bioreactor and should enhance algae growth rate by a factor of 10. The experiments showed an 18 percent reduction in the energy required for bubble production compared to conventional fine bubbles.

The team predicts that engineering simulation will play an even greater role in the coming design of a commercial-scale bioreactor, when it will become critical to optimize all components of the design to minimize capital expenses.

University of Sheffield 52/163


November 25, 2013

53/163


November 25, 2013

54/163


November 25, 2013

55/163


November 25, 2013

56/163


November 25, 2013

57/163


November 25, 2013

58/163


November 25, 2013

59/163


November 25, 2013

60/163


November 25, 2013

Realizace HPC výpočtů • RSM – Remote Solve Manager ...implementace, uživatelské rozhraní, konfigurace, integrace plánovačů PBS/LSF, monitoring...

• Specifická konfigurace samostatných komponent ... Mechanical APDL, Fluent a CFX, aktivace GPU...

• ANSYS Cloud ...NICE DCV, EngineFrame, Vcollab, EKM...

61/163


November 25, 2013

14.5.7 ANSYS Structural & Fluid Dynamics Academic Portfolio Product/Family ACADEMIC ASSOCIATE

Task variants available

ANSYS Academic Associate

1,5

ANSYS Academic Associate CFD

1,5

ANSYS Academic Associate HPC

1

ACADEMIC RESEARCH ANSYS Academic Research

1,5,25

ANSYS Academic Research Mechanical

1,5,25

ANSYS Academic Research CFD

1,5,25

ANSYS Academic Research Autodyn

1,5,25

ANSYS Academic Research Electronics Thermal

1,5,25

ANSYS Academic Research Offshore/Marine ANSYS Academic Research Polyflow

1,5 1,5,25

ANSYS Academic Research HPC

1

ANSYS Academic Research LS-DYNA

25

ANSYS Academic Research LS-DYNA HPC

1

ACADEMIC TEACHING ANSYS Academic Teaching Advanced

5,25,50

ANSYS Academic Teaching Introductory

5,25,50

ANSYS Academic Teaching Mechanical

5,25,50

ANSYS Academic Teaching CFD

5,25,50

ACADEMIC TOOLBOX

62/163

ANSYS Academic Meshing Tools

1,5,25

ANSYS Academic CFD Turbo Tools

1,5,25

ANSYS Academic Fuel Cell Tools

1,5,25


November 25, 2013

Core Academic Product Levels Increased scope regarding legal terms of use Decreasing Price Increasing capability

Teaching

Research

Academic Toolbox (Add-on Products)

63/163


November 25, 2013

Associate

License Borrow • • • • • • • • •

64/163

Available for ANSYS Academic Teaching “Structural & Fluid Dynamics” products only. Allows one or more tasks from a multiple task license to be borrowed. n-1 tasks may be borrowed from an n task license Maximum borrow duration is 1 week. Early borrow return is enabled. MCAD connections will have the borrow capability. The “unlimited” DesignSpace capability does NOT have the borrow capability. Borrow is turned OFF by default. It is turned on by request, is free & requires an End User certification form & additional language on the license form.


November 25, 2013

Workbench System Level Coupling •For ANSYS Academic products, system coupling is available for the following license combinations:

• 2 or more tasks of ANSYS Academic Research, Associate, Teaching Intro, Teaching Advanced, – User must switch to non shared mode under Preference settings. – 2 tasks are consumed for each Mechanical-Fluent system coupling.

• 1 task of ANSYS Academic Research CFD & 1 task of ANSYS Academic Research • 1 task of ANSYS Academic Research CFD & 1 task of ANSYS Academic Research Mechanical • Same combinations for Associate & Teaching products.

New at 14.0

•Note: System Coupling is not supported by a single task Academic license, at least two tasks must be available to a user as described above! 65/163


November 25, 2013

14.0 Academic Product Features Table:

• http://www.ansys.com/Industries/Academic/ANSYS+Academic+Portfolio

66/163


November 25, 2013

Student Portal Access Privileges Access Level

View Technotes, FAQ's Knowledge resources

View & Download product document ation

Direct Customer ASC, Admin, Prof, User

YES

YES

YES

YES

NO

NO

YES

YES

Direct Student

YES

YES

YES*

NO

YES**

YES

NO

YES

Indirect Customer ASC, Admin, Prof, User

YES

YES

NO

YES

NO

NO

YES

YES

Indirect Student

YES

YES

NO

NO

YES**

YES

NO

YES

67/163


November 25, 2013

Submit Download Download technical all ANSYS support products & Academic requests service packs Student

Access Access Access Student Fluent Ansoft Specific USC OTS pages

Student Registration: • There is a student specific registration page & process: • During registration, students are required to identify their professor (from a drop down selection) associated with their academic product license, this acts as an important verification mechanism, plus it allows us to tie registered students to specific accounts & professors. • The drop down selection of professors is populated from our database. • Each academic account ANSYS Customer Portal user has the option to check “Professor” in their “My Account” settings of the ANSYS Customer Portal. • Students from a given university will only be able to register if there is at least one associated “Professor” in our database. • Registered students will have a user type of “Student” rather than “Customer” and their registration will be active for one year. 68/163


November 25, 2013

Student Portal Login Page • Clear differentiation between the two portals when registering. • Login page is the same regardless. • Once logged in the content changes to student specific

69/163


November 25, 2013

Budoucnost ANSYS HPC • Aktuální vývoj v ANSYSu – HPC technologie pro etrémně rozsáhlé clustery/cloudy • • • •

Rovnoměrný výkon mezi 100-10000 jader pro FEM i CFD Nové řešiče pro FEM: Multilevel PCG, 2D parallel DSPARSE fronts GP-GPUs pro radiaci, UDFs, DEM a další CFD řešiče Hybridní distribuované/sdílené a vektorové technoogie HPC

• Škálovatelnost přes všechny komponenty a během celého procesu simulace • Síťování, nastavení, řešič, I/O, vizualizace, optimalizace,... • Integrace distribuovaného paralelního meshingu spolu s řešičem • Paralelizace pro lineární dynamiku včetně superpozičních metod

– Optimalizace výkonu • Dynamické vyvážení zátěže, optimalizované mapování zdrojů, optimalizace kompilátoru

– Použitelnost • Prostředí pro více-složkovou paralelizaci, podpora plánovačů • Nástroje pro vyšší toleranci chybovosti HW, dohled a debugging 70/163


November 25, 2013

HPC současnost Direct sparse solver – vylepšený algoritmus pro paralelizaci dekompozičních metod (LU) - METIS/ParMETIS Iterační PCG solver – paralelní odvozování předpodmiňovačů ANSYS Fluent - Dělení úlohy podle přenosových charakteristik sítě ANSYS Fluent – Hybridní paralelismus (OpenMP (shared memory)+MPI (distributed)) ANSYS 14 – zkompilován v Intel compileru – využívá nové AVX instrukční sady procesorů Intel/AMD ANSYS 14 – zavedena paralelizace I/O operací

71/163


November 25, 2013

RSM Node1

Node2

Node3

Node4 CFD

RSM ekm.svsfem.cz:10012/rsm/ FEM Wi-fi, LAN, IB

Explicit

http://ekm.svsfem.cz/ekm/

Multiphysics

72/163


November 25, 2013

Electro

Advantages/Disadvantages

SGI UV2 from SVS FEM SVS FEMWorks

73/163


November 25, 2013

HPC – virtuální cluster

74/163


November 25, 2013

Kenneth Wong ,Taming the HPC Queue, ANSYS ADVANTAGE, Volume V, Issue 3, 2011

ANSYS Remote Solve Manager Remote Solve Manager (RSM) ►Three-tiered architecture ►Client, solve manager, compute server

►Supports third-party schedulers ►LSF, PBS, MSCC

Solution Cell Update ►Mechanical, MAPDL , Fluent, CFX and Polyflow

Design Point Update ►All design points can be packaged for solution

via RSM.

75/163


November 25, 2013

Where does RSM Fit in the workflow?

Desktop

Server

Cluster (with 3rd party scheduler)

RSM as a scheduler

RSM as a transport Mechanism

• Submits to RSM itself. • Unit recognition: Jobs • E.g. a run of a solver such as CFX, FLUENT or Mechanical

• Submits through RSM to a high level scheduler such as LSF, PBS or HPC 2008. • Unit recognition: cores

76/163


November 25, 2013

Instalace a konfigurace RSM Při instalaci ANSYSu je nutné zvolit ANSYS Remote Solve Manager Standalone Services

Jako administrátor: "c:\Program Files\ANSYS Inc\v145\RSM\bin\AnsConfigRSM.exe"

-mgr –svr

Nainstalují se služby: ANSYS JobManager service-"c:\Program Files\ANSYS Inc\v145\RSM\bin\Ans.Rsm.JMHost.exe“ Komunikuje na portu 8145/tcp ANSYS ScriptHost service-"c:\Program Files\ANSYS Inc\v145\RSM\bin\Ans.Rsm.SHHost.exe„ Komunikuje na portu 9145/tcp

77/163


November 25, 2013

Konfigurace RSM ? ?

78/163


November 25, 2013 "C:\Program

Před konfigurací je nutné zastavit Službu ANSYS JobManager !

Files\ANSYS Inc\v145\RSM\bin\Ans.Rsm.Wizard.exe"

Konfigurace HPC pro WB

Lokální či vzdálený výpočet

79/163


November 25, 2013

Distribuovaný výpočet

Integrace s PBS/LSF

Checking license availabality

80/163


November 25, 2013

Konfigurace HPC – low level Základní podmínkou je funkční MPI (telnet localhost 8636 pro Platform-MPI) Pro Workbench je podmínkou funkční RSM (RSM se musí nastavit vždy po každé nové verzi) ANSYS krom centralizovaného řešení RSM nabízí i samostatnou podporu HPC jednotlivých komponent nazávisle na RSM ANSYS Mechanical APDL: "c:\Program Files\ANSYS Inc\v145\ansys\apdl\hosts145.ans" CFX : "c:\Program Files\ANSYS Inc\v145\CFX\config\hostinfo.ccl" Fluent: hosts file, Microsoft Job Scheduler LS-DYNA: hosts file

81/163


November 25, 2013

Konfigurace HPC pro ANSYS APDL Konfigurace hosts145.ans lobkowitz winx64 0 4 0 0 c:\temp MPI 1 1 budvar winx64 0 4 0 0 c:\temp MPI 1 1 starobrno winx64 0 4 0 0 c:\temp MPI 1 1 prazdroj winx64 0 4 0 0 c:\temp MPI 1 1 zasedacka winx64 0 2 0 0 c:\temp MPI 1 1

82/163


ANSYS Classic "c:\Program Files\ANSYS Inc\v145\ansys\apdl\hosts145.ans"

November 25, 2013

Konfigurace HPC pro CFD (MPI)

83/163

Fluent, LS-DYNA ../hosts


November 25, 2013

CFX

Lobkowitz Budvar Starobrno Prazdroj zasedacka

"c:\Program Files\ANSYS Inc\v145\CFX\config\hostinfo.ccl"

Struktura hostinfo.ccl Struktura hosts z fluentu, ls-dyna

SIMULATION CONTROL: EXECUTION CONTROL: PARALLEL HOST LIBRARY: HOST DEFINITION: starobrno Installation Root = C:\Program Files\ANSYS Inc\v%v\CFX Host Architecture String = winnt-amd64 Relative Speed = 11.94 Number of Processors =8 END # HOST DEFINITION starobrno HOST DEFINITION: lobkowitz Installation Root = C:\Program Files\ANSYS Inc\v%v\CFX Host Architecture String = winnt-amd64 Relative Speed = 11.94 Number of Processors =8 END # HOST DEFINITION lobkowitz HOST DEFINITION: budvar Installation Root = C:\Program Files\ANSYS Inc\v%v\CFX Host Architecture String = winnt-amd64 Relative Speed = 11.94 Number of Processors =8 END # HOST DEFINITION budvar HOST DEFINITION: bernard2 Installation Root = C:\Program Files\ANSYS Inc\v%v\CFX Host Architecture String = winnt-amd64 Relative Speed = 11.94 Number of Processors =8 END # HOST DEFINITION bernard2 HOST DEFINITION: prazdroj Installation Root = C:\Program Files\ANSYS Inc\v%v\CFX Host Architecture String = winnt-amd64 Relative Speed = 11.94 Number of Processors =8 END # HOST DEFINITION prazdroj END # PARALLEL HOST LIBRARY END # EXECUTION CONTROL END # SIMULATION CONTROL

Optimální nastavení úlohy pro HPC (MPI) - tunning Interconnect prot.: Ib, dapl, tcp, shm:dapl, ofa, tmi, ofed

• HP-MPI •

Message latency and bandwidth (MPI_Pack, MPI_Unpack, MPI_ANY_SOURCE, MPI_Recv_init, MPI_startall, NUMA,…)

•

Multiple network interfaces (MPI_TOPOLOGY, R-server, K-server,…)

•

Processor subscription (optimalizace komunikace s plánovači RSM, PBS, LSF,...)

•

MPI routine selection (multilevel parallelism, process placement,...)

•

– http://www.ncsa.illinois.edu/UserInfo/Resources/Hardware/CommonDoc/HP/M PI/3_understand.html Intel MPI

• • • • •

MPI profile (I_MPI_STATS, I_MPI_STATS_SCOPE) Interconnect (I_MPI_FABRICS, I_MPI_ADJUST_REDUCE, I_MPI_DAPL_SCALABLE_PROGRESS,…) Layout (I_MPI_PERHOST, I_MPI_PIN_PROCESSOR_LIST,…) MPI/OpenMP (OMP_NUM_THREADS) Connection mode (I_MPI_DYNAMIC_CONNECTION, I_MPI_WAIT_MODE, I_MPI_SHM_BYPASS,…)

– http://www.rz.rwth-aachen.de/global/show_document.asp?id=aaaaaaaaaacfigd 84/163


November 25, 2013

Intel MPI

85/163


November 25, 2013

How to select HW resources for ANSYS HPC jobs

86/163


November 25, 2013

Velikost úloh pro efektivní výpočet • Direct sparse: závisí na nejvetší velikosti vstupní fronty: GPU je efektivní pro úlohy od ~1M DOF do ~8M DOF pro 6GB Tesla C2075 or Quadro 6000 • Iterační solvery: závisí na velikosti paměti GPU: GPU je efektivní pro úlohy od ~1M DOF do ~5M DOF pro 6GB Tesla C2075 or Quadro 6000 1k-10k DOF rychlejší Sparse 10k-100k DOF Sparse i PCG podobný výkon >100k DOF rychlejší PCG (méně IO operací)

čas

HPC+GPU

II.

III.

I. DOF Pierre Louat , HPC Technology for Structural Simulations, 2012 Sales Conference Theme and Team Building, France 2012 87/163


November 25, 2013

NVIDIA Inc. http://www.nvidia.com

HPC+GPU+SSD

88/163


November 25, 2013

Pierre Louat , HPC Technology for Structural Simulations, 2012 Sales Conference Theme and Team Building, France 2012

Kepler GPU Architecture for ANSYS Features

Tesla K20

Tesla C2075

GPU Architecture

Kepler

Fermi

DGEMM performance

> 1000 Gigaflops

370 Gigaflops

Memory bandwidth

> 200 GBytes/sec

150 GBytes/sec

Memory size (GDDR5)

6 GigaBytes

6 GigaBytes

CUDA cores

> 1000

448

Available in HP Z8xx Late 2012 MAXIMUS Workstations 89/163


November 25, 2013

Today

90/163


November 25, 2013

Understanding the effect of clock speed Impact of CPU Clock on Application Performance Processor: Xeon X5600 Series Hyper Threading: OFF, TURBO: ON Active cores: 12/node; Memory speed: 1333 MHz (performance measure is improvement relative to CPU Clock 2.66 GHz) 1.40 1.35

Improvement due to Clock

Higher is better

1.30 1.25 1.20 1.15

2.66 GHz 2.93 GHz 3.47 GHz

1.10 1.05 1.00 0.95 0.90 0.85 0.80 Clock Ratio eddy_417K aircraft_2M turbo_500K sedan_4M ANSYS/FLUENT Model

91/163


November 25, 2013

truck_14M

Understanding the effect of clock speed

92/163

•

Effect of increased core operating frequencies on the DMP benchmarks running on 8 cores

•

Influence is highest for sparse solver benchmarks © 2012 SVS FEM spol. s r.o.

November 25, 2013

Understanding the effect of memory bandwidth X5570 quad-core processors have higher performance per core than X5670 six-core processors since X5670 share more resources

1 0.5 0

x5570 x5670

Relative performance for 24-way runs: three X5570 nodes, two X5670 nodes ANSYS FLUENT Rating

16 of 16 cores used in 2 X5570 nodes 16 of 24 cores used in 2 X5670 nodes

93/163


November 25, 2013

10000 100 1

X5570 X5670

Turbo Boost (Intel) / Turbo Core (AMD) • Turbo Boost (Intel)/ Turbo Core(AMD) is a form of over-clocking that allows you to give more GHz to individual processors when others are idle.

• With the Intel’s have seen variable performance with this ranging between 08% improvement depending on the numbers of cores in use. • The graph below for CFX on a Intel X5550. This only sees a maximum of 2.5% improvement.

94/163


November 25, 2013

Turbo Boost (Intel) Solver Sparse

PCG

• • • 95/163

Number of Cores 1 2 4 1 2 4 1 2 4 1 2 4

Mode SMP

DMP

SMP

DMP

Elapsed Time Turbo Off 299 166 121 296 179 139 299 207 186 301 221 180

DELL M6600 I7 2820 @ 2.29 GHz Sparse model : 0.34 M DOF PCG model: 1.56 M DOF © 2012 SVS FEM spol. s r.o.

November 25, 2013

Elapsed Time - Speedup due Turbo On to Turbo On 213 1.40 124 1.34 103 1.17 210 1.41 137 1.31 117 1.19 231 1.29 169 1.22 155 1.20 232 1.30 186 1.19 149 1.21

Recommendation: Activate Turbo Boost

Understanding the effect of memory speed • We can see here the effect of memory speed. • This has implications on how you build your hardware. • Some processors types have slower memory speeds by default.

• On other processors nonoptimally filling the memory channels can slow the memory speed.

96/163


November 25, 2013

Impact of Memory Speed on Benchmark Speed

Understanding the effect of memory speed • We can see here the effect of memory speed.

Impact of DIMM speed on ANSYS/FLUENT Application Performance (Intel Xeon x5670, 2.93 GHz) Hyper Threading: OFF, TURBO: ON Active threads per node: 12

• This has implications on how you build your hardware.

• On other processors nonoptimally filling the memory channels can slow the memory speed.

130% 125% 120% Impact of Memory Speed

• Some processors types have slower memory speeds by default.

(performance measure improvement is relative to memory speed of 1066 MHz)

115% 110% 1066 MHz 1333 MHz

105% 100% 95% 90% 85% 80% eddy_417K

turbo_500K

aircraft_2M

sedan_4M

ANSYS/FLUENT Model

97/163


November 25, 2013

truck_14M

Hyper-threading – ANSYS Fluent • Hyper-Threading Technology makes a single physical processor appear as two logical processors.

Evaluation of Hyperthreading on ANSYS/FLUENT Performance iDataplex M3 (Intel Xeon x5670, 2.93 GHz) TURBO: ON (measurement is improvement relative ot Hyperthtreading OFF)

• This is not the same as physically having two logical processors and does not give double the speedup.

• It is worth noting that this has licensing implications as you would need to oversubscribe the physical cores and hence would need double the HPC Licenses. 98/163


November 25, 2013

HT ON (24 threads on 12 physical cores)

Improvemet due to Hyperthreading .

1.10

Higher is better

• In our tests we’ve seen as high as a 20% increase in performance although you can see the actual performance can be quite variable from the graph opposite.

HT OFF (12 threads on 12 physical cores)

1.05

1.00

0.95

0.90 eddy_417K

turbo_500K

aircraft_2M ANSYS/FLUENT Model

sedan_4M

truck_14M

Hyper-threading – ANSYS Mechanical

• In our tests we’ve seen only a 2% increase in performance although you can see the actual performance can be quite variable . • It is worth noting that this has licensing implications as you would need to oversubscribe the physical cores and hence would need double the HPC Licenses.

Hyper-threading is NOT recommended

99/163


November 25, 2013

AMD vs. Intel • These are the published figures from v13.0 for FLUENT. Notice the results here are per node not per core. • In this case this for the 8 node job this is 192 cores vs. 96 cores. It does not state if SMT (hyper-threading) is used here. • Below is a CFX job run on the slightly slower AMD 6140 and in comparison to the 5670 (2.93GHz 5.8 GT/s QPI rating). It also only has 8 cores instead of the 12 the 6180 has. • We can see that the AMD 6140 requires a 1/3 again the number of cores to keep pace with the Intel x5670. • This is consistent with the above FLUENT results. 100/163


November 25, 2013

Comparing 2 socket vs. 4 socket systems

• Current 4 socket systems come up slower than their 2 socket counterparts (based on Intel Westmere vs. Xeon E7-8837). • Clock speed slower • Memory speed slower • No additional memory bandwidth. Performance of ANSYS Fluent on two-socket and four-socket based systems Performance measure is Fluent Rating (higher values are better) 2-socket based Systems IBM HS22/HS22V Blade, 3550/3650 M3, Dx360 M3 (Xeon 5600 Series)

Nodes

Sockets

Cores

1 2

2 4

12 24

101/163


Fluent Rating 88 173

November 25, 2013

4-socket based Systems IBM HX5 Blade, X3850 (Xeon E7-8837 series)

Nodes

Sockets

Cores

1 1

2 4

16 32

Fluent Rating 96 188

Comparing 1 node vs. 2 node for 4 socket system

• Comparison with equal number of cores between: Locally distributed solution (all cores on the same mother board) Distributed solution over 2 nodes (with a 1 Gbit network) Performance of ANSYS Mechanical on one node and two node based systems Performance measure is elapsed time (lower values are better) 1 Node 4-socket based Systems Dell R910 4 socket X7560 @ 2.26 GHZ 256 GB RAM

2 Node 4-socket based Systems Dell R910 4 socket X7560 @ 2.26 GHZ 256 GB RAM

M DOF

Solver

Cores

Elapsed Time

M DOF

Solver

Cores

Elapsed Time

12.3

PCG

12

1159

12.3

PCG

6+6

952

1.54

Sparse

12

1595

1.54

Sparse

6+6

1558

Conclusion with a same number of core : • •

102/163

PCG solver works better on multiple-node configuration Performance of sparse solver is less sensitive for multiple-node configuration © 2012 SVS FEM spol. s r.o.

November 25, 2013

Understanding the effect of the interconnect • When going for multiple systems linked together the interconnect becomes an important factor.

ANSYS/FLUENT Performance iDataplex M3 (Intel Xeon x5670, 12C 2.93 GHz) Network: Gigabit, 10-Gigabit, 4X QDR Infiniband (QLogic, Voltaire) Hyperthreading: OFF, TURBO: ON Models: truck_14M

• The interconnect is the fabric that connects the nodes.

Voltaire

10-Gigabit

Gigabit

Higher is better

4000

FLUENT Rating

• We can see from the graph opposite with FLUENT how quickly the performance of Gigabit Ethernet drops off.

QLogic 5000

3000

2000

1000

0 12

24

48

96

192

Number of Cores used by a single job

103/163


November 25, 2013

384

768

Understanding the effect of interconnect bandwidth

104/163

•

Effect of interconnect bandwidth on the DMP benchmarks running with 16 MPI processes across 2 nodes

•

Higher bandwidth most relevant for sparse solver benchmarks © 2012 SVS FEM spol. s r.o.

November 25, 2013

Disks/Storage • IDE, SCSI, SAS, SSD Faster

Převažuje-li Sparse/Block Lanczos/out-of-core > SAS & SSD v RAID0

• RAID setups

• RAID 0 – for speed • RAID 1,5 – for redundancy • Parallel File Systems

• Only required for large clusters 105/163


November 25, 2013

ANSYS CFX Partitioning • Optimize parallel partitioning in multi-core clusters (CFX)β

• Partitioner determines number of connections between partitions and optimizes part.-host assignments

• Re-use previous results to initialize calculations on large problem (CFX) β

• Large case interpolation for cases with >~100M nodes

P6

P2

P7

P1

P8 P3

P5

• Eliminates ‘isolated’ partition spots Dramatically reduced partitioning times for cases with fluid-solid interfaces and very large numbers of regions © 2012 SVS FEM spol. s r.o.

November 25, 2013

Partitioning step finds adjacency amongst partitions; partitions with max adjacency are grouped on same compute nodes

P1

P3

P2

P7

P5

P6

P4

P8

Compute Node 1

• Clean up of coupled partitioning option for multi-domain cases (CFX)

106/163

P4

Compute Node 2

107/163


November 25, 2013

Fluids I/O Asynchronous I/O for Linux Fluent Total write time 3-5x quicker over NFS Even larger speed-ups on bigger cases and local disk (up to 10x)

108/163

Mesh

File

Location

Async I/O

Time

15M

Cas

NFS

OFF

217s

15M

Cas

NFS

ON

62s

15M

Dat

NFS

OFF

113s

15M

Dat

NFS

ON

8s

30M

Cas

NFS

OFF

207s

30M

Cas

NFS

ON

75s

30M

Dat

NFS

OFF

144s

30M

Dat

NFS

ON

10s


November 25, 2013

Optimální nastavení úlohy pro HPC FEM

Mechanical APDL // Performance Guide

SPARSE: obecně 3D, bázové fce vyššího stupně > vhodnější pro GPU (vyjímka modální analýza) PCG: Low Level difficulty > vhodnější pro GPU

SMP: max do 8 jader (nejvyšší efektivita při výpočtu na 4) DMP: u FEM nejefektivejší do 32 jader, použitelné maximum 128, u CFD dobrá škálovatelnost až do 1024 jader, ale lze použít až do 3072 jader In-core: optimální režim výpočtu v RAM (oproti out-of-core = swap file). In-core lze vynutit správnou alokací paměti Ansys145 –m ### –db ### Nebo pomocí příkazu: BCSOPTION,,INCORE příp. DSPOPTION,,INCORE

Pro FEM maximalizace I/O -> RAID0 + >4x SSD/SAS, propustnost >500 MB/s, lze si pomoci s RAMDiskem (linux). U clusteru je vhodné lokální I/O storage pro každé PC. Pro CFD maximalizace síťové propustnosti, propustnosti RAM/bus, fragmentace sítě, nastavení MPI RAM (in-core, FEM): SPARSE solver: SMP  10GB/1mil DOFs, DMP  10GB/1mil DOFs * 1/#cores (BCSoption, DSPoption) PCG solver: SMP  1GB/1mil DOFs, DMP  2GB/1mil DOFs * 1/#cores (PCGOPT,LevelDiff; MSAVE,on) (vyšší Level  více RAM/rychlejší) Block-Lanczos: SMP  15-20GB/1mil DOFs (MODOPTION,BlockSize) (větší blok  více RAM/rychlejší) PCG-Lanczos: SMP  2GB/1mil DOFs, DMP  3GB/1mil DOFs * 1/#cores (PCGOPT,5; MSAVE,on) optimální pro výpočet přes více uzlů v non-shared mem. Supernode: SMP  mezi Block a PCG Lanczos (SNOPTION,BlockSize)

Kontakty: redukce pomocí

CNCHECK,TRIM

Redukce při spojování souborů po DMP výpočtu: DMPOPTION (specifikuje, které soubory se mají spojit – RSM, EMAT, ESAV,...), COMBINE/RESCOMBINE

Metody doménového dělení: DDOPTION,GREEDY/METIS 109/163


November 25, 2013

Optimální nastavení úlohy pro HPC CFD (Fluent) • Nastavení dělení domény: – Parallel → Auto Partition (15 metod) – Parallel → Partitioning and Load Balancing... (manuální)

110/163


November 25, 2013

Optimální nastavení úlohy pro HPC CFD (CFX) • Pro řešení jedné úlohy se snažte používat stroje s podobným výkonem • Vyhněte se používání swapovacího souboru • Používejte rychlou síťovou konektivitu (1Gbps/Infiniband) • Inifiband je podporován pouze přes Platform-MPI • Před spuštěním úlohy zajistěte dostupnost uzlů (nesmí být sdílené jinou úlohou)

• Vyhněte se přidělování většího počtu vláken než je počet jader • Používejte „rozumný“ počet partitions. Orientačně počet part.~=počet jader

111/163


November 25, 2013

Monitoring-Ganglia

112/163


November 25, 2013

113/163


November 25, 2013

ANSYS Cloud NICE DCV, EngineFrame, Vcollab, EKM

114/163


November 25, 2013

Interaktivní vzdálený přístup - Cloud

• Soubory jsou výhradně jen na HPC prvcích – efektivní a pokročilá správa dat, spolupráce • Celá simulace probíhá zcela ve vzdáleném režimu (Pre/Solv/Post) • Dostupná a plně implementovatelná technologie současnosti (u těch nejnáročnějších)

115/163


November 25, 2013

Simulace online

116/163


November 25, 2013

Implementace EKM

117/163


November 25, 2013

•

Připojení různých datových zdrojů

•

Prohledávání celé globální struktury na jeden klik

Online vizualizace výsledků

118/163


November 25, 2013

Řízení projektu • Uživatelem definovaná struktura • Modelování toku výr. procesu

• Šablony (web) • Možnost vytvářet nadstavby (XML, Python,...)

• Silná podpora metadat dalších aplikací

119/163


November 25, 2013

VCollab SVS FEM s.r.o. se stala oficiálním partnerem firmy VCOLLAB. Aplikace VCOLLAB nabízí vizualizaci CAD/CAE dat ANSYSu, ale i SW třetích stran

120/163


November 25, 2013

VCollab vizualizace Nabízí velmi propracovaný SW pro převod CAD/CAE dat do vysoce komprimovaného jednotného vizualizačního formátu, který může být poskytován vašim zákazníkům/konzultantům/managerům, bez nutnosti vlastnit vlastní CAE SW. Výstupy mohou být snadno integrovány do MS PowerPointu, Excelu či prezentovány na webu (viz www.svsfem.cz) CAE Results File Size ( MB)

CAX File Size (MB)

File size Reductio n

ABAQUS (FIL)

2,930

47.4

98%

MSC NASTRAN

289

46.4

84%

MSC MARC

243

23.9

90%

ANSYS

14,000

92.0

99%

LS DYNA

363

145.0

60%

FLUENT

347

13.1

96%

CAE Software

121/163


November 25, 2013

NICE DCV

122/163


November 25, 2013

Solution of Large models

123/163


November 25, 2013

Substructuring allows for collaborative work or memory efficient harmonic and transient simulations

                                        

124/163


November 25, 2013

                    

Reduce linear components Full components

Reduced model                      

                  

                    

125/163


November 25, 2013

Compute the solution by assembling the reduced models

                     

                   

                       126/163


November 25, 2013

                    

Post-process on full model or areas of interest

127/163


November 25, 2013

Overview – Solution Steps Generation Pass (Step 1)

SubStructure 1 1. Assemble the M, K 2. Static reduction 3. Modal analysis

SubStructure 2 1. Assemble the M, K 2. Static reduction 3. Modal analysis Superelement 2

Superelement 1

USE Pass (Step 2)

Expansion Pass (Step 3)

128/163

... ...

SubStructure N 1. Assemble the M, K 2. Static reduction 3. Modal analysis Superelement N

1.Couple superelements 2. Analysis

Transfer back to physical coordinate of substructure 1


November 25, 2013

Transfer back to physical coordinate of substructure 2

...

Transfer back to physical coordinate of substructure N

Static Substructuring & CMS

 Guyan Reduction procedure  inertia forces are negligible compared to elastic forces  Net result: the reduced stiffness matrix is exact, whereas the reduced mass and damping matrices are approximate

Note: Choosing master DOF is an important step in a reduced analysis, impacting accuracy of results 129/163


November 25, 2013

Component Mode Synthesis  CMS is a type of substructuring which performs a modal analysis of a structure based on independent modal analyses of its parts  The synthesis involves making the components work together as a single structure by satisfying inter-component compatibility and equilibrium constraints  Master DOF are required only at interface nodes

130/163


November 25, 2013

CMS with ANSYS Workbench

131/163


November 25, 2013

APDL macros embedded in the simulation tree for generation, use and expansion pass

132/163


November 25, 2013

Results are available through standard operations

133/163


November 25, 2013

When to use substructuring? Reduce memory consumption Allow collaborative work and create a library of components Reduce solution time for harmonic and transient analyses You can perform efficient design variations by reusing components

134/163


November 25, 2013

Comparing the accuracy of a CMS analysis to a standard one

135/163


November 25, 2013

IT4Innovations Případová studie řešení pro intenzivní HPC výpočty. Srovnání dílčích výsledků pro CERIT a UV2k

Petr Koňas

IT4Innovations& národní superpočítačové centrum 136/163


November 25, 2013

Benchmarky Přes 400 benchmarkových úloh. Více jak 300 stránek výsledků z benchmarků. 3 týdny testování a optimalizace.

ANSYS testy Workbench Mechanical CFX Fluent LS-DYNA

137/163


November 25, 2013

Benchmarky Workbench MAPDL

138/163

JCG solver, symmetric matrix, 1100k DOFS, transient, linear, thermal analysis

PCG solver, symmetric matrix, 11800k DOFs, static, linear, structural analysis with single load step

PCG Lanczos eigensolver, symmetric matrix, 3000k DOFs, modal, linear, structural analysis requesting 15 modes



November 25, 2013

Benchmarky Workbench MAPDL Sparse solver, symmetric matrix, 400k DOFs, static, nonlinear, structural analysis

Sparse solver, non-symmetric matrix, 1000k DOFs, static, nonlinear, thermal-electric coupled field analysis

139/163


November 25, 2013

Sparse solver, symmetric matrix, 2300k DOFs, transient, nonlinear, structural analysis

Benchmarky Workbench MAPDL Sparse solver, symmetric matrix, 1000k DOFs, harmonic, linear, structural analysis requesting 4 frequencies.

Sparse solver, symmetric matrix, 2100k DOFs, static, nonlinear, structural analysis with 1 iteration

140/163



November 25, 2013

Srovnání výsledků pro různé clustery Benchmarky Workbench MAPDL CERIT, max.accel=17x

UV2000, max.accel=8.3x JCG solver, symmetric matrix, 1100k DOFS, transient, linear, thermal analysis

CERIT=E5-2643 IT4I=E5-2665 UV2k=E5-4617

141/163


November 25, 2013

Srovnání výsledků pro různé clustery Benchmarky Workbench MAPDL CERIT, max.accel=8.2x

UV2000, max.accel=11.3x PCG solver, symmetric matrix, 11800k DOFs, static, linear, structural analysis with single load step

CERIT=E5-2643 IT4I=E5-2665 UV2k=E5-4617

142/163


November 25, 2013


UV2000, max.accel=15.1x

CERIT=E5-2643 IT4I=E5-2665 UV2k=E5-4617

143/163



November 25, 2013


UV2000, max.accel=32x PCG solver, symmetric matrix, 10700k DOFs, static, linear, structural analysis with single load step

CERIT=E5-2643 IT4I=E5-2665 UV2k=E5-4617

144/163


November 25, 2013


UV2000, max.accel=8.1x Sparse solver, symmetric matrix, 400k DOFs, static, nonlinear, structural analysis

CERIT=E5-2643 IT4I=E5-2665 UV2k=E5-4617

145/163


November 25, 2013


UV2000, max.accel=16x Sparse solver, non-symmetric matrix, 1000k DOFs, static, nonlinear, thermal-electric coupled field analysis

CERIT=E5-2643 IT4I=E5-2665 UV2k=E5-4617

146/163


November 25, 2013


UV2000, max.accel=9.3x Sparse solver, symmetric matrix, 2300k DOFs, transient, nonlinear, structural analysis

CERIT=E5-2643 IT4I=E5-2665 UV2k=E5-4617

147/163


November 25, 2013


UV2000, max.accel=15x Sparse solver, symmetric matrix, 1000k DOFs, harmonic, linear, structural analysis requesting 4 frequencies.

CERIT=E5-2643 IT4I=E5-2665 UV2k=E5-4617

148/163


November 25, 2013


UV2000, max.accel=13x Sparse solver, symmetric matrix, 2100k DOFs, static, nonlinear, structural analysis with 1 iteration

CERIT=E5-2643 IT4I=E5-2665 UV2k=E5-4617

149/163


November 25, 2013


UV2000, max.accel=13.1x Sparse solver, symmetric matrix, 4900k DOFs, static, nonlinear, structural analysis with 1 iteration

CERIT=E5-2643 IT4I=E5-2665 UV2k=E5-4617

150/163


November 25, 2013

CERIT=E5-2643, IT4I=E5-2665,UV2k=E5-4617

Srovnání výsledků pro různé clustery Benchmarky Workbench MAPDL, normalizované výsledky

151/163

JCG solver, symmetric matrix, 1100k DOFS, transient, linear, thermal analysis





November 25, 2013

CERIT=E5-2643, IT4I=E5-2665,UV2k=E5-4617

Srovnání výsledků pro různé clustery Benchmarky Workbench MAPDL, normalizované výsledky Sparse solver, symmetric matrix, 400k DOFs, static, nonlinear, structural analysis

Sparse solver, non-symmetric matrix, 1000k DOFs, static, nonlinear, thermal-electric coupled field analysis

152/163


November 25, 2013

Sparse solver, symmetric matrix, 2300k DOFs, transient, nonlinear, structural analysis

CERIT=E5-2643, IT4I=E5-2665,UV2k=E5-4617

Srovnání výsledků pro různé clustery Benchmarky Workbench MAPDL, normalizované výsledky Sparse solver, symmetric matrix, 1000k DOFs, harmonic, linear, structural analysis requesting 4 frequencies.


153/163



November 25, 2013

Benchmarky Fluent

417K - 3d,ke,seg Reacting flow with eddy-dissipation

1.8M - 3d,rke,coup-imp External flow over aircraft win

4M - 3d,ke,seg, external, hybrid aerodynamics around a car 14M - 3d,des,seg External flow around a truck bod

154/163


November 25, 2013

Benchmarky Fluent

111M - 3d,des,seg External flow around a truck body

155/163


November 25, 2013

Benchmarky CFX Transonic flow around an airfoil. Flow is 2D the mesh is extruded to give a 3D meshes of various sizes • Turbulent SST, ideal gas, heat transfer • Default advection scheme (high resolution) • Global mesh size: 9,933,000 nodes, 9,434,520 elements (all hexahedra)

Transonic flow around an airfoil. Flow is 2D the mesh is extruded to give a 3D meshes of various sizes • Turbulent SST, ideal gas, heat transfer • Default advection scheme (high resolution) • Global mesh size: 47,773,000 nodes, 47,172,600 elements (all hexahedra)

Transonic flow around an airfoil. Flow is 2D the mesh is extruded to give a 3D meshes of various sizes • Turbulent SST, ideal gas, heat transfer • Default advection scheme (high resolution) • Global mesh size: 104,533,000 nodes, 103,779,720 elements (all hexahedra)

Low speed external aero around a LeMansstyle car • Turbulent k-e, ideal gas, heat transfer • Default advection scheme (high resolution) • Global mesh size: 1,861,696 nodes, 10,285,401 elements (all tetrahedra)

156/163


November 25, 2013

Benchmarky CFX Transonic flow around an airfoil. Flow is 2D the mesh is extruded to give a 3D meshes of various sizes • Turbulent SST, ideal gas, heat transfer • Default advection scheme (high resolution) • Global mesh size: 104,533,000 nodes, 103,779,720 elements (all hexahedra)

157/163


November 25, 2013

Benchmarky LS-DYNA Taurus Number of Parts - 778 Number of Nodes - 936258 Number of Shells - 805505 Number of Beams - 4 Number of Solids - 99486 Number of Elements - 1057113 Silverado Number of Parts - 679 Number of Nodes - 942677 Number of Shells - 873144 Number of Beams - 2662 Number of Solids - 53293 Number of Elements - 929131

158/163


Yaris Number of Parts 771 Number of Nodes 998,218 Number of Shells 950,560 Number of Beams 4,497 Number of Solids 19,314 Total Number of Elements 974,383

November 25, 2013

ACS infoday

ANSYS Customization Suite (ACS) Infoday Programovací metody uživatelských úprav programového balíku ANSYS aneb „Jak si vlastnoručně doplnit co Vám v ANSYSu chybí...“ Společnost SVS FEM s.r.o. si Vás dovoluje pozvat na výjimečné setkání zaměřené na představení a názorné ukázky práce v programovacích nástrojích pro uživatelskou modifikaci prostředí ANSYS za účelem rozšíření funkcionality dle vlastní potřeby. Kdy: 3.12. 2013 v 9 hodin. Kde: Hotel Avanti, Střední 61, Brno, 602 00 Cena: zdarma. http://www.svsfem.cz/registrace 159/163


November 25, 2013

SVS FEM s.r.o. SVS FEM s.r.o. Škrochova 3886/42 615 00 Brno-Židenice Tel. : +420 543 254 554, +420 543 254 555 Fax.: +420 543 254 556 Email: [email protected] Web: http://www.svsfem.cz

160/163


November 25, 2013

Děkuji za pozornost...

161/163


November 25, 2013

ANSYS Fluent - MetaCentrum

ANSYS Fluent - MetaCentrum

Suggest Documents

ansys fluent guide

ANSYS FLUENT UDF Manual

ansys fluent 12.0

ANSYS Fluent UDF Manual

ansys fluent guide

ansys fluent 12.0

CFD Modeling Using ANSYS FLUENT

Ansys Fluent R 17.0 - EnginSoft

ANSYS Fluent UDF Manual

Tutorial 1. Introduction to Using ANSYS FLUENT in ANSYS ...

Introduction to Introduction to ANSYS FLUENT

Introduction to Introduction to ANSYS FLUENT - iMechanica

Introduction to Introduction to ANSYS FLUENT

ACCELERATING ANSYS FLUENT 15.0 USING NVIDIA GPUS

Introduction to Introduction to ANSYS FLUENT

Introduction to Introduction to ANSYS FLUENT - iMechanica

ANSYS FLUENT Performance Benchmark and Profiling

Introduction to Introduction to ANSYS FLUENT

ACCELERATING ANSYS FLUENT 15.0 USING NVIDIA GPUS

ANSYS FLUENT 12.0 UDF Manual

modeling sootblower jet effectiveness with ansys fluent

Benchmark of Femlab, Fluent and Ansys

Introduction to Introduction to ANSYS FLUENT - iMechanica

Introduction to Introduction to ANSYS FLUENT