1	Sizing, Stabilizing, and Securing Lawson 8.x Mid-Atlantic User Group June 11, 2004
2	Ringmaster :: John Eisenschmidt
3	Introduction :: Feld Entertainment Who is Feld Entertainment? Family owned, family entertainment company Producer of live shows that play in 47 countries: Ringling Bros. and Barnum & Bailey Circus Disney On Ice and Disney Live Produced the former Siegfried and Roy at the Mirage Center for Elephant Conservation and Williston Retirement Center for Elephants Vertically integrated, fast moving company Microsoft shop
4	Storyline:: Feld and Lawson Applications Implemented 6.x Human Resources on AS/400 in 1996 Migrated to 7.x Human Resources on Windows in 2000 Implemented 8.x Financials and Procurement on Windows in January 2003 Re-implemented Human Resources over five weeks in December 2003, consolidating it with Financials and Procurement Hardware Started with a three server implementation in October 2002 Consolidated to a single server implementation in June 2004
5	Lawson-related products Feld owns In production Applications: AM, AP, CB, BN, GL, FB, HR, MA, PA, PO, PR, TA BSI Tax Factory Enterprise Reporting Microsoft Add-ins Not yet in production: Applications AC, IC, RQ Process Flow Self-service Employee Manager Vendor
6	Conventions :: The Lawson Architecture
7	Sizing :: Outline Web Tier Application Tier Database Tier System
8	Sizing :: Web Tier Web server overhead: Low CPU utilization IIS is Multithreaded, runs well on Windows Apache 1.x uses fork(), runs well on Unix Low RAM utilization Significant I/O if access logging turned on 8.0 Environment and IOS require a servlet container Jakarta Tomcat IBM Websphere 8.1 Environment and IOS require server-side Java BEA WebLogic IBM Websphere
9	Sizing :: Application Tier CPU Utilization higher on Windows than Unix (fork() inside a thread) Batch jobs can be very CPU intensive Disk Creation and deletion of print files, COBOL work files, XML work files, job logs, and system log files Memory 8MB per LACOBRTS 8MB per RMI worker thread Every streamida process gets a corresponding db process MKS Toolkit Remember that Lawson’s Windows release is their Unix codebase MKS has its own set of knobs and dials to tune Network Utilization high if database is on a separate server
10	Sizing :: Database Tier CPU Not as CPU intensive as Applications Makes use of multiprocessor support Disk Fast physical disks, more spindles are better RAID Avoid RAID-5 (including HP’s ADG) – Parity Calculation Combinations of RAID-0 and RAID-1 to balance speed and redundancy Be sure your RAID stripe side and your OS file system block size jive Minimize number of file groups / table spaces Consider your growth when deciding on capacity Memory Database should not be set to use all of the system memory Do not starve the OS of it’s fair share Network Utilization very high if applications are on a separate server
11	Sizing :: System Considerations Bottlenecks Can never eliminate a bottleneck, only move it Growth Better to buy more server and “grow into” it then add capacity later and have to re-tune (kernel parameters, environment, database) Incorrect sizing before implementation results in a “crippled go-live” until you can fix -- always harder to fix a system once in production Redundancy Fans Disk Controllers Disks Network Cards Power Supplies Test your redundancies before you go live!!! Sizing Workbook The recommendations Lawson makes based on your sizing workbook are too small!
12	Stabilizing :: Outline Web Tier Application Tier Database Tier System
13	Stabilizing :: Web Tier Log files %WEBDIR%/logan.log %CGIDIR% ags.log ; dme.log ; logan.log %CGIDIR/rmi% logan.log ios.log (gets very large) rmi-proxy.log ; rmi-standard.log ; rmi-tcp.log ; rmi-transport.log %TOMCAT_HOME%/logs/jasper-<date>.log Java Web and Application tier should be running the same stable release of 1.31 (>>java –version) Synchronize JAR files %GENDIR%/java/jar and %TOMCAT_HOME%/lib Check Java class path Servlets and CGIs Lawson has replaced many CGIs with Servlets CGI’s are stateless and somewhat slower Problems with CGIs and Servlets can be debugged by “touching” log files ags.log ; dme.log ; pts.log If you’re having problem with the Web Tier, start troubleshooting with servlet calls: https://server/servlet/Reflux?_PDL=LOGAN&_TKN=RD30.1 https://server/servlet/Xpress?_PDL=LOGAN&_TKN=RD30.1 Security Cache Be mindful that Portal caches LAUA Security in three places: Web Browser ; Web Server ; RMI Users will need to clear browser cache along with an IOSCacheRefresh for LAUA changes to take effect
14	Stabilizing :: Application Tier [Windows] Usermap Run delusers to cleanup orphaned usermap entries Remove UIDs of 0 (not root on Unix) Clear Usermap cache daily: listusermap -n GEN Database Check integrity once a month: integ GEN Periodic full garbage collection: dbreorg –G GEN Checkpoint Table (CKPOINT) Be sure all users have access to the LA system code Check periodically to see that it is clearing out Job cleanup Daily cleanup of waiting jobs and corrupt job records deljobhst -w 'curdate' - 1 jobinteg -d Pathing Make sure system path is correct Registry Be sure users have permission to “Set Value” on: HKEY\LM\SOFTWARE\LAWSON SOFTWARE\LAWSON INSIGHT ENVIRONMENT\GLOBAL\SECURITY\BATCH Work file cleanup COBOL work files: find %LAWDIR%\prod\work -mtime +30 -print -exec rm -r {} ; XML work files: find %TEMPDIR% -mtime +30 -print -exec rm -r {} ;
15	Stabilizing :: Database Tier Anti-virus software Be sure it is not scanning your database files Database Driver [MDAC/Library] Version(s) Must be synchronized between Web and Application Tier If not, you will end up with duplicate records and index errors Maintenance Indexes should be rebuilt Database index is a balanced B-Tree structure, and even though the database can maintain for your automatically, they should be manually rebuilt from time to time Statistics should be updated Any RDBMS cost-based optimizer (CBO) requires statistics to be up to date for optimal query plans Storage [SQL Server] Database should be shrunk periodically [Oracle] Table spaces should be coalesced periodically [Any] Your database instance should be destroyed and recreated at least twice a year Schema Object Ownership [SQL Server] Be sure that the database is owned by Lawson user, and that all objects belong to [dbo].: A bldmsf2000ddl will change object ownership, and can affect your ability to use expsysdb and impexp [Oracle] Application data, LOGAN data, and Tax Factory should be in their own Oracle instance to avoid namespace collision Lawson user should be the schema owner
16	Stabilizing :: System File system Lawson creates and deletes many small files. Use a commercial defrag utility like Diskkeeper Ideally, Lawson Applications and Database should be on different file systems: keeps database from fragmenting takes into consideration different Master File Table or inode needs Kernel Tuning Parameters Lawson uses System V shared memory, and the minimums are usually insufficient Sun provides the bible on tuning SYS V Shared memory: http://docs.sun.com/db/doc/806-6779/6jfmsfr7q?a=view LACONFIG [Windows] “Desktop Heap Size” Windows has a fixed heap of 48MB to track all interactive and non-interactive processes (can be changed in the registry) Every Lawson program consumes about 2.7K of Desktop Heap If DHS is set to 4096K, then 12 desktops can be created that support ~1500 processes Tradeoff: DHS too large: insufficient desktops to support things like Terminal Services DHS too small: insufficient heap to track all the processes Lawson creates “Deadlock Timeout” allows debugging of a race condition (thank Feld) Metrics [Windows] CPU: Use perfmon to check “CPU Work Queue Length” [Windows] Memory: Be sure Commit Charge < Physical Memory Reoccurring Maintenance Rotate Application log files monthly ladb.log ; latm.log ; lajs.log ; secadmin.log ; sec.log Turn over system log files monthly Event Log on Windows messages on Unix
17	Securing :: Outline Web Tier Application Tier Database Tier System
18	Securing :: Web Tier Add-ins Access Flag LX00/LX90 defines who can(not) access Lawson via Add-ins SSL Allow only HTTPS connections through the web Only Process Flow will not work with SSL Add a second virtual host for Process Flow Run everything else through HTTPS Close unnecessary ports If your users are all using Portal, and your system is on the Internet, only HTTPS (port 443) should be open Web server security Turn off directory browsing [Windows] Microsoft provides IIS lockdown tool
19	Securing :: Application Tier Auditing Security Involves cooperation between IT and Functional Areas Functional areas should ensure that Lawson security matches audit requirements Someone in AP who can cut checks cannot add/edit a vendor, for example Should be reviewed on a quarterly basis Feld maintains a summary security matrix to simplify this process LAUA System Code Security Form Security Data Security Drill-Around (Element) Security Backup your security classes with secdump, not just a backup of the secdb Inter-Application Security AP processors require an authority code HR can be further restricted with HR09 PO processors require a buyer code RW access controlled externally LOGAN Database Security RD30 records are not automatically removed when LAUA user is Stored in SISETUP, SIRDIDHDR, LOMXVALUE, and WEBUSER tables in LOGAN RD41 records (for Enterprise Reporting access) Stored in SISETUP, SIRDIDHDR and SIRDXREFDTL tables in LOGAN
20	Securing :: Database Tier Capital <database> File Has plain-text database username and password in it [Unix] Can be chmod’ed to 440 External Authentication Default Lawson install is “Application Pass-thru Security” One user defined in database who proxies for all users Access to data is controlled through the Application layer Lawson also supports External Database Authentication Each user defined in database No username/password stored in capital <database> file Pros: More granular access control and auditing Cons: LATM must be turned off, RMI will not work with LATM ODBC/JDBC Access Every RDBMS publishes itself as available on your network Ports are open for vendor client tools to connect remotely Leaving your RDBMS open for remote connections leaves your vulnerable [Unix] Close those ports and administer from a terminal session [Windows] Close your SQL Server ports and administer on the LAN from Terminal Services
21	Securing :: System Auditing Take advantage of OS level auditing [Windows] Turn on auditing of successful and failed logins [Unix] Review log files in /var/log or /var/adm [Unix] Confiscate the root password and start using sudo(8) Access [Unix] Do not allow access to your system via FTP If people need to be able to download print files, give a limited number of people access via scp [Windows] File and printer sharing should not be turned on If people need to be able to download print files, restrict who is allowed to brose to those shares Remove IPC and C$/D$ file shares from the system On-Disk Backups The disaster you plan for is not the most likely scenario Feld does 30 days of nightly, on-disk backups of: Databases: Application ; GEN ; LOGAN Windows Registry Each individual security class Why? Because administrators make mistakes, and restoring from tape is SLOW! If you have the disk space, make use of it Security Sweeps Periodic port scans with tools like nmap [Windows] Microsoft Baseline Security Analyzer (MS-BSA) Whatever OS or database you use, administrators should be receiving critical patch notifications from the vendor Security Sweeps should include your desktops: No amount of network or system security will protect you if your client machines are compromised Check for Trojans, spy bots, keystroke loggers
22	Blowout :: Questions?