Python Packaging

Brief History 

Python builds were traditionally done using:

• distutils - Python built-in, uses requirements.txt to list dependencies, setup.py for configuration
• ISSUES: split of deps and config could cause conflicts, and load order confusion
  
  
• setuptools - extended distutils, added automatic install of dependencies

Other packaging tools:

• wheel - PEP 427 - replaced eggs - build tool with better support for C-library integrations
• creates .pyc files during installation, to match local interpreter
• supported in ip >= 1.4 and setuptools >= 0.8
• PyPI - Python Package Index - online registry of built py libs

among others.

Poetry

Adds main project descritor pyproject.toml to contain project metadata, config, and dependencies.

Some features:

• can declare different dependency sets, i.e. for prod, dev, and test
• creates file poetry.lock to define frozen dependencies

Useful Commands

> Create a New Project

poetry new my-package

> Installing existing Project - exclude some dependencies

poetry install --without test,docs

> Installing existing Project - only include some dependencies

poetry install --only dev

> Install or Fix existing Project - sync local to canonical

poetry install --sync

> Get Info about an Installed Lib

poetry show <package-name>

TOML Example

From https://towardsdatascience.com/ditch-requirements-use-poetry-00a936fe9b6d:

[tool.poetry]
name = "my_project"
version = "0.1.0"
description = "Description of my project"
package-mode = true
authors = [
"My Team <team@my_company.com>",
]

[tool.poetry.dependencies]
python = "~3.10"
fastapi = "^0.111.1"
pydantic = "^2.8.2"
Jinja2 = "^3.1.2"
uvicorn = {extras = ["standard"], version = "^0.30.3"}

[tool.poetry.group.dev.dependencies]
pytest = "^8.2.2"
black = "^24.3.0"
pre-commit = "^3.8.0"
mypy = "^1.11"
flake8 = "^7.1.0"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.flake8]
max-line-length = 99
extend-ignore = "E203"

Include From Source

Add the following to include a project in development as a dependency.

[[tool.poetry.source]]
name = "my-artifact-registry-repo"
url = "https://europe-west2-python.pkg.dev/my-gcpproject/my-af-repo/simple"
priority = "explicit"

Docker Cheat Sheet

----------------------------------------------------------------------
References
----------------------------------------------------------------------

Docker Engine -- runtime and ops https://docs.docker.com/engine/
Create Docker Containers -- build your own https://docs.docker.com/build/
Orchestrate Containers -- combine containers https://docs.docker.com/compose/

----------------------------------------------------------------------
Basic Docker - Overview
----------------------------------------------------------------------

Basic Docker Architecture

• Image as container - Binary images of s/w systems published as Containers
• Versioned - Containers are versioned
• Decentralized - Your server downloads/stores containers and their parents...warning, this gets large!

Basic Docker Lifecycle

• Pull containers to use or extend
• Run containers and expose local resources to them
• Extend existing containers to create custom run-anywhere binaries

----------------------------------------------------------------------
Basic Docker - Setup
----------------------------------------------------------------------

Install
sudo apt install docker.io
sudo apt install docker-compose

Running without sudo
Assign your user into the docker group. Add group if necessary.

sudo groupadd docker
sudo usermod -a -G docker myuser
# refresh groups or restart server

Other post-install tasks
See https://docs.docker.com/install/linux/linux-postinstall/

Configure - start on boot
On CentOS:
sudo systemctl [enable|disable] docker

----------------------------------------------------------------------
Docker service
----------------------------------------------------------------------

Starting the daemon
sudo service docker start

Start/Stop Docker Service
Debian:

sudo service docker start
CentOS:
sudo systemctl [start|stop] docker 

Test Docker Service
sudo docker run hello-world

----------------------------------------------------------------------
Managing Images
----------------------------------------------------------------------

List all Images
docker images

Image Info - i.e. How do I use this image?
docker image inspect <imageid>

Rename Image...by aliasing the image
docker tag <imageid> <alias>

Remove Image (untagging)
docker rmi <alias>

Remove Unused Images

docker image prune [-a]

Move docker image location off to another drive (for large files)

sudo service docker stop
ps faux|grep docker
sudo ls /var/lib/docker
mv /var/lib/docker <new location> 
sudo service docker start

----------------------------------------------------------------------
Running Containers
----------------------------------------------------------------------

See running containers

docker ps

Launch container from image
docker run -it [-d] [--rm] [--network <name>]  <image_id> 

where:
-d starts detached (daemon)
--rm removes on exit
--network network name to use (see `docker network ls`)

Retain image changes
docker commit <container_id> <group>/<alias>

Attach and execute shell
docker exec -it <container_id> bash

Build image from local Dockerfile
docker build . -t <alias>

Attach to running shell
docker attach <container_id>

Drop out of running shell - but leave running

ctl-p ctl-d

View container logging

docker logs <container_id>

----------------------------------------------------------------------
Creating Images
----------------------------------------------------------------------

Create a Dockerfile with a base image

Search DockerHub for a suitable base
FROM python:latest

Add some standard labels

LABEL org.opencontainers.image.title="Data Service" \

      org.opencontainers.image.version="0.0.1" \

      org.opencontainers.image.authors="aaa@baz.com,bbb@baz.com"

Add some content

ADD data-svc/ /home/python/data-svc/.

Open port for web service

EXPOSE 8080

Attach volumes for in/out of data

VOLUME ["/home/python/data", "/var/log/data-svc"]

Launch a service on container start

ENTRYPOINT ["flask", "--app", "data-svc", "run"]

Build the Docker Image

From the directory containing Dockerfile...

docker build -t <group>/<artifact>:<version> .

----------------------------------------------------------------------
Docker and Docker-Compose Runtime
----------------------------------------------------------------------

Start docker-compose containers
docker-compose up -d

My Default vimrc File

 Finally putting this down, because I keep re-creating it.

The .vimrc file which normally goes in your user home dir.  There is a default in the vim install dir as well.

Suggestions welcome!

"For gVim - real GUI
set nocompatible
source $VIMRUNTIME/vimrc_example.vim
source $VIMRUNTIME/mswin.vim
behave mswin

" Disable viminfo
set viminfo=""

" to see all colors:
" ls -al /usr/share/vim/vim7*/colors
":colo evening
:colo blue
":colo slate
":colo darkblue
":colo shine

"set lines=45 columns=120

set tabstop=4 softtabstop=4 shiftwidth=4 expandtab autoindent smarttab
set nocindent
set encoding=utf-8 fileencoding=utf-8
set nobackup nowritebackup noswapfile autoread
set number ruler
set hlsearch incsearch ignorecase smartcase

"bash like auto-completion
set wildmenu
set wildmode=list:longest
"
inoremap
"
" for lusty explorer
noremap glr \lr
noremap glf \lf
noremap glb \lb

" use cntrl-h/j/k/l to switch between splits
map j
map k
map l
map h

" display the status line with the filename
set laststatus=2
set noerrorbells
set foldmethod=syntax
set ls=2
set mouse=v

" disable folding
set nofoldenable

HTTPS Connections with Groovy using Apache HTTPClient

Sample definition of Keystores, their usage and setting up of an HTTPS client connection.

Implemented as a Groovy script, meaning there is no top-level class per se. Could probably be more concise.

//imports removed for brevity


   // Housekeeping - see what the User wants
   //   Groovy note: not declaring a var puts it into "the binding", which makes it globally accessible
   //      * this is only valid for Groovy scripts
   //      * declaring a type (or def) of a var automatically scopes it locally
   config = new Config()
   parseInput(args)

   baseUrl = "https://" + config.host + ":" + config.port + "/myapp/"

   debug "\tBase URL: " + baseUrl

   println "Loading keys..."

   // Create client keystore - this needs to contain:
   //   Client's certificate (i.e., Subject CN == clientname)
   //   Client's private key
   // Needs to be a JKS keystore
   clientKeys  = createKeystore(config.clientKeyFile, config.clientKeyPass)
                                                                                          
   // Truststore for server verification - this needs to contain:
   //   Server certificate: Subject CN == hostname
   //   Server private key
   // Needs to be a JKS keystore
   serverTrust  = createKeystore(config.serverKeyFile, config.serverKeyPass)

   debug "\tclient key: " + clientKeys
   debug "\tserver key: " + serverTrust

   try {
      // ** Create Connection **
      socketFactory = new SSLSocketFactory(
                   SSLSocketFactory.TLS,
                   clientKeys,
                   config.clientKeyPass,
                   serverTrust,
                   null,
                   new TrustSelfSignedStrategy(),                 // not a good choice for production
                   SSLSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER); // not a good choice for production

      // Connect socket to enable client PKI authentication
      HttpParams params = new BasicHttpParams();
      params.setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 1000);

      SSLSocket socket = (SSLSocket) socketFactory.createSocket(params);
      socket.setEnabledCipherSuites("SSL_RSA_WITH_RC4_128_MD5");

      InetSocketAddress address = new InetSocketAddress(config.host, Integer.parseInt(config.port));
      socketFactory.connectSocket(socket, address, null, params);

      sch = new Scheme("https", new Integer(config.port), socketFactory);

      httpclient = new DefaultHttpClient();
      httpclient.getConnectionManager().getSchemeRegistry().register(sch);

      // Create a local instance of cookie store
      cookieStore = new BasicCookieStore();

      // Create local HTTP context
      localContext = new BasicHttpContext();
      // Bind custom cookie store to the local context
      localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);


      // ****** Example usage *******

      // This is the map-based impl - produces same output as the object-based
      // .... put your own data in here
      def data = [
         "_method" : "POST" ,
         "displayName" : row[DISPLAY_NAME],
         "imageUrlSmall" : row[IMAGE_URL_SMALL] ,
         "imageUrlLarge" : row[IMAGE_URL_LARGE],
         "width" : row[WIDTH],
         "height" : row[HEIGHT]
      ]

      String dataStr = JSONSerializer.toJSON(data)
      try {
         int retries = 5
         while (!created && retries-- > 0) {
         rsp = post (baseUrl + "widget", ["data":dataStr])

         matcher = rsp =~ /\"success\":([\w]+)/

         if (matcher[0]) {
            val =  matcher[0][1]
            debug "\tCREATE RESPONSE: ${val}"

            created = Boolean.parseBoolean(val)
         }

         // This is just for logging
         if (!created) {
            println "ERROR: Creation failed with response: ${rsp}"

            // Detect message failure and retry
            //   Detect failure by GUID invalid format
            matcher = rsp =~ /Property [name].*does not match the required pattern/
            if (matcher[0]) {
               debug "\tERROR RECEIVED:: name is not correct format: ${guid}"
            }
         }
         } // end while
      }
      catch (Exception e) {
          debug "Error creating listing: " + e
          e.printStackTrace()
          guid = null
      }
}

  /**
  *  Return a Keystore from a given filename and password
  */
  def KeyStore createKeystore (filename, password) {
      debug "Default KeyAlgo: " + KeyManagerFactory.getDefaultAlgorithm()
      debug "Default KeyStore: " + KeyStore.getDefaultType()  // jks

      // Pull off file extension
      keyType = filename[-3..-1]
      debug "Requested KeyType: " + keyType

      // Client keystore, for client authentication
      KeyStore keys
      switch (keyType) {
         case "p12":
            keys  = KeyStore.getInstance("pkcs12");
            break
         default:
            keys  = KeyStore.getInstance(KeyStore.getDefaultType());
      }

      FileInputStream instream = new FileInputStream(new File(filename));
      try {
         keys.load(instream, password.toCharArray());
      } finally {
         try { instream.close(); } catch (Exception ignore) {}
      }
      return keys
   }

  def post(url, paramMap) {
      HttpPost httpPost = new HttpPost(url)
      String result

      // Add specific params
      List  nvps = new ArrayList ();
      // These params always present
      nvps.add(new BasicNameValuePair("sample1", "3.6.0-GA"))
      nvps.add(new BasicNameValuePair("dojo.preventCache", "1302893700594"))

      paramMap.each() { key, value ->
         debug "${key} ==> ${value}"
         // Expect the caller to JSON-ize each param, as required
         if (value instanceof List) {
             // Add value multiple times with same key
            value.each {
               nvps.add(new BasicNameValuePair(key, it.toString()))
            }
         }
         else {
            nvps.add(new BasicNameValuePair(key, value))
         }
      }

      httpPost.setEntity(new UrlEncodedFormEntity(nvps, HTTP.UTF_8))

      // Debug
      debug(dump(httpPost))

      debug "Posting request: " + httpPost
      HttpResponse response = httpclient.execute(httpPost, localContext);
      //String response = httpclient.execute(httpPost, new BasicResponseHandler());

      debug "#### RESPONSE TYPE: " + response.class.name

      HttpEntity entity = response.getEntity();

      debug "#### RESPONSE BODY RECEIVED: " + entity.toString()

      // ** Dump results **
      debug "\t" + response.getStatusLine()
      StringBuilder sb = new StringBuilder(255)
      if (entity != null) {

         InputStream ris = entity.getContent()
         OutputStream os = new ByteArrayOutputStream(BUFFER_SIZE)
         byte[] buffer = new byte[BUFFER_SIZE];
         try {
            while ((l = ris.read(buffer)) != -1) {
                os.write(buffer, 0, l)
                sb.append(os.toString())
                os.reset()
            }
         } finally {
             ris.close()
         }

         debug "\tEntity contentLength: " + sb.length()

         result = sb?.toString()

         debug "\t-------------------------------------"
         List cookies = cookieStore.getCookies();
         for (int i = 0; i < cookies.size(); i++) {
            println("\tCookie: " + cookies.get(i));
         }                                                                                    
         debug "\t-------------------------------------"

      }
      else {
          println "Error connecting...check connection details"
          //!! EXIT HERE
          return
      }

      debug "-------------------------------------"
      if (result && result.size() > 256) {
         debug "RESPONSE: " + result.substring(0, 256)
      }
      else {
         debug "RESPONSE: " + result
      }
      debug "-------------------------------------"

      return result
  }

   def debug(stmt) {
       if (config.verbose) {
           println stmt
       }
   }

   def dump(HttpPost post) {
       StringBuilder sb = new StringBuilder(128)

       sb.append("\n\t------------------------------")
       sb.append("\n\tRequestURL: ").append(post.getURI())
       sb.append("\n\tType: ").append(post.getMethod())
       sb.append("\n\tParams: ").append(post.getParams())
       org.apache.http.params.BasicHttpParams
       sb.append("\n\t------------------------------")

       return sb.toString()
   }
         
  class Config {
      String host = "localhost"
      String port = "8443"
      boolean verbose = false
      String inputFile
      String clientKeyFile
      String clientKeyPass
      String serverKeyFile
      String serverKeyPass
  }

   def parseInput(args) {
       if (!args || args.length < 1) {
           usage()
       }
       for(int i=0; i < args.length; i++) {
           it = args[i]
           println "Processing arg: " + it
           switch (it) {
           case "-v":
               debug "Verbose output enabled "
               config.verbose = true
               break
           case "-h":
               it = args[++i]
               debug "Setting Host/Port from: " + it
               // Parse out host and port
               def serverAddr = it.split(":")
               try {
                  if (serverAddr.length > 0 && serverAddr[0] != null) {
                     config.host = serverAddr[0]
                  }
                  if (serverAddr.length > 1 && serverAddr[1] != null) {
                     config.port = serverAddr[1]
                  }
               } catch(Exception e) {
                   usage()
               }
               break
           case "-clientKeys":
               it = args[++i]
               debug "Setting client keyfile to: " + it
               config.clientKeyFile = it
               break
           case "-clientKeyPass":
               it = args[++i]
               debug "Setting client keyfile pwd to: " + it
               config.clientKeyPass = it
               break
           case "-serverKeys":
               it = args[++i]
               debug "Setting server keyfile to: " + it
               config.serverKeyFile = it
               break
           case "-serverKeyPass":
               it = args[++i]
               debug "Setting server keyfile pwd to: " + it
               config.serverKeyPass = it
               break
           default:
               debug "Setting inputFile to: " + it
               config.inputFile = it;
           }
       }
   }

Log4j Sample Config

I often seem to need a reference to log4j and can't find anything online that matches up with my expected results.

So here is my sample from a JBoss server that produces output logs:

log/server.log
log/proj-synchronization.log
log/proj-long-queries.log

What I like about this config is:

• logs asynchronously
• allows tuning of debug down to specific packages
• shows routing of different packages to different output logs

NOTE: many blocks are commented out and left in as examples -- I've tried to denote these by separating their header line.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">

<!-- ===================================================================== -->
<!--                                                                       -->
<!--  Log4j Configuration                                                  -->
<!--                                                                       -->
<!-- ===================================================================== -->

<!-- $Id: log4j.xml 56612 2006-09-07 15:12:39Z thomas.diesler@jboss.com $ -->

<!--
   | For more configuration information and examples see the Jakarta Log4j
   | website: http://jakarta.apache.org/log4j
 -->

<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false">

   <!-- ================================= -->
   <!-- Preserve messages in a local file -->
   <!-- ================================= -->

   <!-- A time/date based rolling appender -->
   <appender name="FILE" class="org.jboss.logging.appender.DailyRollingFileAppender">
      <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
      <param name="File" value="${jboss.server.log.dir}/server.log"/>
      <param name="Append" value="false"/>
      <param name="Threshold" value="DEBUG"/>

     <param name="MaxFileSize" value="100KB"/>

      <!-- Rollover at midnight each day -->
      <param name="DatePattern" value="'.'yyyy-MM-dd"/>

      <!-- Rollover at the top of each hour
      <param name="DatePattern" value="'.'yyyy-MM-dd-HH"/>
      -->

      <layout class="org.apache.log4j.PatternLayout">
         <!-- The default pattern: Date Priority [Category] Message\n -->
         <param name="ConversionPattern" value="%d [%t] %-5p [%c] %m%n"/>

         <!-- The full pattern: Date MS Priority [Category] (Thread:NDC) Message\n
         <param name="ConversionPattern" value="%d %-5r %-5p [%c] (%t:%x) %m%n"/>
          -->
      </layout>
   </appender>

   <appender name="PROJ-SYNCHRONIZATION" class="org.jboss.logging.appender.DailyRollingFileAppender">
      <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
      <param name="File" value="${jboss.server.home.dir}/log/proj-synchronization.log"/>
      <param name="Append" value="true"/>
      <param name="DatePattern" value="'.'yyyy-MM-dd"/>
      <layout class="org.apache.log4j.PatternLayout">
         <param name="ConversionPattern" value="%d [%t] %-5p [%c] %m%n"/>
      </layout>
   </appender>

   <appender name="PROJ-WEBSERVICE" class="org.jboss.logging.appender.DailyRollingFileAppender">
      <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
      <param name="File" value="${jboss.server.home.dir}/log/proj-webservice.log"/>
      <param name="Append" value="true"/>
      <param name="DatePattern" value="'.'yyyy-MM-dd"/>
      <layout class="org.apache.log4j.PatternLayout">
         <param name="ConversionPattern" value="%d [%t] %-5p [%c] %m%n"/>
      </layout>
   </appender>

   <appender name="QUERY" class="org.jboss.logging.appender.DailyRollingFileAppender">
      <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
      <param name="File" value="${jboss.server.home.dir}/log/proj-long-queries.log"/>
      <param name="Append" value="true"/>
      <param name="DatePattern" value="'.'yyyy-MM-dd"/>
      <layout class="org.apache.log4j.PatternLayout">
         <param name="ConversionPattern" value="%d [%t] %-5p [%c] %m%n"/>
      </layout>
   </appender>


   <!-- A size based file rolling appender

   <appender name="FILE" class="org.jboss.logging.appender.RollingFileAppender">
     <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
     <param name="File" value="${jboss.server.log.dir}/server.log"/>
     <param name="Append" value="false"/>
     <param name="MaxFileSize" value="500KB"/>
     <param name="MaxBackupIndex" value="1"/>

     <layout class="org.apache.log4j.PatternLayout">
       <param name="ConversionPattern" value="%d %-5p [%c] %m%n"/>
     </layout>       
   </appender>
   -->

   <!-- ============================== -->
   <!-- Append messages to the console -->
   <!-- ============================== -->
   <appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
      <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
      <param name="Target" value="System.out"/>
      <param name="Threshold" value="INFO"/>

      <layout class="org.apache.log4j.PatternLayout">
         <param name="ConversionPattern" value="%d{ABSOLUTE} %-5p [%c{1}] %m%n"/>
      </layout>
   </appender>

   <!-- ====================== -->
   <!-- More Appender examples -->
   <!-- ====================== -->

   <!-- Buffer events and log them asynchronously -->
   <appender name="ASYNC" class="org.apache.log4j.AsyncAppender">
     <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
     <appender-ref ref="FILE"/>
     <appender-ref ref="CONSOLE"/>
     <appender-ref ref="SMTP"/>
   </appender>

   <!-- EMail events to an administrator

   <appender name="SMTP" class="org.apache.log4j.net.SMTPAppender">
     <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
     <param name="Threshold" value="ERROR"/>
     <param name="To" value="admin@myhost.domain.com"/>
     <param name="From" value="nobody@myhost.domain.com"/>
     <param name="Subject" value="JBoss Sever Errors"/>
     <param name="SMTPHost" value="localhost"/>
     <param name="BufferSize" value="10"/>
     <layout class="org.apache.log4j.PatternLayout">
       <param name="ConversionPattern" value="[%d{ABSOLUTE},%c{1}] %m%n"/>
     </layout>
   </appender>
   -->

   <!-- Syslog events

   <appender name="SYSLOG" class="org.apache.log4j.net.SyslogAppender">
     <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
     <param name="Facility" value="LOCAL7"/>
     <param name="FacilityPrinting" value="true"/>
     <param name="SyslogHost" value="localhost"/>
     <layout class="org.apache.log4j.PatternLayout">
       <param name="ConversionPattern" value="[%d{ABSOLUTE},%c{1}] %m%n"/>
     </layout>
   </appender>
   -->

   <!-- Log events to JMS (requires a topic to be created)

   <appender name="JMS" class="org.apache.log4j.net.JMSAppender">
     <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
     <param name="Threshold" value="ERROR"/>
     <param name="TopicConnectionFactoryBindingName" value="java:/ConnectionFactory"/>
     <param name="TopicBindingName" value="topic/MyErrorsTopic"/>
   </appender>
   -->

   <!-- Log events through SNMP

   <appender name="TRAP_LOG" class="org.apache.log4j.ext.SNMPTrapAppender">
     <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
     <param name="ImplementationClassName" value="org.apache.log4j.ext.JoeSNMPTrapSender"/>
     <param name="ManagementHost" value="127.0.0.1"/>
     <param name="ManagementHostTrapListenPort" value="162"/>
     <param name="EnterpriseOID" value="1.3.6.1.4.1.24.0"/>
     <param name="LocalIPAddress" value="127.0.0.1"/>
     <param name="LocalTrapSendPort" value="161"/>
     <param name="GenericTrapType" value="6"/>
     <param name="SpecificTrapType" value="12345678"/>
     <param name="CommunityString" value="public"/>
     <param name="ForwardStackDEBUGWithTrap" value="true"/>
     <param name="Threshold" value="DEBUG"/>
     <param name="ApplicationTrapOID" value="1.3.6.1.4.1.24.12.10.22.64"/>
     <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d,%p,[%t],[%c],%m%n"/>
     </layout>
   </appender>
   -->

   <!--  Emit events as JMX notifications

   <appender name="JMX" class="org.jboss.monitor.services.JMXNotificationAppender">
      <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
     
      <param name="Threshold" value="WARN"/>
      <param name="ObjectName" value="jboss.system:service=Logging,type=JMXNotificationAppender"/>
     
      <layout class="org.apache.log4j.PatternLayout">
         <param name="ConversionPattern" value="%d %-5p [%c] %m"/>
      </layout>
   </appender>
   -->
  
   <!-- ================ -->
   <!-- Limit categories -->
   <!-- ================ -->

   <!-- Limit the org.apache category to INFO as its DEBUG is verbose -->
   <category name="org.apache">
      <priority value="ERROR"/>
   </category>

   <category name="org.apache.log4j">
      <priority value="DEBUG"/>
   </category>


   <!-- Limit the org.jboss.serial (jboss-serialization) to INFO as its DEBUG is verbose -->
   <category name="org.jboss.serial">
      <priority value="ERROR"/>
   </category>

   <!-- Limit the org.jgroups category to WARN as its INFO is verbose -->
   <category name="org.jgroups">
      <priority value="ERROR"/>
   </category>

   <!-- Limit the jacorb category to WARN as its INFO is verbose -->
   <category name="jacorb">
      <priority value="ERROR"/>
   </category>

   <!-- Limit JBoss categories  -->
   <category name="org.jboss">
      <priority value="ERROR"/>
   </category>

   <category name="org.springframework">
      <priority value="ERROR"/>
   </category>

   <category name="org.springframework.web">
      <priority value="error"/>
   </category>

   <category name="org.springframework.scheduling">
      <priority value="error"/>
   </category>

   <category name="org.quartz">
      <priority value="error"/>
   </category>

   <category name="javawebparts">
      <priority value="ERROR"/>
   </category>

   <!-- Limit the JSR77 categories -->
   <category name="org.jboss.management">
      <priority value="ERROR"/>
   </category>

   <category name="org.hibernate">
      <priority value="warn"/>
   </category>

   <category name="org.hibernate.SQL">
      <priority value="warn"/>
   </category>

   <category name="net.sf.ehcache">
      <priority value="ERROR"/>
   </category>

   <category name="com.proj.ws">
      <priority value="error"/>
   </category>

   <category name="com.myapp">
      <priority value="DEBUG"/>
   </category>
   <category name="com.myapp.service">
      <priority value="DEBUG"/>
   </category>
   <category name="com.myapp.spring">
      <priority value="DEBUG"/>
   </category>

   <category name="com.myapp.service.UserService">
      <priority value="error"/>
   </category>

   <!-- Flooring utils are not usually useful in troubleshooting -->
   <category name="com.myapp.util">
      <priority value="DEBUG"/>
   </category>

   <category name="com.myapp.servlet.filter">
      <priority value="DEBUG"/>
   </category>

   <category name="com.proj.ws.synchronization" additivity="false">
      <priority value="error"/>
      <appender-ref ref="PROJ-SYNCHRONIZATION"/>
   </category>

   <category name="com.proj.ws.longQuery" additivity="false">
      <priority value="ERROR"/>
      <appender-ref ref="QUERY"/>
   </category>

   <!-- Enable JBossWS message tracing
    <priority value="error" class="org.jboss.logging.XLevel"/>
   -->
   <category name="jbossws.SOAPMessage">
     <priority value="error"/>
   </category>

   <!-- Decrease the priority threshold for the org.jboss.varia category -->
   <category name="org.jboss.varia">
     <priority value="error"/>
   </category>

   <!-- Show the evolution of the DataSource pool in the logs [inUse/Available/Max]

   <category name="org.jboss.resource.connectionmanager.JBossManagedConnectionPool">
     <priority value="error" class="org.jboss.logging.XLevel"/>
   </category>
   -->

   <!--

      | An example of enabling the custom DEBUG level priority that is used
      | by the JBoss internals to diagnose low level details. This example
      | turns on DEBUG level msgs for the org.jboss.ejb.plugins package and its
      | subpackages. This will produce A LOT of logging output.
   <category name="org.jboss.system">
     <priority value="DEBUG" class="org.jboss.logging.XLevel"/>
   </category>
   <category name="org.jboss.ejb.plugins">
     <priority value="DEBUG" class="org.jboss.logging.XLevel"/>
   </category>
   -->
 
   <!--
       | Logs these events to SNMP:

           - server starts/stops
           - cluster evolution (node death/startup)
           - When an EJB archive is deployed (and associated verified messages)
           - When an EAR archive is deployed
         
   <category name="org.jboss.system.server.Server">
     <priority value="INFO" />
     <appender-ref ref="TRAP_LOG"/>
   </category>
 
   <category name="org.jboss.ha.framework.interfaces.HAPartition.lifecycle">
     <priority value="INFO" />
     <appender-ref ref="TRAP_LOG"/>
   </category>

   <category name="org.jboss.deployment.MainDeployer">
     <priority value="ERROR" />
     <appender-ref ref="TRAP_LOG"/>
   </category>
  
   <category name="org.jboss.ejb.EJBDeployer">
     <priority value="INFO" />
     <appender-ref ref="TRAP_LOG"/>
   </category>
  
   <category name="org.jboss.deployment.EARDeployer">
     <priority value="INFO" />
     <appender-ref ref="TRAP_LOG"/>
   </category>
 
   -->

   <!-- ======================= -->
   <!-- Setup the Root category -->
   <!-- ======================= -->

   <root>
       <appender-ref ref="FILE"/>
       <appender-ref ref="CONSOLE"/>
       <!-- use if routing all categories to an appender-ref <level value="OFF" /> -->
   </root>

   <!-- Clustering logging -->

   <!-- Uncomment the following to redirect the org.jgroups and
      org.jboss.ha categories to a cluster.log file.

   <appender name="CLUSTER" class="org.jboss.logging.appender.RollingFileAppender">
     <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
     <param name="File" value="${jboss.server.log.dir}/cluster.log"/>
     <param name="Append" value="false"/>
     <param name="MaxFileSize" value="500KB"/>
     <param name="MaxBackupIndex" value="1"/>

     <layout class="org.apache.log4j.PatternLayout">
       <param name="ConversionPattern" value="%d %-5p [%c] %m%n"/>
     </layout>
   </appender>
   <category name="org.jgroups">
     <priority value="DEBUG" />
     <appender-ref ref="CLUSTER"/>
   </category>
   <category name="org.jboss.ha">
     <priority value="DEBUG" />
     <appender-ref ref="CLUSTER"/>
   </category>
   -->

</log4j:configuration>

Viewing Markdown files in Firefox on Linux

There are Firefox plugins for viewing Markdown files - but neither of the big ones - Markdown Editor and Markdown Viewer - worked at all on my linux distro (Mint 15), even with the mimetypes.rdf workaround. There are of course editors like UberWriter that will do some special formatting of MD files, but none I have found really track well to the actual HTML produced by the browser plugins. Since many of my colleagues use Firefox on Windows where the plugins work, I wanted to know what they would look like. In desperation I hacked a quick and dirty solution using a Python translator, markdown2.

• first install pip
  
  sudo apt-get install python-pip
  
• install markdown2
  
  pip install markdown2
• for me this had already run 'python setup.py install'
  • if it doesn't for you then run manually
• test at command line - should be available
  
  markdown2
• now try all together - assumes firefox is on your path
  
  markdown2 README.MD > README.html; firefox README.html
• add as a shell alias - modify ~/.bashrc and add
  
  

     function mark() { 
        markdown2 "$@" > "$@".html; 
       firefox "$@".html
     }
  

• now try
  

  source ~/.bashrc 
  mark README.MD
  

PKI Security Cheat Sheet

This is a work in progress.

----------------------------------------------------------------------
Using OpenSSL - most common activities
----------------------------------------------------------------------
Generally used for X509 artifacts, i.e. the more open standard.

Dump X509 certificate(CRT) content - assumes PEM format
openssl x509 -in certificate.crt -text -noout

Dump X509 certificate(CRT) content - specify input format, PEM/DER
openssl x509 -inform DER -in site.crt
NB:Try changing the format on error: "Expecting: TRUSTED CERTIFICATE"

Dump a pkcs12 user identity certificate
openssl pkcs12 -info -in keyStore.p12

Dump private key content
openssl rsa -in host.key -text

----------------------------------------------------------------------
Using OpenSSL - creating and modifying keys
----------------------------------------------------------------------

Create a private key
openssl req -out CSR.csr -new -newkey rsa:2048 -nodes -keyout privateKey.key

----------------------------------------------------------------------
Using keytool - most common activities
----------------------------------------------------------------------
Generally used for working with Java keystore(JKS) files.

List contents of a JKS
keytool -list -v -keystore keystore.jks

Dump a cert
keytool -printcert -v -file host.crt

Export a cert from a JKS for given alias
keytool -export -alias sitename -file sitename.crt -keystore keystore.jks

List default JVM CA certs
keytool -list -v -keystore $jAVA_HOME/jre/lib/security/cacerts

----------------------------------------------------------------------
Debugging an SSL Connection
----------------------------------------------------------------------
You are trying to set up a Java webserver fronting SSL and having issues.

Test the connection using openSSL to see what SSL it supports
openssl s_client -connect mysite.com:443

Enable SSL debug
Add the following to the JVM startup command:
-Djavax.net.debug=[ssl|all]

and see this to understand the output.
This will often lead you to the cause of the connection issues.

----------------------------------------------------------------------
Resources
----------------------------------------------------------------------