perrys_head.select{ |thought| thought.tidbit }

mort_calc gem

2010-02-16T15:35:00.002-06:00

I recently published the mort_calc gem at gemcutter.org. The code can be found at http://github.com/perry3819/mort_calc/.

The gem calculates the APR and monthly payment for a mortgage in the United States.

Calculating the monthly payment is straight forward.

C = Loan amount

E = Extra costs

r = monthly interest rate = interest rate / 1200

N = amortization term in months

An iterative approach is needed to find the APR. The equation for the APR follows.

a = APR/1200

N = amortization term in months

P = Monthly payment (including all fees paid by borrower)

C = Loan amount

The following graph shows the APR calculation plotted for N = 360, P = 2500, and C = 400,000. The APR is where the line crosses the A axis.

I chose the Newton-Raphson method to quickly find a precise solution.

The MortgageUtil.calculate_apr method provides a starting value of the interest rate, which makes the Newton-Raphson converge on a accurate solution between 1 and 3 passes.

Installation

% sudo gem install gemcutter

% gem tumble

% sudo gem install mortgage_calc

Example

MySecrets

2009-11-12T21:35:00.003-06:00

I developed MySecrets to manages my personal secrets/passwords. I had a lot of fun doing BDD with cucumber. Pickle works very nicely with Machinist to reduce step definition code. I used authlogic for authentication and formtastic to reduce the boiler plate code in my views.

I had some trouble integrating cucumber, pickle, machinist, and authlogic. A user has 0 to many secrets and secrets are protected by authlogic. So cucumber features that needed to add secrets and validate them were a little tricky.

This scenario creates a user, login in, creates a secret, shows the secret, and finally validates the secret.

The show pages are handled with a polymorphic_path.

The actual user creation and login steps are handled here.

The secret is validated with:

All of the coded can be found at http://github.com/perry3819/mysecrets

pwfoo - Ruby Gem

2009-11-12T18:46:00.004-06:00

pwfoo is a gemcutter.org hosted gem that I developed to generate random passwords, score the strength of passwords, and generate random seeds.

Generate password

Determine password strength

The SrandSeedGenerator is interesting because it uses the current time and currently running processes to generate random seeds.

pwfoo can be installed from gemcutter.org.

NHibernate Optimization - Part 2 - Data Access Object (DAO)

2009-10-28T21:38:00.005-05:00

In the first post on NHibernate Optimization, mapping file configuration was discussed. This post will focus on code optimization in data access objects (DAOs).

Load objects by ID.
Object should be loaded by ID whenever possible to benefit from first-level cache (session cache). Whenever an object is saved or loaded it is added to NHibernate's first-level cache with the ID as the key. As a reminder when NHibernate loads an object, it first looks in the first-level cache, then query cache, followed by second-level cache and finally SQL is executed. Therefore, loading an object by ID will be very quick if it is already in first-level cache.

Sometimes the UI layer needs to request an object with a business key rather than a surrogate key. In this case, a thread-safe Dictionary with key=business key and value=surrogate key should be considered. The UI layer requests the object by business key from the service layer. The service layer finds the surrogate key in the Dictionary and uses it to query the DAO. This is a little extra work, but can reap appreciable performance benefits.

Use HQL, DTOs, and/or hand-coded ADO.NET in bottlenecks.
Oftentimes the UI layer presents a screen that summarizes the state of many different objects, which can force many separate and expensive database calls that return much more data that is actually necessary. A good way to optimize this is to use HQL or hand-coded ADO.NET to SELECT only the required data, using aggregate functions if necessary, and return the data in a non-hibernate mapped data transfer object (DTO).

This post scratched the surface of code optimizations in the data access layer. Please stand by for upcoming code optimization recommendations.

Interval Rails Caching

2009-10-26T13:41:00.017-05:00

Feedpoint (source) is a blog aggregator written with Ruby on Rails deployed on a linux server. It retrieves RSS content from blogs and displays them sorted by date at http://feedpoint.hertler.org. Retrieving the RSS content is an expensive step and caching is needed to speed up HTTP requests. Google searching for a way to expire page caching on a timed interval did not yield promising results so I developed a very easy custom strategy through the use of Linux cron jobs.

The Rails code for the page caching is in home_controller.rb.

caches_page and expire_page are ActionController methods. caches_page caches the the :index and :planet_redpoint actions and expire_page expires them.

With this setup, http://feedpoint.hertler.org/home/ will load the :index into page cache and http://feedpoint.hertler.org/home/expire_cache will expire the cache. At this point, monkeys could be trained to navigate to the expire_cache URL every ten minutes or a cron job could be set up. This post will focus on setting up the cron job.

The cron job was configured by entering crontab -e from a terminal and then adding the following line:

This job uses cURL to call the expire_page action every ten minutes. Good cron tutorials are all over the web and beyond the scope of this post.

One added benefit of this approach is that if a blogger is anxious for a recently posted blog to show up on the world famous http://feedpoint.hertler.org site, it can manually be refreshed right away!

NHibernate Optimization - Part 1 - Mapping Files

2009-10-26T09:31:00.003-05:00

NHibernate is a powerful object/relational persistence and query service that has deep market penetration in the Java space and is rapidly gaining acceptance in the .Net community. NHibernate is particular beneficial to domain-driven projects as developers can focus on good OO design and have the database automatically generated from the domain model. Since developers are focused on classes and not tables, performance can suffer because little thought is given to how SQL is generated and executed. This is the first post in a series that will focus on strategies to monitor performance, best practices, and optimizing N/Hibernate code.

The Problem
Improper data access code and/or hibernate mapping files can cause NHibernate to execute inefficient queries, execute the same queries over and over, and/or retrieve much more data than needed. The three main areas of concern when dealing with Nhibernate performance are the mapping files, data access objects (DAOs), and monitoring/profiling tools.

Mapping file optimization
Improper fetch strategies are the most common mistakes in Nhibernate mapping files. Many times developers simply copy and paste an existing mapping file and just change the table and column names without giving any thought to fetching strategies used. As a general rule, lazy associations should be used by default because no matter what fetching strategy is used, non-lazy relationships will be fully loaded into memory executing SQL queries if necessary.

The Nhibernate developer should read and understand the fetching strategy section of the reference documentation. Batch fetching and subselect fetching can significantly reduce the number of SQL queries executed.

Another mapping element that can speed up the NHibernate application is second-level caching. A good idea for all applications is to enable read-only cache on "look-up" data that never changes. Nonstrict-read-write works well for objects that are mostly read and seldom written. More on caching will follow in upcoming posts.

In summary, use lazy associations by default, choose the write fetching strategy, and include second-level caching when appropriate.

chroot sftp for blogger.com Tutorial

2009-10-23T10:39:00.003-05:00

I use blogger.com to author and publish this blog. After clicking the PUBLISH POST button, blogger.com moves the post to hertler.org, which means that I need to give blogger.com access to my server.

To do this securely, I only allow blogger.com access to hertler.org through SFTP and I chroot the blogger.com user to /home/blogger so that only blog files are permitted to be updated. I set this up on my ubuntu virtual machine through hostingrails.com, which by the way is a fantastic hosting provider.

Here are the steps to chroot blogger.com:
1) Verify that you have an OpenSSH >= 4.9. Update OpenSSH if needed.

root@web1:/# ssh -V
OpenSSH_5.1p1 Debian-3ubuntu1, OpenSSL 0.9.8g 19 Oct 2007

2) Create the blogger.com user and change the owner and group of its home directory to root.

root@web1:/# /usr/sbin/useradd blogger
root@web1:/# passwd blogger   
root@web1:/# chown -R root.root /home/blogger

3) Edit /etc/ssh/sshd_config to set up the chroot.

Subsystem sftp internal-sftp
Match user blogger
      ChrootDirectory /home/blogger
      AllowTcpForwarding no
      ForceCommand internal-sftp

Be sure to delete Subsystem sftp /usr/lib/openssh/sftp-server. More information on

4) Restart ssh.

/etc/init.d/ssh restart

5) Verify that you can sftp in with the blogger user and that the root directory for the user is /home/blogger.

perry-hertlers-mac-pro:~ perry$ sftp blogger@hertler.org
Connecting to hertler.org...
blogger@hertler.org's password: 
sftp> pwd
Remote working directory: /
sftp> ls
testfileinbloggerhomedir.txt           
sftp>

OK, now we have the chrooted sftp access for blogger set up, but we still need to server those files on the web site. We can simply set up a symbolic link to /home/blogger from our virtual host directory.

root@web1:/# ln -s /home/blogger /var/www/blog.hertler.org/current

Now we are securely giving blogger.com sftp access to our web server.

britetab.com

2009-08-14T12:43:00.004-05:00

I started working on a new project last week for britetab.com. which allows one to create and share resumes using custom images, photos, and videos.

My britetab resume, shows just a few of the available features.

Install mysql gem on OS X 10.5

2009-08-10T12:38:00.003-05:00

I had a little trouble installing the mysql gem on OS X 10.5 this morning so I thought I would post the solution.

1) Download the mysql binaries from apple http://support.apple.com/kb/TA25017?viewlocale=en_US
2) Unpack the binaries.
3) sudo gem install mysql -- --with-mysql-config=/usr/bin/mysql_config (be sure to give an absolute path to the config file)
4) done

Feedpoint - Rails Blog Aggregator

2009-08-10T11:28:00.014-05:00

I finished up the 0.1 release of feedpoint.

Feedpoint is a blog aggregator. Companies may find it useful for showcasing blogs of its employees.

So far, the whole application is written with Ruby on Rails. Additional front-end frameworks are planned including JavaFX and Silverlight.

Privileged users may add new feeds through a web interface.

The main screen displays all of the posts sorted by date.

TFS - Stop and Start Server

2009-07-24T13:43:00.005-05:00

After some failed attempts at stopping and starting servers through third party tasks, I had quick success using the exec command in the TFSBuild.proj:


<PropertyGroup>
    <web-site-id>1</web-site-id>
    <server-cmd-prefix>C:\Windows\system32\cscript.exe c:\Inetpub\AdminScripts\adsutil.vbs</server-cmd-prefix>
    <stop-server-cmd-line>$(server-cmd-prefix) STOP_SERVER W3SVC/$(web-site-id)</stop-server-cmd-line>
    <start-server-cmd-line>$(server-cmd-prefix) START_SERVER W3SVC/$(web-site-id)</start-server-cmd-line>
 </PropertyGroup>
  
 <Target Name="StopSite">
    <Message Text="stopping site..."/>
    <Exec Command="$(stop-server-cmd-line)"/>
 </Target>

 <Target Name="StartSite">
    <Message Text="starting site..."/>
    <Exec Command="$(start-server-cmd-line)"/>
 </Target>

To find the web-site-id, refer to Scott Forsyth's Blog.

AES Encryption Wrapper for Ruby

2009-07-20T11:24:00.017-05:00

I thought I would share my AES encryption wrapper for Ruby. The attached code utilizes openssl.

require 'openssl'

module AESCrypt
  DEFAULT_CIPHER_TYPE = 'aes-256-cbc'
  CRYPT_STRUCT = Struct.new(:encrypted_data, :key, :iv)

  def AESCrypt.encrypt(plain_text, opts={})
    cipher = OpenSSL::Cipher::Cipher.new(DEFAULT_CIPHER_TYPE)
    cipher.encrypt
    opts.reverse_merge! :key => cipher.random_key, :iv => cipher.random_iv
    cipher.key = random_key = opts[:key]
    cipher.iv = random_iv = opts[:iv]
    encrypted_data = cipher.update(plain_text)
    encrypted_data << cipher.final
    CRYPT_STRUCT.new(encrypted_data, random_key, random_iv)
  end

  def AESCrypt.decrypt(crypt_struct)
    cipher = OpenSSL::Cipher::Cipher.new(DEFAULT_CIPHER_TYPE)
    cipher.decrypt
    cipher.key = crypt_struct.key
    cipher.iv = crypt_struct.iv
    cipher.update(crypt_struct.encrypted_data) + cipher.final
  end
  
end

This code is pretty straightforward. One thing of interest is that we are using a random AES key and initialization vector on lines 10 and 11 respectively. This results in consumers of this API needing to store the key and initialization vector in order to decrypt, which is why a CRYPT_STRUCT is returned on line 14.

The source and test files are available at the my_password_manager github site.

When is "last"?

2009-07-13T16:17:00.010-05:00

One of the extreme programming rules is Optimize Last. The rule is stated as

Do not optimize until the end. Never try to guess what the system's bottle neck will be. Measure it!
Make it work, make it right, then make it fast.

When is "last"? Is it one week before the application goes into production, at the end of an iteration, at the end of the story, or at the end of a pairing session?

Another question: what and how much should be optimized?

Though the answers to these questions may very a little from project to project, in my experience "last" should mean both at the end of the stories and end of iteration and just the bottlenecks should be optimized with the aid of a profiling tool.

Optimization strategy:

Obtain non-functional performance requirements for the system very early in the project. If there is know target, are we not wasting our time optimizing? By the way, "as fast as possible" is not a sufficient requirement.
Set up automated performance tests. You may even want to go as far a having it run as part of your continuous integration build and having the build fail if certain performance thresholds are not met.
Near the end of story completion, profile the new code. I use ANTS Profiler for C# and JProfiler for Java. Obvious bottlenecks with minimal code changes for optimization should be implemented without hesitation. Continue to fix bottlenecks until the application is meeting its performance requirements.
Near the end of the iteration, profile the whole application. Consider having team members rotate through application-wide optimization responsibilities every iteration.

Selling Agile Development Practices to Your Manager

2009-07-07T11:23:00.005-05:00

There have been some comments that some of you developers are very interested in adopting agile development practices, but your manager isn't seeing the benefit.

There are two important points to make to your manager.

Incremental development practices prevent defects and catch them early
The cost of a defect drastically goes up the further along it is in the project life cycle

Incremental development practices are geared around preventing defects and catching them early with comprehensive tests, frequent integration, and knowledge sharing. Take a look at my Incremental Development Presentation for more information.

A simple google search yields a lot of evidence to support the high cost of defects. The cost of defects relative to project life-cycle looks a lot like the image below.

The Multi-core Problem

2009-07-06T15:29:00.003-05:00

If you don't have any experience with functional programming languages, do yourself a favor and take the time to watch this presentation by Joe Armstrong who is the Erlang programming language creator.

One of the main points that Mr. Armstrong makes is that as time goes on sequential programs will get slower because they do not utilize more than one core and current trends are to have more and slower cores to reduce power consumption.

Functional programming languages like Erlang can utilize as many cores as necessary because there is no mutable state, there is no shared memory, and actors are completely isolated.

FirstMem - a tool for verbatim text memorization

2009-07-06T14:51:00.003-05:00

I have recently spent some time learning Ruby and Rails. When I learn a new language I typically write an easy application so that the focus is not on solving a particularly difficult problem, but on learning the new technology. To learn Ruby on Rails, I wrote http://firstmem.hertler.org. The code is hosted at http://code.google.com/p/firstmem/

As I was writing the code I was constantly shocked at how little code it took to perform complex tasks and how easy the Rails framework is to learn. It almost felt like cheating!

Hudson Client for iPhone

2009-06-12T13:50:00.003-05:00

I was planning on using Hudson for CI anyway on my next project, but now Hudson Helper is making me really impatient!

Sweet Spot

2009-06-12T08:29:00.004-05:00

Bud Caddell constructed an interesting venn diagram and posted it on his blog. This motivated me to think about where I am right now in my career. I am probably in the region where What we do well and What we can be paid to do intersect, but probably not entirely in What we want to do.

This is normal for me. Throughout my career, I have constantly sought to learn new technologies and paradigms in my spare time and then utilize those skills professionally. This whole cycle occurs for me usually in less than a year. Right now, I am rapidly coming up to speed in functional programming languages because of not only their elegance, but the multi-core utilization problem they solve (more on this to follow). Functional programming is What I want to do and is probably What I can be paid to do, but is not yet something I can do well so I need to Learn more. I'm working on it....

TDD in the Real World Slides

2009-05-30T09:13:00.006-05:00

Here are the TDD in the Real World slides for the Chicago Code Camp presentation that I gave on May 30th.

Chicago Code Camp

2009-05-19T16:14:00.003-05:00

Be sure to attend Chicago Code Camp on May 30th at the College of Lake County (CLC), in Grayslake, IL.

I will be doing a presentation entitled TDD in the Real World. There will be a lot of real-time coding! It should be fun.

Do The Simplest Thing That Could Possibly Work (DTSTTCPW)

2009-02-23T09:26:00.009-06:00

DTSTTCPW means not writing code to "future requirements" and not implementing the current requirements in a fancy-overly-complicated-cool-sure-to-impress-peers manner. I am convinced that many designs are driven by how cool they would look in a power point presentation rather than pragmatism!

Some examples of DTSTTCPW violation include writing unnecessary reusable code, writing unneeded highly decoupled code, writing code that goes above and beyond requirements, and introducing a design pattern into the code with no immediate benefit (e.g. strategy pattern with only one strategy). The remainder of this blog entry will focus on writing unnecessary reusable code as this is the most common mistake.

On a recent project, we were losing a lot of time trying to integrate with an unsupported web service. Up to seven people were involved from different sides of the planet. I started asking questions like 1) what does the web service do? 2) where does it get its data? 3) what business logic does it perform? It turns out that the web service was just executing a stored procedure and performing no business logic. Even worse, the stored procedure what local to our application's database!

Apparently the motivation was that at some point in the future other applications would potentially reuse this web service. Many readers of this might think that this is smart because someone was planning for the future. Most people have been in meetings or peer reviews where a developer is congratulated on his forward thinking for implementing a highly reusable design. Only a few years ago, I would have been impressed, too. Now, ......not so much. The reasons:

Reusablity comes at a high cost - Implementing requirements in a way that takes into account other uses requires a lot of brainstorming and iterations to get right.
You are not going to need the reusability anyway. Writing from experience, these sure to happen future requirements seldom happen. In lean terms, reusability provides no value to the customer.
Added complexity - necessarily, a reusable design requires more complexity and probably more lines of code. This results in not only higher up front costs, but higher
maintenance costs.

Common objections:

Yeah, but I once wrote this reusable code that was used by many other developers on a wide variety of applications!

How many times did you write reusable code that wasn't reused?
How much did the reusable code cost the customer?
Did the customer benefit from the reusable code or just people within your organization, team, or department?
Were other teams forced to use your code even though it didn't really meet their requirements?

Sometimes DTSTTCPW results in poor performance or non-functional requirements not being satisfied.

Then you did not DTSTTCPW. Instead you DTSTT. Non-functional requirements need to be taken into account when deciding on the simplest thing that could possibly work.

Favor Composition Over Inheritance

2008-11-26T12:04:00.013-06:00

Put another way, favor "has-a" over "is-a" relationships.

What's so bad about inheritance?

Subclasses are tightly coupled to their base classes. Many times subclasses make assumptions about the context in which a method it overrides is getting called, which means that small implementation changes in the base class can produce unexpected behavior.
Dependencies of the subclass are hard-wired making unit testing difficult. One cannot mock a base class of a subclass.

It is definitely fine to inherit from interfaces; it's the concrete implementations that cause all of the problems.

An added benefit of composition relationships is that behavior can be changed at run-time (Strategy Pattern).

A(nother) Case for Pair-Programming

2008-11-11T15:06:00.011-06:00

The benefits of pair-programming include:

Faster
Fewer bugs
Higher quality code (continuous peer review and brainstorming)
Shared understanding of code-base
Can learn from partner
Better design
Team building
Higher confidence in code
Higher satisfaction

Drawbacks:

Scheduling problems (developer schedules are different)
Personality conflicts

Most studies show that a pair is 40 to 80% faster than an individual, which means that two individuals are faster than a pair. However, the resultant code is higher quality in that it has a better design and fewer bugs. It has been extensively documented that the cost and time it takes to fix bugs found by testers or customers is many times larger than the cost of the originally written code, which effectively means that over the whole project life-cycle pair-programming is faster.

Scheduling problems can be solved, but personality conflicts can be a real problem. A good pair programmer leaves his/her ego at the door, is flexible, and most importantly communicates well. One mitigation strategy is to frequently rotate pairs.

Testable Code

2008-11-03T09:44:00.003-06:00

One of the best ways to ensure testable code is practice Test Driven Development. How can code not be testable if it is already tested?

Another good idea is to write code that is easy to test in isolation, which is mostly done by mocking its dependencies. "Mockable" classes are achieved with:

Injecting dependencies with an Inversion of Control framework such as Spring
Avoiding singleton and static methods
Favoring composition over inheritance

Mock Example

2008-10-24T15:50:00.002-05:00

Mocking is a great way to test a class in isolation. Not only can determinisitic results of the mocked class be forced, but parameters to the mocked object's methods can be tested.

Here is a unit test that uses mockito as its mocking framework. It is organized in the increasingly popular Arrange / Act / Assert format. This tests a yet to be written subclass of Spring's AbstractJDomPayloadEndpoint.

  1 @Test
  2 public void validDepthOfField( ) throws Exception {
  3     stub( _depthOfFieldCalculator.calculateDepthOfField( _expectedDofParams ) ).toReturn( _depthOfField );
  4
  5     final PhotoEndpoint photoEndpoint =
  6         new PhotoEndpoint( _depthOfFieldCalculator );
  7     final Element dofElem =
  8         photoEndpoint.invokeInternal( _depthOfFieldRequest );
  9       
 10     Assert.assertEquals( _depthOfField.getFarLimit(),
 11             _parseDouble( dofElem, _farLimitExpression));
 12     Assert.assertEquals( _depthOfField.getNearLimit(),
 13             _parseDouble(dofElem, _nearLimitExpression));
 14     Assert.assertEquals( _depthOfField.getHyperFocalDistance(),
 15             _parseDouble(dofElem, _hyperFocusDistanceExpression));
 16 }

Line 3: _depthOfFieldCalculator was mocked with mockito in JUnit's @Before. If calculateDepthOfField(...) is called with a object matching _expectedDofParams(equals overridden), _depthOfField will be returned. Important to note that if the calculateDepthOfField is called with the wrong parameter, the mock will return null.
Lines 5-6: Create a class that does not yet exist with our mock.
Lines 7-8: Call the endpoint's invoke method that is under test.
Lines 10-15: Assert that the returned Element is valid.