Wednesday, March 30, 2011

Creating large in-memory database on AppEngine

For high performance AppEngine app development, it often comes in handy to store read-only database in memory. Accessing in-memory data is not only orders of magnitude faster than making DataStore or memcached calls but also a lot cheaper as it does not incur any API cost.

However, if you encode data as a Python dictionary or list with a million entries, your app will most likely crash on AppEngine, throwing a distasteful exceptions.MemoryError: Exceeded soft process size limit with 299.98 MB. "But I'm only loading 10MB of data!", you proclaim. Unfortunately, Python may temporarily consume over a gigabyte of memory while parsing and constructing a multi-megabyte dictionary or list.

The first thing you should consider is to simplify the data structure. If possible, flatten your database into one-dimensional lists, which enjoy a smaller memory footprint than dictionaries and multi-level nested lists.

Next, try data serialization using the pickle library. Be sure to use protocol version 2 for maximum efficiency and compactness. For example:
# To serialize data
pickle.dump(data, open('data.bin', 'w'), pickle.HIGHEST_PROTOCOL)
# To deserialize data
data = pickle.load(open(os.path.join(os.path.dirname(__file__), 'data.bin'), 'r'))

As AppEngine does not support the much faster cPickle module ("cPickle" is aliased to "pickle" on AppEngine), your app may time out if you try to unpickle millions of records. One effective solution is to store your data in homogeneous arrays to take advantage of array's highly efficient serialization implementation. Suppose you have a list of a million signed integers, you may first convert the list into a typed array and save it in a binary file:
array.array('i', data).tofile(open('data.bin', 'w'))
Deserializing the array literally takes just a few milliseconds on AppEngine:
data = array.array('i')
data.fromfile(open(os.path.join(os.path.dirname(__file__), 'data.bin'), 'r'), 1000000)

One caveat: To load more than 10MB of data, you will have to split the database into multiple files to work around AppEngine's size limit of static files.

Sunday, March 6, 2011

Book Notes: Pragmatic Thinking and Learning

In his book Pragmatic Thinking and Learning: Refactor Your Wetware, Andy Hunt explores interesting aspects of how human brain works, and more importantly, how we can rewire our "wetware" to be more effective learners.

Tip 1. Always consider the context.

Nothing exists in isolation.

Five stages on the journey from novice to expert:
  • Novices, who need fixed rules to follow.
  • Advanced beginners, who can try new tasks on their own but have no holistic understanding yet.
  • Competent, who can solve novel problems and apply advice from experts.
  • Proficient, who can evaluate their past performance and self-correct.
  • Expert, who can tap into a vast body of experience and work from intuition.

Tip 2. Use rules for novices, intuition for experts.

Tip 3. Know what you don't know.

Deliberate practice requires four conditions:
  • You need a well-defined task.
  • The task needs to be challenging but doable.
  • The environment needs to supply informative feedback.
  • It should also provide opportunities for repetition and correction of errors.

Tip 4. Learn by watching and imitating.

  • Imitate
  • Assimilate
  • Innovate

Tip 5. Keep practicing in order to remain expert.

Tip 6. Avoid formal methods if you need creativity, intuition, or inventiveness.

Tip 7. Learn the skill of learning.

Understanding skills acquisition is a skill itself.

Your brain has two modes (commonly known as left-brain and right-brain thinking):
  • L-mode (linear processing mode) is critical for sequential reasoning and problem solving.
  • R-mode (asynchronous, holistic, rich mode) is responsible for intuition and creativity.

Tip 8. Capture all ideas to get more of them.

Take advantage of external, preferably digital information organizers.

Tip 9. Learn by synthesis as well as by analysis.

Tip 10. Strive for good design; It really works better.

Creativity comes from the selection and assembly of the right components in the right presentation in the right context.

Tip 11. Rewire your brain with belief and constant practice.

Just thinking that your brain has more capacity for learning makes it so.

Tip 12. Add sensory experience to engage more of your brain.

Tip 13. Lead with R-mode; follow with L-mode.

Tip 14. Use metaphor as the meeting place between R-mode and L-mode.

Tip 15. Cultivate humor to build stronger metaphors.

The Morning Pages Technique for writers:
  • Write your morning pages first thing in the morning.
  • Write at least three pages.
  • Do not censor what your write.
  • Do not skip a day.

Tip 16. Step away from the keyboard to solve hard problems.

Yoga, meditation, breathing techniques, and martial arts all affect how your brain processes information.

Tip 17. Change your viewpoint to solve the problem.

Common cognitive biases to be aware of:
  • Anchoring
  • Fundamental attribution error
  • Self-serving bias
  • Need for closure
  • Confirmation bias
  • Exposure effect
  • Hawthorne effect
  • False memory
  • Symbolic reduction fallacy
  • Nominal fallacy
  • Confusion of correlation and causation

Tip 18. Watch the outliers; "rarely" doesn't mean "never".

Tip 19. Be comfortable with uncertainty.

Tip 20. Trust ink over memory; every mental read is a write.

Tip 21. Hedge your bets with diversity (of thinking to keep from falling victim to your generation's particular set of biases).

Tip 22. Allow for different bugs in different people.

Don't try to change other people's temperament to match your own.

Tip 23. Act like you've evolved; breathe, don't hiss.

Let your lizard reaction pass. Control your emotions.

Tip 24. Trust intuition, but verify.

Tip 25. Create SMART objectives to reach your goals.

SMART objectives are:
  • S - Specific
  • M - Measurable
  • A - Achievable
  • R - Relevant
  • T - Time-boxed

Tip 26. Plan your investment in learning deliberately.

Tip 27. Discover how you learn best (by experimenting with different learning modes).

Myers-Briggs types are not destiny. You can always choose to act differently.

Tip 28. Form study groups to learn and teach.

Tip 29. Read deliberately.

  • Survey the book in question to get a good overview without delving into any details.
  • Write down questions you want answered.
  • Read the book in its entirety; Recite, recall, rephrase, and reread important bits.
  • Finally review the material; Take notes and maybe draw mind maps to help visualization of information.

Tip 30. Take notes.

Tip 31. Write on: documenting is more important than documentation.

Tip 32. See it. Do it. Teach it.

Teaching forces you to constantly retrieve information which is critical for learning.

Tip 33. Play more in order to learn more.

Tip 34. Learn from similarities; unlearn from differences.

Tip 35. Explore, invent, and apply in your environment.

Tip 36. See without judging and then act.

Cultivate nonjudgmental awareness; don't try to get it right the first time, but notice when it goes wrong.

Tip 37. Give yourself permission to fail; It's the path to success.

Tip 38. Groove your mind for success.

Create mental conditions that you'd experience once you learn to perform at a higher level.

Tip 39. Learn to pay attention.

Learn the basics of meditation to more efficiently allocate your limited "attentional resources".

Tip 40. Make thinking time.

It takes time to marinate ideas. Sitting around doing nothing is part of the creative process.

Tip 41. Use a wiki to manage information and knowledge.

Tip 42. Establish rules of engagement to manage interruptions.

Avoid distractions; reduce context switching cost.

Tip 43. Send less email, and you'll receive less email.

Tip 44. Choose your own tempo for an email conversation.

Tip 45. Mask interrupts to maintain focus.

Tip 46. Use multiple monitors to avoid context switching.

Tip 47. Optimize your personal workflow.

Change is hard. Suggestions to help you manage effective change:
  • Start with a plan.
  • Avoid inaction.
  • Take time to develop new habits.
  • Belief is real. You have to believe that change is possible.
  • Take small, next steps. Choose a small, achievable goal, and reward yourself for reaching it. Rinse and repeat.

Tip 48. Grab the wheel. You can't steer on autopilot.

You need to constantly reevaluate yourself and your condition.

Reading Pragmatic Thinking and Learning: Refactor Your Wetware didn't reshape my thinking habits overnight, but it definitely helped me develop a higher level of consciousness of my state of mind.